block truncated newton methods for parallel optimization truncated newton methods are a class of optimization methods suitable for large scale problems at each iteration a search direction is obtained by approximately solving the newton equations using an iterative method in this way matrix costs and second derivative calculations are avoided hence removing the major drawbacks of newton s method in this form the algorithms are well suited for vectorization further improvements in performance are sought by using block iterative methods for computing the search direction in particular conjugate gradient type methods are considered computational experience on a hypercube computer is reported indicating that on some problems the improvements in performance can be better than that attributable to parallelism alone

a numerical study of the limited memory bfgs method and the truncated newton method for large scale optimization this paper examines the numerical performances of two methods for large scale optimization a limited memory quasi newton method l bfgs and a discrete truncated newton method tn various ways of classifying test problems are discussed in order to better understand the types of problems that each algorithm solves well the l bfgs and tn methods are also compared with the polak ribiere conjugate gradient method

limited memory bfgs systems with diagonal updates abstract in this paper we investigate a formula to solve systems of the form b i x y where b is a limited memory bfgs quasi newton matrix and is a positive constant these types of systems arise naturally in large scale optimization such as trust region methods as well as doubly augmented lagrangian methods we show that provided a simple condition holds on b 0 and the system b i x y can be solved via a recursion formula that requires only vector inner products this formula has complexity m 2 n where m is the number of l bfgs updates and n m is the dimension of x

some numerical experiments with variable storage quasi newton algorithms this paper describes some numerical experiments with variable storage quasi newton methods for the optimization of some large scale models coming from fluid mechanics and molecular biology in addition to assessing these kinds of methods in real life situations we compare an algorithm of a buckley with a proposal by j nocedal the latter seems generally superior provided that careful attention is given to some nontrivial implementation aspects which concern the general question of properly initializing a quasi newton matrix in this context we find it appropriate to use a diagonal matrix generated by an update of the identity matrix so as to fit the rayleigh ellipsoid of the local hessian in the direction of the change in the gradient also a variational derivation of some rank one and rank two updates in hilbert spaces is given

on the use of stochastic hessian information in optimization methods for machine learning this paper describes how to incorporate sampled curvature information in a newton cg method and in a limited memory quasi newton method for statistical learning the motivation for this work stems from supervised machine learning applications involving a very large number of training points we follow a batch approach also known in the stochastic optimization literature as a sample average approximation approach curvature information is incorporated in two subsampled hessian algorithms one based on a matrix free inexact newton iteration and one on a preconditioned limited memory bfgs iteration a crucial feature of our technique is that hessian vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient the efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition

on the oscillations of infinite periodic beams subjected to a moving concentrated force abstract a study is presented of the oscillation of an infinite beam which rests on identical periodic simple elastic supports the study shows how a combination of two different approaches the previous author s 1 and mead s 2 allows one to remove an unnatural restriction on the parameters of the problem the beam s transverse deflection is caused by a harmonic concentrated force moving steadily along the beam it is supposed that the beam had been at rest before the force approached and returned to rest after the force had moved away it is further supposed that the beam deflections at any pair of points separated on the periodic support spacing obey the special condition this condition means that the steady beam oscillations take place and allows one to consider just one beam segment between neighbouring supports the beam deflection is governed within the segment by the euler bernoulli partial differential equation four boundary conditions and the above mentioned suppositions the fourier transformation is used to solve the problem

dynamic response of micro periodic composite rods with uncertain parameters under moving random load in this paper axial vibrations of a finite micro periodic composite rod with uncertain parameters under a moving random load are investigated the solution of the problem was found by using the random dynamic influence function and applying the perturbation method the average tolerance approach was also used to pass from differential equations with periodic coefficients to differential equations with constant coefficients two types of the moving random load are considered a normal stationary or non stationary process the continuous load model clm and a random train of moving forces the discrete load model dlm finally a numerical example is provided to demonstrate the algorithm in the context of a computer implementation with slight modification the algorithm for longitudinal vibrations of a rod could also be applied to the dynamics of periodic beams plates and shells with random parameters subjected to stochastic moving loads

adjoint implementation of rosenbrock methods applied to variational data assimilation problems abstract in the past decade the variational method has been successfully applied in data assimilation problems for atmospheric chemistry models in 4d var data assimilation a minimization algorithm is used to find the set of control variables which minimizes the weighted least squares distance between model predictions and observations over the assimilation window using the adjoint method the gradient of the cost function can be computed fast at the expense of few function evaluations making the optimization process very efficient for large scale models the high storage requirements and the difficulty of implementing the adjoint code when sophisticated integrators are used to solve the stiff chemistry make the assimilation a very intensive computational process if the sparse structure of the chemical models is carefully exploited rosenbrock methods have been proved to be reliable chemistry solvers because of their outstanding stability properties and conservation of the linear invariants of the system in this paper we present an efficient implementation of the adjoint code for the rosenbrock type methods which can reduce the storage requirements of the forward model and is suitable for automatization the adjoint code is completely generated using symbolic preprocessing and automatic differentiation tools which allow flexibility and require minimal user intervention

mixing linear svms for nonlinear classification in this paper we address the problem of combining linear support vector machines svms for classification of large scale nonlinear datasets the motivation is to exploit both the efficiency of linear svms lsvms in learning and prediction and the power of nonlinear svms in classification to this end we develop a lsvm mixture model that exploits a divide and conquer strategy by partitioning the feature space into subregions of linearly separable datapoints and learning a lsvm for each of these regions we do this implicitly by deriving a generative model over the joint data and label distributions consequently we can impose priors on the mixing coefficients and do implicit model selection in a top down manner during the parameter estimation process this guarantees the sparsity of the learned model experimental results show that the proposed method can achieve the efficiency of lsvms in the prediction phase while still providing a classification performance comparable to nonlinear svms

optimal algorithms for well conditioned nonlinear systems of equations we propose solving nonlinear systems of equations by function optimization and we give an optimal algorithm which relies on a special canonical form of gradient descent the algorithm can be applied under certain assumptions on the function to be optimized that is an upper bound must exist for the norm of the hessian whereas the norm of the gradient must be lower bounded due to its intrinsic structure the algorithm looks particularly appealing for a parallel implementation as a particular case more specific results are given for linear systems we prove that reaching a solution with a degree of precision spl epsiv takes spl theta n sup 2 k sup 2 log sup k sub spl epsiv k being the condition number of a and n the problem dimension related results hold for systems of quadratic equations for which an estimation for the requested bounds can be devised finally we report numerical results in order to establish the actual computational burden of the proposed method and to assess its performances with respect to classical algorithms for solving linear and quadratic equations

numerical methods for electronic structure calculations of materials the goal of this article is to give an overview of numerical problems encountered when determining the electronic structure of materials and the rich variety of techniques used to solve these problems the paper is intended for a diverse scientific computing audience for this reason we assume the reader does not have an extensive background in the related physics our overview focuses on the nature of the numerical problems to be solved their origin and the methods used to solve the resulting linear algebra or nonlinear optimization problems it is common knowledge that the behavior of matter at the nanoscale is in principle entirely determined by the schrodinger equation in practice this equation in its original form is not tractable successful but approximate versions of this equation which allow one to study nontrivial systems took about five or six decades to develop in particular the last two decades saw a flurry of activity in developing effective software one of the main practical variants of the schrodinger equation is based on what is referred to as density functional theory dft the combination of dft with pseudopotentials allows one to obtain in an efficient way the ground state configuration for many materials this article will emphasize pseudopotential density functional theory but other techniques will be discussed as well

approximating extent measures of points we present a general technique for approximating various descriptors of the extent of a set p of n points in rd when the dimension d is an arbitrary fixed constant for a given extent measure and a parameter e 0 it computes in time o n 1 eo 1 a subset q p of size 1 eo 1 with the property that 1 e p q p the specific applications of our technique include e approximation algorithms for i computing diameter width and smallest bounding box ball and cylinder of p ii maintaining all the previous measures for a set of moving points and iii fitting spheres and cylinders through a point set p our algorithms are considerably simpler and faster in many cases than previously known algorithms

reduced hessian quasi newton methods for unconstrained optimization quasi newton methods are reliable and efficient on a wide range of problems but they can require many iterations if the problem is ill conditioned or if a poor initial estimate of the hessian is used in this paper we discuss methods designed to be more efficient in these situations all the methods to be considered exploit the fact that quasi newton methods accumulate approximate second derivative information in a sequence of expanding subspaces associated with each of these subspaces is a certain reduced approximate hessian that provides a complete but compact representation of the second derivative information approximated up to that point algorithms that compute an explicit reduced hessian approximation have two important advantages over conventional quasi newton methods first the amount of computation for each iteration is significantly less during the early stages this advantage is increased by forcing the iterates to linger on a manifold whose dimension can be significantly smaller than the subspace in which curvature has been accumulated second approximate curvature along directions that lie off the manifold can be reinitialized as the iterations proceed thereby reducing the influence of a poor initial estimate of the hessian these advantages are illustrated by extensive numerical results from problems in the cute test set our experiments provide strong evidence that reduced hessian quasi newton methods are more efficient and robust than conventional bfgs methods and some recently proposed extensions

a novel variational algorithmic formulation for wrinkling at finite strains based on energy minimization application to mesh adaption this paper is concerned with an efficient novel algorithmic formulation for wrinkling at finite strains in contrast to previously published numerical implementations the advocated method is fully variational more precisely the parameters describing wrinkles or slacks together with the unknown deformation mapping are computed jointly by minimizing the potential energy of the considered mechanical system furthermore the wrinkling criteria are naturally included within the presented variational framework the proposed method allows to employ three dimensional constitutive models without any additional modification i e a projection in plane stress space is not required analogously to the wrinkling parameters the non vanishing out of plane component of the strain tensor results conveniently from relaxing the respective helmholtz energy of the membrane the proposed framework is very general and does not rely on any assumption regarding the symmetry of the material i e arbitrary anisotropic hyperelastic models can be consistently taken into account the advantages associated with such a variational method are manifold for instance it opens up the possibility of applying standard optimization algorithms to the numerical implementation this is especially important for highly non linear or singular problems such as wrinkling on the other hand minimization principles provide a suitable basis for a posteriori error estimation and thus for adaptive finite element formulations as a prototype a variational error indicator leading to an efficient h adaption for wrinkling is briefly discussed the performance of the wrinkling approach is demonstrated by selected finite element analyses

principal component analysis combined with truncated newton minimization for dimensionality reduction of chemical databases the similarity and diversity sampling problems are two challenging optimization tasks that arise in the analysis of chemical databases as a first step to their solution we propose an efficient projection refinement protocol based on the principal component analysis pca and the truncated newton minimization method implemented by our package tnpack pca tnpack we show that pca can provide the same initial guess as the singular value decomposition svd for the optimization task of solving the distance geometry optimization problem if each column of a database matrix has a mean of zero hence our pca tnpack approach is analogous to the svd tnpack projection refinement protocol that we developed recently for visualizing large chemical databases using pca tnpack and the merck mddr database mdl drug data report we further investigate the projection refinement procedure with regards to the preservation of the original clusters of chemical compounds the accuracy of similarity and diversity sampling of chemical compounds and the potential application in the study of structure activity relationships we also explore by simple experiments accuracy and efficiency aspects of the pca tnpack procedure compared to those of a global optimization algorithm simulated annealing as implemented by the program package simann in terms of producing the projection mapping of a database numerical results show that the 2d pca tnpack mapping can preserve the distance relationships of the original database and is thus valuable as a first step in similarity and diversity applications of course the generation of a global rather than local minimizer and its interpretation in terms of pharameceutical applications remains a challenge since all numerical tests are performed on the merck mddr database results are representative of realistic cases encountered in the field of drug design and may help analyze properties of medicinal compounds

accelerated conjugate gradient algorithm with finite difference hessian vector product approximation for unconstrained optimization in this paper we propose a fundamentally different conjugate gradient method in which the well known parameter b k is computed by an approximation of the hessian vector product through finite differences for search direction computation the method uses a forward difference approximation to the hessian vector product in combination with a careful choice of the finite difference interval for the step length computation we suggest an acceleration scheme able to improve the efficiency of the algorithm under common assumptions the method is proved to be globally convergent it is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear but the reduction in function values is significantly improved numerical comparisons with conjugate gradient algorithms including conmin by shanno and phua d f shanno k h phua algorithm 500 minimization of unconstrained multivariate functions acm trans math softw 2 1976 87 94 scalcg by andrei n andrei scaled conjugate gradient algorithms for unconstrained optimization comput optim appl 38 2007 401 416 n andrei scaled memoryless bfgs preconditioned conjugate gradient algorithm for unconstrained optimization optim methods softw 22 2007 561 571 n andrei a scaled bfgs preconditioned conjugate gradient algorithm for unconstrained optimization appl math lett 20 2007 645 650 and new conjugacy condition and related new conjugate gradient by li tang and wei g li c tang z wei new conjugacy condition and related new conjugate gradient methods for unconstrained optimization j comput appl math 202 2007 523 539 or truncated newton tn by nash s g nash preconditioning of truncated newton methods siam j on scientific and statistical computing 6 1985 599 616 using a set of 750 unconstrained optimization test problems show that the suggested algorithm outperforms these conjugate gradient algorithms as well as tn

algorithms for a minimum volume enclosing simplex in three dimensions we develop a combinatorial algorithm for determining a minimum volume simplex enclosing a set of points in cal r 3 if the convex hull of the points has n vertices then our algorithm takes theta n 4 time combining our exact but slow algorithm with a simple but crude approximation technique we also develop an varepsilon approximation algorithm the algorithm computes in o n 1 varepsilon 6 time a simplex whose volume is within 1 varepsilon factor of the optimal for any varepsilon 0

a limited memory quasi newton trust region method for box constrained optimization by means of wolfe conditions strategy we propose a quasi newton trust region method to solve box constrained optimization problems this method is an adequate combination of the compact limited memory bfgs and the trust region direction while the generated point satisfies the wolfe conditions and therefore maintains a positive definite approximation to the hessian of the objective function the global convergence and the quadratic convergence of this method are established under suitable conditions finally we compare our algorithms iwtral and ibwtral with an active set trust region algorithm astral xu and burke 2007 on the cutest box constrained test problems presented by gould et al 2015 numerical results show that the presented method is competitive and totally interesting for solving box constrained optimization

net and prune a linear time algorithm for euclidean distance problems we provide a general framework for getting expected linear time constant factor approximations and in many cases fptas s to several well known problems in computational geometry such as k center clustering and farthest nearest neighbor the new approach is robust to variations in the input problem and yet it is simple elegant and practical in particular many of these well studied problems which fit easily into our framework either previously had no linear time approximation algorithm or required rather involved algorithms and analysis a short list of the problems we consider include farthest nearest neighbor k center clustering smallest disk enclosing k points kth largest distance kth smallest m nearest neighbor distance kth heaviest edge in the mst and other spanning forest type problems problems involving upward closed set systems and more finally we show how to extend our framework such that the linear running time bound holds with high probability

identification of boundary conditions in a nonlinear shallow water flow abstract in this paper we present sufficient smoothness results on the solution of a nonlinear shallow water problem in order to obtain an existence result for the associated adjoint problem we introduce a necessary but not sufficient existence condition for the functional minimum the control is made on the velocity boundary condition in considering observations at isolated points moreover we give a set of numerical results obtained in a real situation

calculation of force distribution for a periodically supported beam subjected to moving loads abstract in this study a novel model for a periodically supported beam subjected to moving loads was developed using a periodicity condition on reaction forces this condition together with fourier transforms and dirac combs forms a relation between the beam displacement and support reaction forces this relation explains the force distribution at the supports and holds for any type of support and foundation behaviors based on this relation a system equivalence for a periodically supported beam is presented in this paper an application to non ballasted viscoelastic supports is presented as an example and the results clearly match the existing model next an approximation of real time responses was developed for the moving loads as periodic series the comparison shows that this approximation can be used for a limited number of loads if the distances between loads are sufficiently large the system equivalence for a periodically supported beam is efficient for supports with linear behavior and could be extended to other behaviors

action recognition based on overcomplete independent components analysis existing works on action recognition rely on two separate stages 1 designing hand designed features or learning features from video data 2 classifying features using a classifier such as svm or adaboost motivated by two observations 1 independent component analysis ica is capable of encoding intrinsic features underlying video data and 2 videos of different actions can be easily distinguished by their intrinsic features we propose a simple but effective action recognition framework based on the recently proposed overcomplete ica model after a set of overcomplete ica basis functions are learned from the densely sampled 3d patches from training videos for each action a test video is classified as the class whose basis functions can reconstruct the sampled 3d patches from the test video with the smallest reconstruction error the experimental results on five benchmark datasets demonstrate that the proposed approach outperforms several state of the art works

newton type minimization via the lanczos method this paper discusses the use of the linear conjugate gradient method developed via the lanczos method in the solution of large scale unconstrained minimization problems at each iteration of a newton type method the direction of search is defined as the solution of a quadratic subproblem when the number of variables is very large this subproblem may be solved using the linear conjugate gradient method of hestenes and stiefel we show how the equivalent lanczos characterization of the linear conjugate gradient method may be exploited to define a modified newton method which can be applied to problems that do not necessarily have positive definite hessian matrices at all points of the region of interest this derivation also makes it possible to compute a negative curvature direction at a stationary point the idea of a truncated newton method is to perform only a limited number of iterations of the quadratic subproblem this effectively gives a search direction that interpolates between the steepest de

a algorithm for convex minimization for convex minimization we introduce an algorithm based on space decomposition the method uses a bundle subroutine to generate a sequence of approximate proximal points when a primal dual track leading to a solution and zero subgradient pair exists these points approximate the primal track points and give the algorithm s or corrector steps the subroutine also approximates dual track points that are gradients needed for the method s newton predictor steps with the inclusion of a simple line search the resulting algorithm is proved to be globally convergent the convergence is superlinear if the primal dual track points and the objective s hessian are approximated well enough

truncated newton training algorithm for neurocomputational viscoplastic model we present an estimate approach to compute the viscoplastic behavior of a polymer matrix composite under different thermomechanical environments this investigation incorporates computational neural network as the tool for determining the creep behavior of the composite we propose a new second order learning algorithm for training the multilayer networks training in the neural network is generally specified as the minimization of an appropriate error function with respect to parameters of the network weights and learning rates corresponding to excitory and inhibitory connections we propose here a technique for error minimization based on the use of the truncated newton tn large scale unconstrained minimization technique with quadratic convergence rate this technique offers a more sophisticated use of the gradient information compared to simple steepest descent or conjugate gradient methods in this work we briefly specify the necessary details for implementing the tn method for training the neural networks that predicts the viscoplastic behavior of the polymeric composite we provide comparative experimental results and explicit model results to verify the effectiveness of the neural networks based model these results verify the superiority of the present approach compared to the explicit modeling scheme moreover the present study demonstrates for the first time the feasibility of introducing the tn method with quadratic convergence rate to the field of neural networks

approximating the radii of point sets we consider the problem of computing the outer radii of point sets in this problem we are given integers n d and k where k le d and a set p of n points in re d the goal is to compute the outer k radius of p denoted by cal r k p which is the minimum over all d k dimensional flats f of max p in p d p f where d p f is the euclidean distance between the point p and flat f computing the radii of point sets is a fundamental problem in computational convexity with many significant applications the problem admits a polynomial time algorithm when the dimension d is constant u faigle w kern and m streng math program 73 1996 pp 1 5 here we are interested in the general case in which the dimension d is not fixed and can be as large as n where the problem becomes np hard even for k 1 it is known that r k p can be approximated in polynomial time by a factor of 1 varepsilon for any varepsilon 0 when d k is a fixed constant m babdoiu s har peled and p indyk in proceedings of the acm symposium on the theory of computing 2002 s har peled and k varadarajan in proceedings of the acm symposium on computing geometry 2002 a polynomial time algorithm that guarantees a factor of o sqrt log n approximation for r 1 p the width of the point set p is implied by the results of nemirovski roos and terlaky math program 86 1999 pp 463 473 and nesterov handbook of semidefinite programming theory algorithms kluwer academic publishers norwell ma 2000 in this paper we show that r k p can be approximated by a ratio of o sqrt log n for any 1 leq k leq d thus matching the previously best known ratio for approximating the special case r 1 p the width of point set p our algorithm is based on semidefinite programming relaxation with a new mixed deterministic and randomized rounding procedure we also prove an inapproximability result that gives evidence that our approximation algorithm is doing well for a large range of k we show that there exists a constant delta 0 such that the following holds for any 0 eps 1 there is no polynomial time algorithm that approximates r k p within log n delta for all k such that k leq d d varepsilon unless np subseteq dtime 2 log m o 1 our inapproximability result for r k p extends a previously known hardness result of brieden discrete comput geom 28 2002 pp 201 209 and is proved by modifying brieden s construction using basic ideas from probabilistically checkable proofs pcp theory

approaches to robust pole assignment abstract robust pole assignment is a non linear optimization problem in many variables we describe numerical methods for determining robust or well conditioned so lutions to the problem of pole assignment by state feedback the solutions are chosen to minimize various objective functions based on the condition number of the eigenvector matrix careful choice of parametrization and objective function avoids singularities and artificial variable constraints explicit formulae for the gradient and hessian permit rigorous stopping criteria and rapid local convergence several computational examples are included

global convergence properties of the two new dependent fletcher reeves conjugate gradient methods in this paper we propose two new dependent fletcher reeves conjugate gradient methods arising from different choice for the scalar b k we make two different kinds of estimations of upper bounds of b k with respect to b k f r which are based on abel theorem of non convergent series of positive items with several different line searches global convergence results are established for the two new methods which extend the previous dependent fletcher reeves conjugate gradient methods

a subexponential algorithm for abstract optimization problems an abstract optimization problem aop is a triple h phi where h is a finite set a total order on 2 h and phi an oracle that for given f subseteq g subseteq h either reports that f min f mid f subseteq g or returns a set f subseteq g with f e 2 sqrt n o sqrt 4 n ln n oracle calls n h in contrast any deterministic algorithm needs to make 2 n 1 oracle calls in the worst case the algorithm is applied to the problem of finding the distance between two n vertex or n facet convex polyhedra in d space and to the computation of the smallest ball containing n points in d space for both problems we give the first subexponential bounds in the arithmetic model of computation

minimum width rectangular annulus in this paper we identify a minimum width rectangular annulus that encloses a given set of n points in a plane we propose an o n 2logn time and o n space algorithm for this problem to the best of our knowledge this is the first sub cubic algorithm for a rectangular annulus for arbitrary orientation

global convergence of online limited memory bfgs global convergence of an online stochastic limited memory version of the broyden fletcher goldfarb shanno bfgs quasi newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established lower and upper bounds on the hessian eigenvalues of the sample functions are shown to suffice to guarantee that the curvature approximation matrices have bounded determinants and traces which in turn permits establishing convergence to optimal arguments with probability 1 experimental evaluation on a search engine advertising problem showcase reductions in convergence time relative to stochastic gradient descent algorithms

bfgs with update skipping and varying memory we give conditions under which limited memory quasi newton methods with exact line searches will terminate in n steps when minimizing n dimensional quadratic functions we show that although all broyden family methods terminate in n steps in their full memory versions only bfgs does so with limited memory additionally we show that full memory broyden family methods with exact line searches terminate in at most n p steps when p matrix updates are skipped we introduce new limited memory bfgs variants and test them on nonquadratic minimization problems

a primal dual method for large scale image reconstruction in emission tomography in emission tomography images can be reconstructed from a set of measured projections using a maximum likelihood ml criterion in this paper we present a primal dual algorithm for large scale three dimensional image reconstruction the primal dual method is specialized to the ml reconstruction problem the reconstruction problem is extremely large in several of our data sets the hessian of the objective function is the product of a 1 4 million by 63 million matrix and its scaled transpose as such we consider only approaches that are suitable for large scale parallel computation we apply a stabilization technique to the system of equations for computing the primal direction and demonstrate the need for stabilization when approximately solving the system using an early terminated conjugate gradient iteration we demonstrate that the primal dual method for this problem converges faster than the logarithmic barrier method and considerably faster than the expectation maximization algorithm the use of extrapolation in conjunction with the primal dual method further reduces the overall computation required to achieve convergence

the limited memory conjugate gradient method in theory the successive gradients generated by the conjugate gradient method applied to a quadratic should be orthogonal however for some ill conditioned problems orthogonality is quickly lost due to rounding errors and convergence is much slower than expected a limited memory version of the nonlinear conjugate gradient method is developed the memory is used to both detect the loss of orthogonality and to restore orthogonality an implementation of the algorithm is presented based on the cg descent nonlinear conjugate gradient method limited memory cg descent l cg descent possesses a global convergence property similar to that of the memoryless algorithm but has much better practical performance numerical comparisons to the limited memory bfgs method l bfgs are given using the cuter test problems

a tool for the analysis of quasi newton methods with application to unconstrained minimization the bfgs update formula is shown to have an important property that is inde pendent of the algorithmic context of the update and that is relevant to both constrained and unconstrained optimization the bfgs method for unconstrained optimization using a variety of line searches including backtracking is shown to be globally and superlinearly convergent on uniformly convex problems the analysis is particularly simple due to the use of some new tools introduced in this paper

modeling textured images using generalized long correlation models the long correlation lc models are a general class of random field rf models which are able to model correlations extending over large image regions with few model parameters the lc models have seen limited use due to lack of an effective method for estimating the model parameters in this work we develop an estimation scheme for a very general form of this model and demonstrate its applicability to texture modeling applications the relationship of the generalized lc models to other classes of rf models namely the simultaneous autoregressive sar and markov random field mrf models is shown while it is known that the sar model is a special case of the lc model we show that the mrf model is also encompassed by this model consequently the lc model may be considered as a generalization of the sar and mrf models

a new conjugate gradient method with guaranteed descent and an efficient line search a new nonlinear conjugate gradient method and an associated implementation based on an inexact line search are proposed and analyzed with exact line search our method reduces to a nonlinear version of the hestenes stiefel conjugate gradient scheme for any inexact line search our scheme satisfies the descent condition bf g k sf t bf d k le frac 7 8 bf g k 2 moreover a global convergence result is established when the line search fulfills the wolfe conditions a new line search scheme is developed that is efficient and highly accurate efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer high accuracy is achieved by using a convergence criterion which we call the approximate wolfe conditions obtained by replacing the sufficient decrease criterion in the wolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion numerical comparisons are given with both l bfgs and conjugate gradient methods using the unconstrained optimization problems in the cute library

a practical pr conjugate gradient method only using gradient abstract in this paper we consider the polak ribiere or polak ribiere plus conjugate gradient method for solving optimality condition of an unconstrained minimization problem we give two new steplength rules only using gradient and under gradient lipschitz assumption prove this method s global convergence correspondingly then we develop a practical polak ribiere plus method whose steplength is located by one inequality only using gradient and report promising numerical results on high accuracy solution for some standard test problems when compared to the state of art methods in this research direction importantly our work provides a new idea of devising a practical version of the celebrated polak ribiere or polak ribiere plus method

on nonlinear generalized conjugate gradient methods the generalized conjugate gradient method see 1 is an iterative method for nonsymmetric linear systems we obtain generalizations of this method for nonlinear systems with nonsymmetric jacobians we prove global convergence results

minimum width double strip and parallelogram annulus abstract in this paper we study the problem of computing a minimum width double strip or parallelogram annulus that encloses a given set of n points in the plane a double strip is a closed region in the plane whose boundary consists of four parallel lines and a parallelogram annulus is a closed region between two edge parallel parallelograms we present several first algorithms for these problems among them are o n 2 and o n 3 log n time algorithms that compute a minimum width double strip and parallelogram annulus respectively when their orientations can be freely chosen

a fully polynomial time approximation scheme for the smallest diameter of imprecise points abstract given a set d d 1 d n of imprecise points modeled as disks the minimum diameter problem is to locate a set p p 1 p n of fixed points where p i d i such that the furthest distance between any pair of points in p is as small as possible this introduces a tight lower bound on the size of the diameter of any instance p in this paper we present a fully polynomial time approximation scheme fptas for computing the minimum diameter of a set of disjoint disks that runs in o n 2 1 time then we relax the disjointness assumption and we show that adjusting the presented fptas will cost o n 2 2 time we also show that our results can be generalized in r d when the dimension d is an arbitrary fixed constant

a theoretical and experimental study of the symmetric rank one update this paper first discusses computational experience using the sri update in conventional line search and trust region algorithms for unconstrained optimization the experiments show that the sri is very competitive with the widely used bfgs method they also indicate two interesting features the final hessian approximations produced by the sri method are not generally appreciably better than those produced by the bfgs and the sequences of steps produced by the sri do not usually seem to have the uniform linear independence property that is assumed in recent convergence analysis this paper presents a new analysis that shows that the sri method with a line search is n 1 step q superlinearly convergent without the assumption of linearly independent iterates this analysis assumes that the hessian approximations are positive definite and bounded asymptotically which from computational experience are reasonable assumptions

accelerated scaled memoryless bfgs preconditioned conjugate gradient algorithm for unconstrained optimization abstract an accelerated scaled memoryless bfgs preconditioned conjugate gradient algorithm for solving unconstrained optimization problems is presented the basic idea is to combine the scaled memoryless bfgs method and the preconditioning technique in the frame of the conjugate gradient method the preconditioner which is also a scaled memoryless bfgs matrix is reset when the beale powell restart criterion holds the parameter scaling the gradient is selected as a spectral gradient for the steplength computation the method has the advantage that in conjugate gradient algorithms the step lengths may differ from 1 by two order of magnitude and tend to vary unpredictably thus we suggest an acceleration scheme able to improve the efficiency of the algorithm under common assumptions the method is proved to be globally convergent it is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear but the reduction in the function values is significantly improved in mild conditions the algorithm is globally convergent for strongly convex functions computational results for a set consisting of 750 unconstrained optimization test problems show that this new accelerated scaled conjugate gradient algorithm substantially outperforms known conjugate gradient methods scalcg 3 4 5 6 conmin by shanno and phua 1976 1978 42 43 hestenes and stiefel 1952 25 polak ribiere polyak 1969 32 33 dai and yuan 2001 17 dai and liao 2001 t 1 14 conjugate gradient with sufficient descent condition 7 hybrid dai and yuan 2001 17 hybrid dai and yuan zero 2001 17 cg descent by hager and zhang 2005 2006 22 23 as well as quasi newton lbfgs method 26 and truncated newton method by nash 1985 27

parallel quasi newton methods for unconstrained optimization we discuss methods for solving the unconstrained optimization problem on parallel computers when the number of variables is sufficiently small that quasi newton methods can be used we concentrate mainly but not exclusively on problems where function evaluation is expensive first we discuss ways to parallelize both the function evaluation costs and the linear algebra calculations in the standard sequential secant method the bfgs method then we discuss new methods that are appropriate when there are enough processors to evaluate the function gradient and part but not all of the hessian at each iteration we develop new algorithms that utilize this information and analyze their convergence properties we present computational experiments showing that they are superior to parallelization either the bfgs methods or newton s method under our assumptions on the number of processors and cost of function evaluation finally we discuss ways to effectively utilize the gradient values at unsuccessful trial points that are available in our parallel methods and also in some sequential software packages

discrete second order adjoints in atmospheric chemical transport modeling atmospheric chemical transport models ctms are essential tools for the study of air pollution for environmental policy decisions for the interpretation of observational data and for producing air quality forecasts many air quality studies require sensitivity analyses i e the computation of derivatives of the model output with respect to model parameters the derivatives of a cost functional defined on the model output with respect to a large number of model parameters can be calculated efficiently through adjoint sensitivity analysis while the traditional first order adjoint models give the gradient of the cost functional with respect to parameters second order adjoint models give second derivative information in the form of products between the hessian of the cost functional and a vector representing a perturbation in sensitivity analysis a search direction in optimization an eigenvector etc in this paper we discuss the mathematical foundations of the discrete second order adjoint sensitivity method and present a complete set of computational tools for performing second order sensitivity studies in three dimensional atmospheric ctms the tools include discrete second order adjoints of runge kutta and of rosenbrock time stepping methods for stiff equations together with efficient implementation strategies numerical examples illustrate the use of these computational tools in important applications like sensitivity analysis optimization uncertainty quantification and the calculation of directions of maximal error growth in three dimensional atmospheric ctms

machine learning of linear differential equations using gaussian processes this work leverages recent advances in probabilistic machine learning to discover governing equations expressed by parametric linear operators such equations involve but are not limited to ordinary and partial differential integro differential and fractional order operators here gaussian process priors are modified according to the particular form of such operators and are employed to infer parameters of the linear equations from scarce and possibly noisy observations such observations may come from experiments or black box computer simulations as demonstrated in several synthetic examples and a realistic application in functional genomics employ probabilistic machine learning to discover governing equations expressed by parametric linear operators proper placement of gaussian process priors allows one to efficiently infer model parameters via maximum likelihood estimation a general treatment of inverse problems governed by linear operators leading to model discovery from just a handful of noisy measurements

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

iterated tabu search for the circular open dimension problem abstract this paper investigates the circular open dimension problem codp which consists of packing a set of circles of known radii into a strip of fixed width and unlimited length without overlapping the objective is to minimize the length of the strip in this paper codp is solved by a series of sub problems each corresponding to a fixed strip length for each sub problem an iterated tabu search approach named its is proposed its starts from a randomly generated solution and attempts to gain improvements by a tabu search procedure after that if the obtained solution is not feasible a perturbation operator is subsequently employed to reconstruct the incumbent solution and an acceptance criterion is implemented to determine whether or not accept the perturbed solution as a supplementary method the length of the strip is determined in monotonously decreasing way with the aid of some post processing techniques the search terminates and returns the best found solution after the allowed computation time has been elapsed computational experiments based on numerous well known benchmark instances show that its produces quite competitive results with respect to the best known results while the computational time remains reasonable for each instance

interior point methodology for 3 d pet reconstruction interior point methods have been successfully applied to a wide variety of linear and nonlinear programming applications this paper presents a class of algorithms based on path following interior point methodology for performing regularized maximum likelihood ml reconstructions on three dimensional 3 d emission tomography data the algorithms solve a sequence of subproblems that converge to the regularized maximum likelihood solution from the interior of the feasible region the nonnegative orthant the authors propose two methods a primal method which updates only the primal image variables and a primal dual method which simultaneously updates the primal variables and the lagrange multipliers a parallel implementation permits the interior point methods to scale to very large reconstruction problems termination is based on well defined convergence measures namely the karush kuhn tucker first order necessary conditions for optimality the authors demonstrate the rapid convergence of the path following interior point methods using both data from a small animal scanner and monte carlo simulated data the proposed methods can readily be applied to solve the regularized weighted least squares reconstruction problem

a maximum a posteriori estimation framework for robust high dynamic range video synthesis high dynamic range hdr image synthesis from multiple low dynamic range exposures continues to be actively researched the extension to hdr video synthesis is a topic of significant current interest due to potential cost benefits for hdr video a stiff practical challenge presents itself in the form of accurate correspondence estimation of objects between video frames in particular loss of data resulting from poor exposures and varying intensity makes conventional optical flow methods highly inaccurate we avoid exact correspondence estimation by proposing a statistical approach via maximum a posterior estimation and under appropriate statistical assumptions and choice of priors and models we reduce it to an optimization problem of solving for the foreground and background of the target frame we obtain the background through rank minimization and estimate the foreground via a novel multiscale adaptive kernel regression technique which implicitly captures local structure and temporal motion by solving an unconstrained optimization problem extensive experimental results on both real and synthetic data sets demonstrate that our algorithm is more capable of delivering high quality hdr videos than current state of the art methods under both subjective and objective assessments furthermore a thorough complexity analysis reveals that our algorithm achieves better complexity performance tradeoff than conventional methods

a globally convergent bfgs method for nonlinear monotone equations without any merit functions r since 1965 there has been significant progress in the theoretical study on quasi newton methods for solving nonlinear equations especially in the local convergence analysis however the study on global convergence of quasi newton methods is relatively fewer especially for the bfgs method to ensure global convergence some merit function such as the squared norm merit function is typically used in this paper we propose an algorithm for solving nonlinear monotone equations which combines the bfgs method and the hyperplane projection method we also prove that the proposed bfgs method converges globally if the equation is monotone and lipschitz continuous without differentiability requirement on the equation which makes it possible to solve some nonsmooth equations an attractive property of the proposed method is that its global convergence is independent of any merit function we also report some numerical results to show efficiency of the proposed method

learning translation invariant kernels for classification appropriate selection of the kernel function which implicitly defines the feature space of an algorithm has a crucial role in the success of kernel methods in this paper we consider the problem of optimizing a kernel function over the class of translation invariant kernels for the task of binary classification the learning capacity of this class is invariant with respect to rotation and scaling of the features and it encompasses the set of radial kernels we show that how translation invariant kernel functions can be embedded in a nested set of sub classes and consider the kernel learning problem over one of these sub classes this allows the choice of an appropriate sub class based on the problem at hand we use the criterion proposed by lanckriet et al 2004 to obtain a functional formulation for the problem it will be proven that the optimal kernel is a finite mixture of cosine functions the kernel learning problem is then formulated as a semi infinite programming sip problem which is solved by a sequence of quadratically constrained quadratic programming qcqp sub problems using the fact that the cosine kernel is of rank two we propose a formulation of a qcqp sub problem which does not require the kernel matrices to be loaded into memory making the method applicable to large scale problems we also address the issue of including other classes of kernels such as individual kernels and isotropic gaussian kernels in the learning process another interesting feature of the proposed method is that the optimal classifier has an expansion in terms of the number of cosine kernels instead of support vectors leading to a remarkable speedup at run time as a by product we also generalize the kernel trick to complex valued kernel functions our experiments on artificial and real world benchmark data sets including the usps and the mnist digit recognition data sets show the usefulness of the proposed method

partitioned variable metric updates for large structured optimization problems this paper presents a minimization method based on the idea of partitioned updating of the hessian matrix in the case where the objective function can be decomposed in a sum of convex element functions this situation occurs in a large class of practical problems including nonlinear finite elements calculations some theoretical and algorithmic properties of the update are discussed and encouraging numerical results are presented

automatic preconditioning by limited memory quasi newton updating this paper proposes a preconditioner for the conjugate gradient method cg that is designed for solving systems of equations ax bi with different right hand side vectors or for solving a sequence of slowly varying systems ak x bk the preconditioner has the form of a limited memory quasi newton matrix and is generated using information from the cg iteration the automatic preconditioner does not require explicit knowledge of the coefficient matrix a and is therefore suitable for problems where only products of a times a vector can be computed numerical experiments indicate that the preconditioner has most to offer when these matrix vector products are expensive to compute and when low accuracy in the solution is required the effectiveness of the preconditioner is tested within a hessian free newton method for optimization and by solving certain linear systems arising in finite element models

the stability of gradient like methods this paper deals with the stability properties of gradient like methods for nonlinear system of equations the fridman variant and the conjugate gradient method the main part of the article is to analyse the total exponential stability of these methods with respect to strongly consistent disturbances of the jacobian some numerical experiments are also given

steady state response of an infinite string on a non linear visco elastic foundation to moving point loads the steady state response of a long string on a non linear visco elastic foundation to uniformly moving constant point loads is studied the phase plane is employed to construct a phase portrait that determines the string response this portrait consists of trajectories of the autonomous system that represent the string displacement everywhere except at the loading points and vertical segments that satisfy the boundary conditions in these points it is shown that the string response depends crucially on the ratio between the load velocity and the wave speed in the string if the load moves slower than the waves in the string then the response is nearly symmetric with respect to the load and decays exponentially with the distance form the load if the load velocity exceeds the wave speed then the string exhibits a wave pattern which extends well behind the load in front of the load in this case the string is not disturbed

adjoint based acoustic inversion for the physical characterization of a shallow water environment recently the concept of adjoint modeling has been introduced in shallow water acoustics for solving inverse problems analytical adjoints have been derived for normal modes and for both the standard parabolic equation and claerbout s wide angle approximation wape this paper proposes the application of a semiautomatic adjoint approach that has been successfully applied in the past for multidimensional variational data assimilation in meteorological and climate modeling starting from a modular graph representation of the underlying forward model a programming tool facilitates the generation and coding of both the tangent linear and the adjoint models the potential of this numerical adjoint approach for the physical characterization of a shallow water environment is illustrated with two applications for geoacoustic inversion and ocean acoustic tomography using claerbout s wape in combination with nonlocal boundary conditions furthermore the adjoint optimization is extended to multiple frequencies and

quadratic projection based feature extraction with its application to biometric recognition this paper presents a novel quadratic projection based feature extraction framework where a set of quadratic matrices is learned to distinguish each class from all other classes we formulate quadratic matrix learning qml as a standard semidefinite programming sdp problem however the conventional interior point sdp solvers do not scale well to the problem of qml for high dimensional data to solve the scalability of qml we develop an efficient algorithm termed dualqml based on the lagrange duality theory to extract nonlinear features to evaluate the feasibility and effectiveness of the proposed framework we conduct extensive experiments on biometric recognition experimental results on three representative biometric recognition tasks including face palmprint and ear recognition demonstrate the superiority of the dualqml based feature extraction algorithm compared to the current state of the art algorithms highlightsa novel quadratic projection based feature extraction framework is developed a set of quadratic matrices is learnt to distinguish each class from other classes quadratic matrix learning qml is formulated as an sdp problem an efficient algorithm is developed to solve qml based on lagrange duality theory experiments on biometric recognition show the effectiveness of our algorithm

a time parallel approach to strong constraint four dimensional variational data assimilation a parallel in time algorithm based on an augmented lagrangian approach is proposed to solve four dimensional variational 4d var data assimilation problems the assimilation window is divided into multiple sub intervals that allows parallelization of cost function and gradient computations the solutions to the continuity equations across interval boundaries are added as constraints the augmented lagrangian approach leads to a different formulation of the variational data assimilation problem than the weakly constrained 4d var a combination of serial and parallel 4d vars to increase performance is also explored the methodology is illustrated on data assimilation problems involving the lorenz 96 and the shallow water models

iterative scaling and coordinate descent methods for maximum entropy models maximum entropy maxent is useful in natural language processing and many other areas iterative scaling is methods are one of the most popular approaches to solve maxent with many variants of is methods it is difficult to understand them and see the differences in this paper we create a general and unified framework for iterative scaling methods this framework also connects iterative scaling and coordinate descent methods we prove general convergence results for is methods and analyze their computational complexity based on the proposed framework we extend a coordinate descent method for linear svm to maxent results show that it is faster than existing iterative scaling methods

periodic string response to an impact and a suddenly applied concentrated stationary force an infinite periodic structure unsteady response to a forced excitation is considered any forced excitation can be presented as a sequence or a distribution of impulses the instantaneous impulse is an infinite sum of harmonic forces of the same amplitude and phase whose frequencies fill the infinite band as follows from the fourier transformation of the dirac delta function the solution to the problem of a periodic structure steady state response to excitation by a harmonic concentrated stationary force is obtained by reducing the problem to a difference equation and used to calculate the unsteady response the infinite periodic structure consisting of an infinite stretched string supported by equidistantly spaced identical suspensions is considered each suspension consists of a spring and a dashpot with viscous damping in parallel small transverse oscillations of the string without bending stiffness are considered in order to exclude from the solution the string slope which experience a sudden change at the point where a concentrated force is applied and so at every suspension point the boundary problem is solved over the string unloaded span the string transverse deflection at an arbitrary point of the span as well as its slope are expressed via the deflection values at the beginning and at the end of the span then two neighbour spans are considered together a suspension reaction that depends on the suspension point deflection is connected with the string slopes to the right and to the left of the point the connection involves the string transverse deflection at three successive suspension points and represents the second order difference equations the solution to this equation allows one to express the string deflections at all suspension points via only two which relate to ends of the span where the excitation is applied consideration of this span leads to calculation of the string steady state oscillations and response to an impact to give an example of application the latter is used to calculate the string response to a suddenly applied force

computing maximally separated sets in the plane let s be a set of n points in reals 2 given an integer 1 le k le n we wish to find a maximally separated subset i subseteq s of size k this is a subset for which the minimum among the k choose 2 pairwise distances between its points is as large as possible the decision problem associated with this problem is to determine whether there exists i subseteq s i k so that all k choose 2 pairwise distances in i are at least 2 this problem can also be formulated in terms of disk intersection graphs let d be the set of unit disks centered at the points of s the disk intersection graph g of d has as edges all pairs of disks with nonempty intersection any set i with the above properties is then the set of centers of disks that form an independent set in the graph g this problem is known to be np complete if k is part of the input in this paper we first present a linear time eps approximation algorithm for any constant k next we give exact algorithms for the cases k 3 and k 4 that run in time o n 4 3 polylog n we also present a simpler n o sqrt k time exact algorithm as compared with the recent algorithm in j alber and j fiala j algorithms 52 2004 pp 134 151 for arbitrary values of k

parameter and state model reduction for large scale statistical inverse problems a greedy algorithm for the construction of a reduced model with reduction in both parameter and state is developed for an efficient solution of statistical inverse problems governed by partial differential equations with distributed parameters large scale models are too costly to evaluate repeatedly as is required in the statistical setting furthermore these models often have high dimensional parametric input spaces which compounds the difficulty of effectively exploring the uncertainty space we simultaneously address both challenges by constructing a projection based reduced model that accepts low dimensional parameter inputs and whose model evaluations are inexpensive the associated parameter and state bases are obtained through a greedy procedure that targets the governing equations model outputs and prior information the methodology and results are presented for groundwater inverse problems in one and two dimensions

approximate polytope membership queries in the polytope membership problem a convex polytope k in mathbb r d is given and the objective is to preprocess k into a data structure so that given any query point q in mathbb r d it is possible to determine efficiently whether q in k we consider this problem in an approximate setting given an approximation parameter varepsilon the query can be answered either way if the distance from q to k s boundary is at most varepsilon times k s diameter we assume that the dimension d is fixed and k is presented as the intersection of n halfspaces previous solutions to approximate polytope membership were based on straightforward applications of classic polytope approximation techniques by dudley approx theory 10 1974 pp 227 236 and bentley faust and preparata commun acm 25 1982 pp 64 68 the former is optimal in the worst case with respect to space and the latter is optimal with respect to query time we present four main results first we show how

streaming with minimum space an algorithm for covering by two congruent balls in this paper we design a simple streaming algorithm for maintaining two smallest balls of equal radius in d dimension to cover a set of points in an on line fashion different from most of the traditional streaming models at any step we use the minimum amount of space by only storing the locations and the common radius of the balls previously such a geometric algorithm is only investigated for covering with one ball one center by zarrabi zadeh and chan 2006 16 we give an analysis of our algorithm which is significantly different from the one center algorithm due to the obvious possibility of grouping points wrongly under this streaming model we show that our algorithm has an approximation ratio 2 for d 1 and at most 5 611 for any fixed d 1 we also present lower bounds of 1 5 and 1 604 for the problem in the d 1 and d 1 cases respectively

on the limited memory bfgs method for large scale optimization we study the numerical performance of a limited memory quasi newton method for large scale optimization which we call the l bfgs method we compare its performance with that of the method developed by buckley and lenir 1985 which combines cycles of bfgs steps and conjugate direction steps our numerical tests indicate that the l bfgs method is faster than the method of buckley and lenir and is better able to use additional storage to accelerate convergence we show that the l bfgs method can be greatly accelerated by means of a simple scaling we then compare the l bfgs method with the partitioned quasi newton method of griewank and toint 1982a the results show that for some problems the partitioned quasi newton method is clearly superior to the l bfgs method however we find that for other problems the l bfgs method is very competitive due to its low iteration cost we also study the convergence properties of the l bfgs method and prove global convergence on uniformly convex problems

qn like variable storage conjugate gradients both conjugate gradient and quasi newton methods are quite successful at minimizing smooth nonlinear functions of several variables and each has its advantages in particular conjugate gradient methods require much less storage to implement than a quasi newton code and therefore find application when storage limitations occur they are however slower so there have recently been attempts to combine cg and qn algorithms so as to obtain an algorithm with good convergence properties and low storage requirements one such method is the code conmin due to shanno and phua it has proven quite successful but it has one limitation it has no middle ground in that it either operates as a quasi newton code using o n 2 storage locations or as a conjugate gradient code using 7n locations but it cannot take advantage of the not unusual situation where more than 7n locations are available but a quasi newton code requires an excessive amount of storage

articulated and generalized gaussian kernel correlation for human pose estimation in this paper we propose an articulated and generalized gaussian kernel correlation gkc based framework for human pose estimation we first derive a unified gkc representation that generalizes the previous sum of gaussians sog based methods for the similarity measure between a template and an observation both of which are represented by various sog variants then we develop an articulated gkc agkc by integrating a kinematic skeleton in a multivariate sog template that supports subject specific shape modeling and articulated pose estimation for both the full body and the hands we further propose a sequential body hand pose tracking algorithm by incorporating three regularization terms in the agkc function including visibility intersection penalty and pose continuity our tracking algorithm is simple yet effective and computationally efficient we evaluate our algorithm on two benchmark depth data sets the experimental results are promising and competitive when compared with the state of the art algorithms

a new implementation of the lanczos method in linear problems the lanczos algorithm has proved to be a powerful solution method not only for finding the eigenvalues but for solving linear systems of equations in this work a new implementation of the algorithm is presented for solving linear systems of equations with a sequence of right hand sides the versions of the method proposed in the past treat the right hand side vectors successively by keeping the tridiagonal matrix and the orthonormal basis in fast or secondary storage the new technique handles all approximations to the solution vectors simultaneously without the necessity for keeping the tridiagonal matrix or the orthonormal basis in fast or secondary storage thus when the first solution vector has converged to a required accuracy good approximations to the remaining solution vectors have simultaneously been obtained it then takes fewer iterations to reach the final accuracy by working separately on each of the remaining vectors

on the smoothness constraints for four dimensional data assimilation an algorithm for determination of the weights of smoothness penalty constraints for the four dimensional variational data assimilation technique is proposed and evaluated to study the nature of smoothness penalty constraints a simple nonlinear harmonic oscillator problem is first considered penalizing smoothness constraints is found to make the modified hessian matrix of the cost function more positive definite akin to the idea behind the modified line search newton s methods however the use of the derivative smoothness constraints with a fixed coefficient does not warrant uniform imposition of these constraints at every iteration a remedy is to control the ratio of the smoothness penalty function over the cost function which can dramatically increase the positive definite area on the other hand the large smoothness coefficients obtained from this approach can deteriorate the convergence property of the minimization problem based on these observations an algorithm for tuning the weights of smoothness constraints is proposed to overcome the aforementioned problems the algorithm is first applied to a simple dynamic problem it is then tested on the retrieval of microscale turbulent structures in a simulated convective boundary layer this method is further evaluated on the retrieval of a strong meso scale thunderstorm outflow from doppler radar data the results show that the algorithm yields efficient retrieval

trust region newton method for logistic regression large scale logistic regression arises in many applications such as document classification and natural language processing in this paper we apply a trust region newton method to maximize the log likelihood of the logistic regression model the proposed method uses only approximate newton steps in the beginning but achieves fast convergence in the end experiments show that it is faster than the commonly used quasi newton approach for logistic regression we also extend the proposed method to large scale l2 loss linear support vector machines svm

an active set quasi newton method with projected search for bound constrained minimization we analyze an active set quasi newton method for large scale bound constrained problems our approach combines the accurate active set identification function and the projected search both of these strategies permit fast change in the working set the limited memory method is employed to update the inactive variables while the active variables are updated by simple rules a further division of the active set enables the global convergence of the new algorithm numerical tests demonstrate the efficiency and performance of the present strategy and its comparison with some existing active set strategies

computing a minimum width square annulus in arbitrary orientation in this paper we address the problem of computing a minimum width square annulus in arbitrary orientation that encloses a given set of n points in the plane a square annulus is the region between two concentric squares we present an o n 3 log n time algorithm that finds such a square annulus over all orientations

a discrete newton algorithm for minimizing a function of many variables a newton like method is presented for minimizing a function ofn variables it uses only function and gradient values and is a variant of the discrete newton algorithm this variant requires fewer operations than the standard method whenn 39 and storage is proportional ton rather thann2

an optimal positive definite update for sparse hessian matrices a hessian update is described that preserves sparsity and positive definiteness and satisfies a minimal change property the update reduces to the bfgs update in the dense case and generalises a recent result in siam j nnmer anal 26 1989 pp 727 739 relating to the byrd and nocedal measure function a surprising outcome is that a sparsity projection of the inverse hessian plays a major role it is shown that the hessian itself can be recovered from this information under mild assumptions the update is computed by solving a concave programming problem derived by using the wolfe dual the hessian of the dual is important and plays a similar role to the matrix q that arises in the sparse psb update of toint math comp 31 1977 pp 954 961 this matrix is shown to satisfy the same structural and definiteness conditions as toint s matrix the update has been implemented for tridiagonal systems and some numerical experiments are described these experiments indicate that there is potential for a

a nonmonotone line search technique and its application to unconstrained optimization a new nonmonotone line search algorithm is proposed and analyzed in our scheme we require that an average of the successive function values decreases while the traditional nonmonotone approach of grippo lampariello and lucidi siam j numer anal 23 1986 pp 707 716 requires that a maximum of recent function values decreases we prove global convergence for nonconvex smooth functions and r linear convergence for strongly convex functions for the l bfgs method and the unconstrained optimization problems in the cute library the new nonmonotone line search algorithm used fewer function and gradient evaluations on average than either the monotone or the traditional nonmonotone scheme

renewable estimation and incremental inference in generalized linear models with streaming data sets the paper presents an incremental updating algorithm to analyse streaming data sets using generalized linear models the method proposed is formulated within a new framework of renewable estimation and incremental inference in which the maximum likelihood estimator is renewed with current data and summary statistics of historical data our framework can be implemented within a popular distributed computing environment known as apache spark to scale up computation consisting of two data processing layers the rho architecture enables us to accommodate inference related statistics and to facilitate sequential updating of the statistics used in both estimation and inference we establish estimation consistency and asymptotic normality of the proposed renewable estimator in which the wald test is utilized for an incremental inference our methods are examined and illustrated by various numerical examples from both simulation experiments and a real world data analysis

a truncated newton lanczos method for overcoming limit and bifurcation points in this study procedures for overcoming limit and bifurcation points in large scale structural analysis problems are described and evaluated the methods are based on newton s method for the outer iterations while for the linearized problem in each iteration the preconditioned truncated lanczos method is employed special care is placed upon line search routines for accelerating the convergence properties and enhancing the stability of the outer method the proposed methodology retains all characteristics of an iterative method by avoiding the factorization of the current stiffness matrix the necessary eigenvalue information is retained in the tridiagonal matrix of the lanczos approach

learning harmonium models with infinite latent features undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks one common challenge in learning such models is to determine the number of hidden units that are unknown a priori although bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed bayesian networks very little effort has been made toward applying bayesian nonparametrics to learn undirected latent variable models in this paper we present the infinite exponential family harmonium iefh a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool we also present two important extensions of iefh to 1 multiview iefh for dealing with heterogeneous data and 2 infinite maximum margin harmonium immh for incorporating supervising side information to learn predictive latent features we develop variational inference algorithms to learn model parameters our methods are computationally competitive because of the avoidance of selecting the number of latent units our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iefh and immh inherited from bayesian nonparametrics and max margin learning such results were not available until now and contribute to expanding the scope of bayesian nonparametrics to learn the structures of undirected latent variable models