a time parallel approach to strong constraint four dimensional variational data assimilation a parallel in time algorithm based on an augmented lagrangian approach is proposed to solve four dimensional variational 4d var data assimilation problems the assimilation window is divided into multiple sub intervals that allows parallelization of cost function and gradient computations the solutions to the continuity equations across interval boundaries are added as constraints the augmented lagrangian approach leads to a different formulation of the variational data assimilation problem than the weakly constrained 4d var a combination of serial and parallel 4d vars to increase performance is also explored the methodology is illustrated on data assimilation problems involving the lorenz 96 and the shallow water models

iterative scaling and coordinate descent methods for maximum entropy models maximum entropy maxent is useful in natural language processing and many other areas iterative scaling is methods are one of the most popular approaches to solve maxent with many variants of is methods it is difficult to understand them and see the differences in this paper we create a general and unified framework for iterative scaling methods this framework also connects iterative scaling and coordinate descent methods we prove general convergence results for is methods and analyze their computational complexity based on the proposed framework we extend a coordinate descent method for linear svm to maxent results show that it is faster than existing iterative scaling methods

cohomology of infinitesimal and discrete groups an infinitesimal algebraic group over a field k is a connected algebraic group scheme whose coordinate ring is a finite dimension al k algebra if one begins with a reductive algebraic group g defined over fp then one has a close relationship between the representation theory of g that of its infinitesimal subgroups g defined as the kernels of the frobenius endomorphisms rr g g and that of its finite chevalley subgroups g fq of fq rational points q pd a basic theorem of cline et al i 9 theorem 6 6 relates the cohomology of g in a rational representation to that of the finite groups g fq related work of cline et al 6 and of the authors 1 16 relates the cohomology of g to that of its infinitesimal subgroups a determination of the cohomology of the first infinitesimal subgroup g 1 of g with trivial coefficients has enabled the authors in 14 to make certain explicit cohomology calculations for the finite groups g fq with certain nontrivial coefficients in this paper we continue our study of infinitesimal subgroups of simple algebraic groups by proving various qualitative properties of their cohomology and by extending our computational knowledge in particular we provide a computation which applies in a range of degrees to chevalley groups over the prime field we also investigate the cohomology of infinitesimal subgroups of nilpotent groups enabling us for example to determine the cohomology of their lie algebras in certain cases we apply this lie algebra cohomology computation to determine the integral cohomology of various torsion free nilpotent groups modulo small prime torsion thereby extending a theorem of dwyer 12 in more detail the paper begins in sect 1 with a spectral sequence converging to the cohomology of the first infinitesimal subgroup g which was previously studied in 14 from this spectral sequence it easily follows that h g1 m is a noetherian h g1 k module for each finite dimensional rational gl module m we then apply the work of andersen and jantzen 1 on the gl cohomology of induced modules to obtain the analogous result for the second infinitesimal subgroup gz our analysis also permits a determination of the generic

the limited memory conjugate gradient method in theory the successive gradients generated by the conjugate gradient method applied to a quadratic should be orthogonal however for some ill conditioned problems orthogonality is quickly lost due to rounding errors and convergence is much slower than expected a limited memory version of the nonlinear conjugate gradient method is developed the memory is used to both detect the loss of orthogonality and to restore orthogonality an implementation of the algorithm is presented based on the cg descent nonlinear conjugate gradient method limited memory cg descent l cg descent possesses a global convergence property similar to that of the memoryless algorithm but has much better practical performance numerical comparisons to the limited memory bfgs method l bfgs are given using the cuter test problems

limited memory bfgs systems with diagonal updates abstract in this paper we investigate a formula to solve systems of the form b i x y where b is a limited memory bfgs quasi newton matrix and is a positive constant these types of systems arise naturally in large scale optimization such as trust region methods as well as doubly augmented lagrangian methods we show that provided a simple condition holds on b 0 and the system b i x y can be solved via a recursion formula that requires only vector inner products this formula has complexity m 2 n where m is the number of l bfgs updates and n m is the dimension of x

identification of boundary conditions in a nonlinear shallow water flow abstract in this paper we present sufficient smoothness results on the solution of a nonlinear shallow water problem in order to obtain an existence result for the associated adjoint problem we introduce a necessary but not sufficient existence condition for the functional minimum the control is made on the velocity boundary condition in considering observations at isolated points moreover we give a set of numerical results obtained in a real situation

a tool for the analysis of quasi newton methods with application to unconstrained minimization the bfgs update formula is shown to have an important property that is inde pendent of the algorithmic context of the update and that is relevant to both constrained and unconstrained optimization the bfgs method for unconstrained optimization using a variety of line searches including backtracking is shown to be globally and superlinearly convergent on uniformly convex problems the analysis is particularly simple due to the use of some new tools introduced in this paper

possible spectra of totally positive matrices abstract for any given set s of n distinct positive numbers we construct a symmetric n by n strictly totally positive matrix whose spectrum is s thus in order to be the spectrum of an n by n totally positive matrix it is necessary and sufficient that n numbers be positive and distinct

a trace minimization algorithm for the generalized eigenvalue problem an algorithm for computing a few of the smallest or largest eigenvalues and associated eigenvectors of the large sparse generalized eigenvalue problem ax lambda bx is presented the matrices a and b are assumed to be symmetric and haphazardly sparse with b being positive definite the problem is treated as one of constrained optimization and an inverse iteration is developed which requires the solution of linear algebraic systems only to the accuracy demanded by a given subspace the rate of convergence of the method is established and a technique for improving it is discussed numerical experiments and comparisons with other methods are presented

a framework of constraint preserving update schemes for optimization on stiefel manifold this paper considers optimization problems on the stiefel manifold x mathsf t x i p xtx ip where x in mathbb r n times p x rn p is the variable and i p ip is the p p by p p identity matrix a framework of constraint preserving update schemes is proposed by decomposing each feasible point into the range space of x x and the null space of x mathsf t xt while this general framework can unify many existing schemes a new update scheme with low complexity cost is also discovered then we study a feasible barzilai borwein like method under the new update scheme the global convergence of the method is established with an adaptive nonmonotone line search the numerical tests on the nearest low rank correlation matrix problem the kohn sham total energy minimization and a specific problem from statistics demonstrate the efficiency of the new method in particular the new method performs remarkably well for the nearest low rank correlation matrix problem in terms of speed and solution quality and is considerably competitive with the widely used scf iteration for the kohn sham total energy minimization

accelerated block preconditioned gradient method for large scale wave functions calculations in density functional theory an accelerated block preconditioned gradient abpg method is proposed to solve electronic structure problems in density functional theory this iterative algorithm is designed to solve directly the non linear kohn sham equations for accurate discretization schemes involving a large number of degrees of freedom it makes use of an acceleration scheme similar to what is known as rmm diis in the electronic structure community the method is illustrated with examples of convergence for large scale applications using a finite difference discretization and multigrid preconditioning

action recognition based on overcomplete independent components analysis existing works on action recognition rely on two separate stages 1 designing hand designed features or learning features from video data 2 classifying features using a classifier such as svm or adaboost motivated by two observations 1 independent component analysis ica is capable of encoding intrinsic features underlying video data and 2 videos of different actions can be easily distinguished by their intrinsic features we propose a simple but effective action recognition framework based on the recently proposed overcomplete ica model after a set of overcomplete ica basis functions are learned from the densely sampled 3d patches from training videos for each action a test video is classified as the class whose basis functions can reconstruct the sampled 3d patches from the test video with the smallest reconstruction error the experimental results on five benchmark datasets demonstrate that the proposed approach outperforms several state of the art works

a proximal gradient method for ensemble density functional theory the ensemble density functional theory e dft is valuable for simulations of metallic systems due to the absence of a gap in the spectrum of the hamiltonian matrices although the widely used self consistent field scf iteration method can be extended to solve the minimization of the total energy functional with respect to orthogonality constraints there is no theoretical guarantee on the convergence of these algorithms in this paper we consider an equivalent model with a single variable and a single spherical constraint by eliminating the dependence on the fractional occupancies a proximal gradient method is developed by keeping the entropy term but linearizing all other terms in the total energy functional convergence to the stationary point is established numerical results using the matlab toolbox kssolv can outperform scf consistently on many metallic systems

some numerical experiments with variable storage quasi newton algorithms this paper describes some numerical experiments with variable storage quasi newton methods for the optimization of some large scale models coming from fluid mechanics and molecular biology in addition to assessing these kinds of methods in real life situations we compare an algorithm of a buckley with a proposal by j nocedal the latter seems generally superior provided that careful attention is given to some nontrivial implementation aspects which concern the general question of properly initializing a quasi newton matrix in this context we find it appropriate to use a diagonal matrix generated by an update of the identity matrix so as to fit the rayleigh ellipsoid of the local hessian in the direction of the change in the gradient also a variational derivation of some rank one and rank two updates in hilbert spaces is given

modeling textured images using generalized long correlation models the long correlation lc models are a general class of random field rf models which are able to model correlations extending over large image regions with few model parameters the lc models have seen limited use due to lack of an effective method for estimating the model parameters in this work we develop an estimation scheme for a very general form of this model and demonstrate its applicability to texture modeling applications the relationship of the generalized lc models to other classes of rf models namely the simultaneous autoregressive sar and markov random field mrf models is shown while it is known that the sar model is a special case of the lc model we show that the mrf model is also encompassed by this model consequently the lc model may be considered as a generalization of the sar and mrf models

gradient type optimization methods for electronic structure calculations the density functional theory dft in electronic structure calculations can be formulated as either a nonlinear eigenvalue or a direct minimization problem the most widely used approach for solving the former is the so called self consistent field scf iteration a common observation is that the convergence of scf is not clear theoretically while approaches with convergence guarantees for solving the latter are often not competitive with scf numerically in this paper we study gradient type methods for solving the direct minimization problem by constructing new iterations along the gradient on the stiefel manifold global convergence rate i e convergence to a stationary point from any initial solution as well as local convergence rate can be obtained from the standard theory for gradient methods on manifold directly a major computational advantage is that the computation of linear eigenvalue problems is no longer needed the main costs of our approaches arise from the assembling of the total ene

cohomology of nilpotent groups of class 2 abstract we compute the cohomology rings of a number of nilpotent groups of class 2 for appropriate coefficients and we do some more sample calculations of various cohomology groups the basic tool is a suitable small free resolution for an arbitrary nilpotent group of class 2 constructed in a previous paper

a new conjugate gradient method with guaranteed descent and an efficient line search a new nonlinear conjugate gradient method and an associated implementation based on an inexact line search are proposed and analyzed with exact line search our method reduces to a nonlinear version of the hestenes stiefel conjugate gradient scheme for any inexact line search our scheme satisfies the descent condition bf g k sf t bf d k le frac 7 8 bf g k 2 moreover a global convergence result is established when the line search fulfills the wolfe conditions a new line search scheme is developed that is efficient and highly accurate efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer high accuracy is achieved by using a convergence criterion which we call the approximate wolfe conditions obtained by replacing the sufficient decrease criterion in the wolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion numerical comparisons are given with both l bfgs and conjugate gradient methods using the unconstrained optimization problems in the cute library

a algorithm for convex minimization for convex minimization we introduce an algorithm based on space decomposition the method uses a bundle subroutine to generate a sequence of approximate proximal points when a primal dual track leading to a solution and zero subgradient pair exists these points approximate the primal track points and give the algorithm s or corrector steps the subroutine also approximates dual track points that are gradients needed for the method s newton predictor steps with the inclusion of a simple line search the resulting algorithm is proved to be globally convergent the convergence is superlinear if the primal dual track points and the objective s hessian are approximated well enough

on the use of stochastic hessian information in optimization methods for machine learning this paper describes how to incorporate sampled curvature information in a newton cg method and in a limited memory quasi newton method for statistical learning the motivation for this work stems from supervised machine learning applications involving a very large number of training points we follow a batch approach also known in the stochastic optimization literature as a sample average approximation approach curvature information is incorporated in two subsampled hessian algorithms one based on a matrix free inexact newton iteration and one on a preconditioned limited memory bfgs iteration a crucial feature of our technique is that hessian vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient the efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition

local search based heuristics for global optimization atomic clusters and beyond abstract finding good solutions to large scale hard global optimization problems is a demanding task with many relevant applications it is well known that in order to tackle a difficult problem an algorithm has to incorporate all of the available information on the problem domain however as we will show in this paper some general purpose methods and the ideas on which they are built can provide guidance towards the efficient solution of difficult instances most of this paper will be devoted to heuristic techniques applied to the problem of finding a minimum energy configuration of a cluster of atoms in r 3 this is a very relevant problem with a large set of applications which has triggered considerable research efforts in the last decade we will show how some relatively simple ideas can be used to generate fairly efficient methods which have been employed to discover many new cluster structures in this paper we will introduce some of the main algorithmic ideas which have proven to be particularly successful in the field of global optimization applied to atomic cluster conformation problems we will mainly discuss basin hopping methods as well as their population based variant and some specific technique based on direct moves these methods although designed for the specific problem have a much wider applicability in fact one of the aims of this paper is also that of suggesting that similar ideas can be employed in order to design innovative methods even for totally different global optimization problems like e g circle packing and space trajectory planning

parameter and state model reduction for large scale statistical inverse problems a greedy algorithm for the construction of a reduced model with reduction in both parameter and state is developed for an efficient solution of statistical inverse problems governed by partial differential equations with distributed parameters large scale models are too costly to evaluate repeatedly as is required in the statistical setting furthermore these models often have high dimensional parametric input spaces which compounds the difficulty of effectively exploring the uncertainty space we simultaneously address both challenges by constructing a projection based reduced model that accepts low dimensional parameter inputs and whose model evaluations are inexpensive the associated parameter and state bases are obtained through a greedy procedure that targets the governing equations model outputs and prior information the methodology and results are presented for groundwater inverse problems in one and two dimensions

a practical pr conjugate gradient method only using gradient abstract in this paper we consider the polak ribiere or polak ribiere plus conjugate gradient method for solving optimality condition of an unconstrained minimization problem we give two new steplength rules only using gradient and under gradient lipschitz assumption prove this method s global convergence correspondingly then we develop a practical polak ribiere plus method whose steplength is located by one inequality only using gradient and report promising numerical results on high accuracy solution for some standard test problems when compared to the state of art methods in this research direction importantly our work provides a new idea of devising a practical version of the celebrated polak ribiere or polak ribiere plus method

adjoint implementation of rosenbrock methods applied to variational data assimilation problems abstract in the past decade the variational method has been successfully applied in data assimilation problems for atmospheric chemistry models in 4d var data assimilation a minimization algorithm is used to find the set of control variables which minimizes the weighted least squares distance between model predictions and observations over the assimilation window using the adjoint method the gradient of the cost function can be computed fast at the expense of few function evaluations making the optimization process very efficient for large scale models the high storage requirements and the difficulty of implementing the adjoint code when sophisticated integrators are used to solve the stiff chemistry make the assimilation a very intensive computational process if the sparse structure of the chemical models is carefully exploited rosenbrock methods have been proved to be reliable chemistry solvers because of their outstanding stability properties and conservation of the linear invariants of the system in this paper we present an efficient implementation of the adjoint code for the rosenbrock type methods which can reduce the storage requirements of the forward model and is suitable for automatization the adjoint code is completely generated using symbolic preprocessing and automatic differentiation tools which allow flexibility and require minimal user intervention

mixing linear svms for nonlinear classification in this paper we address the problem of combining linear support vector machines svms for classification of large scale nonlinear datasets the motivation is to exploit both the efficiency of linear svms lsvms in learning and prediction and the power of nonlinear svms in classification to this end we develop a lsvm mixture model that exploits a divide and conquer strategy by partitioning the feature space into subregions of linearly separable datapoints and learning a lsvm for each of these regions we do this implicitly by deriving a generative model over the joint data and label distributions consequently we can impose priors on the mixing coefficients and do implicit model selection in a top down manner during the parameter estimation process this guarantees the sparsity of the learned model experimental results show that the proposed method can achieve the efficiency of lsvms in the prediction phase while still providing a classification performance comparable to nonlinear svms

a trust region direct constrained minimization algorithm for the kohn sham equation the self consistent field scf iteration widely used for computing the ground state energy and the corresponding single particle wave functions associated with a many electron atomistic system is viewed in this paper as an optimization procedure that minimizes the kohn sham ks total energy indirectly by minimizing a sequence of quadratic surrogate functions we point out the similarity and difference between the total energy and the surrogate and show how the scf iteration can fail when the minimizer of the surrogate produces an increase in the ks total energy a trust region technique is introduced as a way to restrict the update of the wave functions within a small neighborhood of an approximate solution at which the gradient of the total energy agrees with that of the surrogate the use of trust regions in scf is not new however it has been observed that directly applying a trust region based scf trscf to the ks total energy often leads to slow convergence we propose to use trscf within a direct constrained minimization dcm algorithm we developed in j comput phys 217 2006 pp 709 721 the key ingredients of the dcm algorithm involve projecting the total energy function into a sequence of subspaces of small dimensions and seeking the minimizer of the total energy function within each subspace the minimizer of a subspace energy function which is computed by the trscf not only provides a search direction along which the ks total energy function decreases but also gives an optimal step length that yields a sufficient decrease in total energy a numerical example is provided to demonstrate that the combination of trscf and dcm is more efficient than scf

block locally optimal preconditioned eigenvalue xolvers blopex in hypre and petsc we describe our software package block locally optimal preconditioned eigenvalue xolvers blopex recently publicly released blopex is available as a stand alone serial library as an external package to petsc portable extensible toolkit for scientific computation a general purpose suite of tools developed by argonne national laboratory for the scalable solution of partial differential equations and related problems and is also built into hypre high performance preconditioners a scalable linear solvers package developed by lawrence livermore national laboratory the present blopex release includes only one solver the locally optimal block preconditioned conjugate gradient lobpcg method for symmetric eigenvalue problems hypre provides users with advanced high quality parallel multigrid preconditioners for linear systems with blopex the same preconditioners can now be efficiently used for symmetric eigenvalue problems petsc facilitates the integration of independently developed application modules with strict attention to component interoperability and makes blopex extremely easy to compile and use with preconditioners that are available via petsc we present the lobpcg algorithm in blopex for hypre and petsc we demonstrate numerically the scalability of blopex by testing it on a number of distributed and shared memory parallel systems including a beowulf system sun fire 880 an amd dual core opteron workstation and ibm bluegene l supercomputer using petsc domain decomposition and hypre multigrid preconditioning we test blopex on a model problem the standard 7 point finite difference approximation of the 3 d laplacian with the problem size in the range of 10 5 10 8

unconstrained energy functionals for electronic structure calculations the performance of conjugate gradient schemes for minimizing unconstrained energy functionals in the context of condensed matter electronic structure density functional calculations is studied the unconstrained functionals allow a straightforward application of conjugate gradients by removing the explicit orthonormality constraints on the quantum mechanical wave functions however the removal of the constraints can lead to slow convergence in particular when preconditioning is used the convergence properties of two previously suggested energy functionals are analyzed and a new functional is proposed which unifies some of the advantages of the other functionals a numerical example derived from a diamond crystal confirms the analysis

numerical methods for electronic structure calculations of materials the goal of this article is to give an overview of numerical problems encountered when determining the electronic structure of materials and the rich variety of techniques used to solve these problems the paper is intended for a diverse scientific computing audience for this reason we assume the reader does not have an extensive background in the related physics our overview focuses on the nature of the numerical problems to be solved their origin and the methods used to solve the resulting linear algebra or nonlinear optimization problems it is common knowledge that the behavior of matter at the nanoscale is in principle entirely determined by the schrodinger equation in practice this equation in its original form is not tractable successful but approximate versions of this equation which allow one to study nontrivial systems took about five or six decades to develop in particular the last two decades saw a flurry of activity in developing effective software one of the main practical variants of the schrodinger equation is based on what is referred to as density functional theory dft the combination of dft with pseudopotentials allows one to obtain in an efficient way the ground state configuration for many materials this article will emphasize pseudopotential density functional theory but other techniques will be discussed as well

on the limited memory bfgs method for large scale optimization we study the numerical performance of a limited memory quasi newton method for large scale optimization which we call the l bfgs method we compare its performance with that of the method developed by buckley and lenir 1985 which combines cycles of bfgs steps and conjugate direction steps our numerical tests indicate that the l bfgs method is faster than the method of buckley and lenir and is better able to use additional storage to accelerate convergence we show that the l bfgs method can be greatly accelerated by means of a simple scaling we then compare the l bfgs method with the partitioned quasi newton method of griewank and toint 1982a the results show that for some problems the partitioned quasi newton method is clearly superior to the l bfgs method however we find that for other problems the l bfgs method is very competitive due to its low iteration cost we also study the convergence properties of the l bfgs method and prove global convergence on uniformly convex problems

qn like variable storage conjugate gradients both conjugate gradient and quasi newton methods are quite successful at minimizing smooth nonlinear functions of several variables and each has its advantages in particular conjugate gradient methods require much less storage to implement than a quasi newton code and therefore find application when storage limitations occur they are however slower so there have recently been attempts to combine cg and qn algorithms so as to obtain an algorithm with good convergence properties and low storage requirements one such method is the code conmin due to shanno and phua it has proven quite successful but it has one limitation it has no middle ground in that it either operates as a quasi newton code using o n 2 storage locations or as a conjugate gradient code using 7n locations but it cannot take advantage of the not unusual situation where more than 7n locations are available but a quasi newton code requires an excessive amount of storage

accelerated scaled memoryless bfgs preconditioned conjugate gradient algorithm for unconstrained optimization abstract an accelerated scaled memoryless bfgs preconditioned conjugate gradient algorithm for solving unconstrained optimization problems is presented the basic idea is to combine the scaled memoryless bfgs method and the preconditioning technique in the frame of the conjugate gradient method the preconditioner which is also a scaled memoryless bfgs matrix is reset when the beale powell restart criterion holds the parameter scaling the gradient is selected as a spectral gradient for the steplength computation the method has the advantage that in conjugate gradient algorithms the step lengths may differ from 1 by two order of magnitude and tend to vary unpredictably thus we suggest an acceleration scheme able to improve the efficiency of the algorithm under common assumptions the method is proved to be globally convergent it is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear but the reduction in the function values is significantly improved in mild conditions the algorithm is globally convergent for strongly convex functions computational results for a set consisting of 750 unconstrained optimization test problems show that this new accelerated scaled conjugate gradient algorithm substantially outperforms known conjugate gradient methods scalcg 3 4 5 6 conmin by shanno and phua 1976 1978 42 43 hestenes and stiefel 1952 25 polak ribiere polyak 1969 32 33 dai and yuan 2001 17 dai and liao 2001 t 1 14 conjugate gradient with sufficient descent condition 7 hybrid dai and yuan 2001 17 hybrid dai and yuan zero 2001 17 cg descent by hager and zhang 2005 2006 22 23 as well as quasi newton lbfgs method 26 and truncated newton method by nash 1985 27

articulated and generalized gaussian kernel correlation for human pose estimation in this paper we propose an articulated and generalized gaussian kernel correlation gkc based framework for human pose estimation we first derive a unified gkc representation that generalizes the previous sum of gaussians sog based methods for the similarity measure between a template and an observation both of which are represented by various sog variants then we develop an articulated gkc agkc by integrating a kinematic skeleton in a multivariate sog template that supports subject specific shape modeling and articulated pose estimation for both the full body and the hands we further propose a sequential body hand pose tracking algorithm by incorporating three regularization terms in the agkc function including visibility intersection penalty and pose continuity our tracking algorithm is simple yet effective and computationally efficient we evaluate our algorithm on two benchmark depth data sets the experimental results are promising and competitive when compared with the state of the art algorithms

a constrained optimization algorithm for total energy minimization in electronic structure calculations a new direct constrained optimization algorithm for minimizing the kohn sham ks total energy functional is presented in this paper the key ingredients of this algorithm involve projecting the total energy functional into a sequence of subspaces of small dimensions and seeking the minimizer of total energy functional within each subspace the minimizer of a subspace energy functional not only provides a search direction along which the ks total energy functional decreases but also gives an optimal step length to move along this search direction numerical examples are provided to demonstrate that this new direct constrained optimization algorithm can be more efficient than the self consistent field scf iteration

limited memory reduced hessian methods for large scale unconstrained optimization limited memory bfgs quasi newton methods approximate the hessian matrix of second derivatives by the sum of a diagonal matrix and a fixed number of rank one matrices these methods are particularly effective for large problems in which the approximate hessian cannot be stored explicitly it can be shown that the conventional bfgs method accumulates approximate curvature in a sequence of expanding subspaces this allows an approximate hessian to be represented using a smaller reduced matrix that increases in dimension at each iteration when the number of variables is large this feature may be used to define limited memory reduced hessian methods in which the dimension of the reduced hessian is limited to save storage limited memory reduced hessian methods have the benefit of requiring half the storage of conventional limited memory methods in this paper we propose a particular reduced hessian method with substantial computational advantages compared to previous reduced hessian methods numerical results from a set of unconstrained problems in the cute test collection indicate that our implementation is competitive with the limited memory codes l bfgs and l bfgs b

a new first order algorithmic framework for optimization problems with orthogonality constraints in this paper we consider a class of optimization problems with orthogonality constraints the feasible region of which is called the stiefel manifold our new framework combines a function value reduction step with a correction step different from the existing approaches the function value reduction step of our algorithmic framework searches along the standard euclidean descent directions instead of the vectors in the tangent space of the stiefel manifold and the correction step further reduces the function value and guarantees a symmetric dual variable at the same time we construct two types of algorithms based on this new framework the first type is based on gradient reduction including the gradient reflection gr and the gradient projection gp algorithms the other one adopts a columnwise block coordinate descent cbcd scheme with a novel idea for solving the corresponding cbcd subproblem inexactly we prove that both gr gp with a fixed step size and cbcd belong to our algorithmic framework

discrete second order adjoints in atmospheric chemical transport modeling atmospheric chemical transport models ctms are essential tools for the study of air pollution for environmental policy decisions for the interpretation of observational data and for producing air quality forecasts many air quality studies require sensitivity analyses i e the computation of derivatives of the model output with respect to model parameters the derivatives of a cost functional defined on the model output with respect to a large number of model parameters can be calculated efficiently through adjoint sensitivity analysis while the traditional first order adjoint models give the gradient of the cost functional with respect to parameters second order adjoint models give second derivative information in the form of products between the hessian of the cost functional and a vector representing a perturbation in sensitivity analysis a search direction in optimization an eigenvector etc in this paper we discuss the mathematical foundations of the discrete second order adjoint sensitivity method and present a complete set of computational tools for performing second order sensitivity studies in three dimensional atmospheric ctms the tools include discrete second order adjoints of runge kutta and of rosenbrock time stepping methods for stiff equations together with efficient implementation strategies numerical examples illustrate the use of these computational tools in important applications like sensitivity analysis optimization uncertainty quantification and the calculation of directions of maximal error growth in three dimensional atmospheric ctms

on the smoothness constraints for four dimensional data assimilation an algorithm for determination of the weights of smoothness penalty constraints for the four dimensional variational data assimilation technique is proposed and evaluated to study the nature of smoothness penalty constraints a simple nonlinear harmonic oscillator problem is first considered penalizing smoothness constraints is found to make the modified hessian matrix of the cost function more positive definite akin to the idea behind the modified line search newton s methods however the use of the derivative smoothness constraints with a fixed coefficient does not warrant uniform imposition of these constraints at every iteration a remedy is to control the ratio of the smoothness penalty function over the cost function which can dramatically increase the positive definite area on the other hand the large smoothness coefficients obtained from this approach can deteriorate the convergence property of the minimization problem based on these observations an algorithm for tuning the weights of smoothness constraints is proposed to overcome the aforementioned problems the algorithm is first applied to a simple dynamic problem it is then tested on the retrieval of microscale turbulent structures in a simulated convective boundary layer this method is further evaluated on the retrieval of a strong meso scale thunderstorm outflow from doppler radar data the results show that the algorithm yields efficient retrieval

reduced hessian quasi newton methods for unconstrained optimization quasi newton methods are reliable and efficient on a wide range of problems but they can require many iterations if the problem is ill conditioned or if a poor initial estimate of the hessian is used in this paper we discuss methods designed to be more efficient in these situations all the methods to be considered exploit the fact that quasi newton methods accumulate approximate second derivative information in a sequence of expanding subspaces associated with each of these subspaces is a certain reduced approximate hessian that provides a complete but compact representation of the second derivative information approximated up to that point algorithms that compute an explicit reduced hessian approximation have two important advantages over conventional quasi newton methods first the amount of computation for each iteration is significantly less during the early stages this advantage is increased by forcing the iterates to linger on a manifold whose dimension can be significantly smaller than the subspace in which curvature has been accumulated second approximate curvature along directions that lie off the manifold can be reinitialized as the iterations proceed thereby reducing the influence of a poor initial estimate of the hessian these advantages are illustrated by extensive numerical results from problems in the cute test set our experiments provide strong evidence that reduced hessian quasi newton methods are more efficient and robust than conventional bfgs methods and some recently proposed extensions

machine learning of linear differential equations using gaussian processes this work leverages recent advances in probabilistic machine learning to discover governing equations expressed by parametric linear operators such equations involve but are not limited to ordinary and partial differential integro differential and fractional order operators here gaussian process priors are modified according to the particular form of such operators and are employed to infer parameters of the linear equations from scarce and possibly noisy observations such observations may come from experiments or black box computer simulations as demonstrated in several synthetic examples and a realistic application in functional genomics employ probabilistic machine learning to discover governing equations expressed by parametric linear operators proper placement of gaussian process priors allows one to efficiently infer model parameters via maximum likelihood estimation a general treatment of inverse problems governed by linear operators leading to model discovery from just a handful of noisy measurements

a feasible method for optimization with orthogonality constraints minimization with orthogonality constraints e g x top x i and or spherical constraints e g vert x vert 2 1 has wide applications in polynomial optimization combinatorial optimization eigenvalue problems sparse pca p harmonic flows 1 bit compressive sensing matrix rank minimization etc these problems are difficult because the constraints are not only non convex but numerically expensive to preserve during iterations to deal with these difficulties we apply the cayley transform a crank nicolson like update scheme to preserve the constraints and based on it develop curvilinear search algorithms with lower flops compared to those based on projections and geodesics the efficiency of the proposed algorithms is demonstrated on a variety of test problems in particular for the maxcut problem it exactly solves a decomposition formulation for the sdp relaxation for polynomial optimization nearest correlation matrix estimation and extreme eigenvalue problems the proposed algorithms run very fast and return solutions no worse than those from their state of the art algorithms for the quadratic assignment problem a gap 0 842 to the best known solution on the largest problem tai256c in qaplib can be reached in 5 min on a typical laptop

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

trust region newton method for logistic regression large scale logistic regression arises in many applications such as document classification and natural language processing in this paper we apply a trust region newton method to maximize the log likelihood of the logistic regression model the proposed method uses only approximate newton steps in the beginning but achieves fast convergence in the end experiments show that it is faster than the commonly used quasi newton approach for logistic regression we also extend the proposed method to large scale l2 loss linear support vector machines svm

iterated tabu search for the circular open dimension problem abstract this paper investigates the circular open dimension problem codp which consists of packing a set of circles of known radii into a strip of fixed width and unlimited length without overlapping the objective is to minimize the length of the strip in this paper codp is solved by a series of sub problems each corresponding to a fixed strip length for each sub problem an iterated tabu search approach named its is proposed its starts from a randomly generated solution and attempts to gain improvements by a tabu search procedure after that if the obtained solution is not feasible a perturbation operator is subsequently employed to reconstruct the incumbent solution and an acceptance criterion is implemented to determine whether or not accept the perturbed solution as a supplementary method the length of the strip is determined in monotonously decreasing way with the aid of some post processing techniques the search terminates and returns the best found solution after the allowed computation time has been elapsed computational experiments based on numerous well known benchmark instances show that its produces quite competitive results with respect to the best known results while the computational time remains reasonable for each instance

orbital minimization method with l 1 regularization abstract we consider a modification of the orbital minimization method omm energy functional which contains an l 1 penalty term in order to find a sparse representation of the low lying eigenspace of self adjoint operators we analyze the local minima of the modified functional as well as the convergence of the modified functional to the original functional algorithms combining soft thresholding with gradient descent are proposed for minimizing this new functional numerical tests validate our approach in addition we also prove the unanticipated and remarkable property that every local minimum of the omm functional without the l 1 term is also a global minimum

fast solution of schrodinger s equation using linear combinations of plane waves abstract a method for solving schrodinger s equation that combines the rayleigh quotient iteration rqi with the conjugate residual cr method is described it is assumed that the eigenfunctions of the hamiltonian operator h are expressed as linear combinations of plane waves which allows the efficient computation of the product h x via fast fourier transform where h is the representation of h in the plane wave basis and x an arbitrary vector the algorithm carries out two kinds of iterations outer loop rqi iterations very few and inner loop cr iterations many more a very small number of orthogonalizations of the set of eigenvectors is required by the proposed method just one per rqi iteration which is one of its key advantages because for large enough systems orthogonalization is the costliest component of the calculation the procedure is much faster than matrix diagonalization and its cost with respect to the size of the plane wave basis n p w has an effective scaling ranging from linear o n p w to quadratic o n p w 2

an active set quasi newton method with projected search for bound constrained minimization we analyze an active set quasi newton method for large scale bound constrained problems our approach combines the accurate active set identification function and the projected search both of these strategies permit fast change in the working set the limited memory method is employed to update the inactive variables while the active variables are updated by simple rules a further division of the active set enables the global convergence of the new algorithm numerical tests demonstrate the efficiency and performance of the present strategy and its comparison with some existing active set strategies

a discrete newton algorithm for minimizing a function of many variables a newton like method is presented for minimizing a function ofn variables it uses only function and gradient values and is a variant of the discrete newton algorithm this variant requires fewer operations than the standard method whenn 39 and storage is proportional ton rather thann2

latent log linear models for handwritten digit classification we present latent log linear models an extension of log linear models incorporating latent variables and we propose two applications thereof log linear mixture models and image deformation aware log linear models the resulting models are fully discriminative can be trained efficiently and the model complexity can be controlled log linear mixture models offer additional flexibility within the log linear modeling framework unlike previous approaches the image deformation aware model directly considers image deformations and allows for a discriminative training of the deformation parameters both are trained using alternating optimization for certain variants convergence to a stationary point is guaranteed and in practice even variants without this guarantee converge and find models that perform well we tune the methods on the usps data set and evaluate on the mnist data set demonstrating the generalization capabilities of our proposed models our models although using significantly fewer parameters are able to obtain competitive results with models proposed in the literature

minimizing the kohn sham total energy for periodic systems abstract we describe how a previously developed constrained minimization algorithm can be adapted to minimize the total energy of a periodic atomistic system under the kohn sham density functional theory framework the algorithm uses the bloch theorem to reduce the complexity of the calculation by working with a number of unit cells separately we present the bloch theorem in terms of linear algebra and point out its implication on the spectral property of the kohn sham hamiltonian

a maximum a posteriori estimation framework for robust high dynamic range video synthesis high dynamic range hdr image synthesis from multiple low dynamic range exposures continues to be actively researched the extension to hdr video synthesis is a topic of significant current interest due to potential cost benefits for hdr video a stiff practical challenge presents itself in the form of accurate correspondence estimation of objects between video frames in particular loss of data resulting from poor exposures and varying intensity makes conventional optical flow methods highly inaccurate we avoid exact correspondence estimation by proposing a statistical approach via maximum a posterior estimation and under appropriate statistical assumptions and choice of priors and models we reduce it to an optimization problem of solving for the foreground and background of the target frame we obtain the background through rank minimization and estimate the foreground via a novel multiscale adaptive kernel regression technique which implicitly captures local structure and temporal motion by solving an unconstrained optimization problem extensive experimental results on both real and synthetic data sets demonstrate that our algorithm is more capable of delivering high quality hdr videos than current state of the art methods under both subjective and objective assessments furthermore a thorough complexity analysis reveals that our algorithm achieves better complexity performance tradeoff than conventional methods

a novel variational algorithmic formulation for wrinkling at finite strains based on energy minimization application to mesh adaption this paper is concerned with an efficient novel algorithmic formulation for wrinkling at finite strains in contrast to previously published numerical implementations the advocated method is fully variational more precisely the parameters describing wrinkles or slacks together with the unknown deformation mapping are computed jointly by minimizing the potential energy of the considered mechanical system furthermore the wrinkling criteria are naturally included within the presented variational framework the proposed method allows to employ three dimensional constitutive models without any additional modification i e a projection in plane stress space is not required analogously to the wrinkling parameters the non vanishing out of plane component of the strain tensor results conveniently from relaxing the respective helmholtz energy of the membrane the proposed framework is very general and does not rely on any assumption regarding the symmetry of the material i e arbitrary anisotropic hyperelastic models can be consistently taken into account the advantages associated with such a variational method are manifold for instance it opens up the possibility of applying standard optimization algorithms to the numerical implementation this is especially important for highly non linear or singular problems such as wrinkling on the other hand minimization principles provide a suitable basis for a posteriori error estimation and thus for adaptive finite element formulations as a prototype a variational error indicator leading to an efficient h adaption for wrinkling is briefly discussed the performance of the wrinkling approach is demonstrated by selected finite element analyses

global convergence properties of the two new dependent fletcher reeves conjugate gradient methods in this paper we propose two new dependent fletcher reeves conjugate gradient methods arising from different choice for the scalar b k we make two different kinds of estimations of upper bounds of b k with respect to b k f r which are based on abel theorem of non convergent series of positive items with several different line searches global convergence results are established for the two new methods which extend the previous dependent fletcher reeves conjugate gradient methods

a nonmonotone line search technique and its application to unconstrained optimization a new nonmonotone line search algorithm is proposed and analyzed in our scheme we require that an average of the successive function values decreases while the traditional nonmonotone approach of grippo lampariello and lucidi siam j numer anal 23 1986 pp 707 716 requires that a maximum of recent function values decreases we prove global convergence for nonconvex smooth functions and r linear convergence for strongly convex functions for the l bfgs method and the unconstrained optimization problems in the cute library the new nonmonotone line search algorithm used fewer function and gradient evaluations on average than either the monotone or the traditional nonmonotone scheme

matrix completion problems abstract throughout the last decades several results have been published in the area of the so called matrix completion problems in this paper we survey several results in this field in particular we describe the possible eigenvalues the characteristic polynomial the invariant polynomials or the number of nontrivial invariant polynomials of a square matrix over a field when some of its entries are prescribed and the others vary finally we present our contribution generalizing some of the previous cases to an n n matrix partitioned into k k blocks with entries in a field when some of its blocks are prescribed and the others vary

principal component analysis combined with truncated newton minimization for dimensionality reduction of chemical databases the similarity and diversity sampling problems are two challenging optimization tasks that arise in the analysis of chemical databases as a first step to their solution we propose an efficient projection refinement protocol based on the principal component analysis pca and the truncated newton minimization method implemented by our package tnpack pca tnpack we show that pca can provide the same initial guess as the singular value decomposition svd for the optimization task of solving the distance geometry optimization problem if each column of a database matrix has a mean of zero hence our pca tnpack approach is analogous to the svd tnpack projection refinement protocol that we developed recently for visualizing large chemical databases using pca tnpack and the merck mddr database mdl drug data report we further investigate the projection refinement procedure with regards to the preservation of the original clusters of chemical compounds the accuracy of similarity and diversity sampling of chemical compounds and the potential application in the study of structure activity relationships we also explore by simple experiments accuracy and efficiency aspects of the pca tnpack procedure compared to those of a global optimization algorithm simulated annealing as implemented by the program package simann in terms of producing the projection mapping of a database numerical results show that the 2d pca tnpack mapping can preserve the distance relationships of the original database and is thus valuable as a first step in similarity and diversity applications of course the generation of a global rather than local minimizer and its interpretation in terms of pharameceutical applications remains a challenge since all numerical tests are performed on the merck mddr database results are representative of realistic cases encountered in the field of drug design and may help analyze properties of medicinal compounds

learning translation invariant kernels for classification appropriate selection of the kernel function which implicitly defines the feature space of an algorithm has a crucial role in the success of kernel methods in this paper we consider the problem of optimizing a kernel function over the class of translation invariant kernels for the task of binary classification the learning capacity of this class is invariant with respect to rotation and scaling of the features and it encompasses the set of radial kernels we show that how translation invariant kernel functions can be embedded in a nested set of sub classes and consider the kernel learning problem over one of these sub classes this allows the choice of an appropriate sub class based on the problem at hand we use the criterion proposed by lanckriet et al 2004 to obtain a functional formulation for the problem it will be proven that the optimal kernel is a finite mixture of cosine functions the kernel learning problem is then formulated as a semi infinite programming sip problem which is solved by a sequence of quadratically constrained quadratic programming qcqp sub problems using the fact that the cosine kernel is of rank two we propose a formulation of a qcqp sub problem which does not require the kernel matrices to be loaded into memory making the method applicable to large scale problems we also address the issue of including other classes of kernels such as individual kernels and isotropic gaussian kernels in the learning process another interesting feature of the proposed method is that the optimal classifier has an expansion in terms of the number of cosine kernels instead of support vectors leading to a remarkable speedup at run time as a by product we also generalize the kernel trick to complex valued kernel functions our experiments on artificial and real world benchmark data sets including the usps and the mnist digit recognition data sets show the usefulness of the proposed method

solving constrained optimization problems using a novel genetic algorithm a novel genetic algorithm is described in this paper for the problem of constrained optimization the algorithm incorporates modified genetic operators that preserve the feasibility of the trial solutions encoded in the chromosomes the stochastic application of a local search procedure and a stopping rule which is based on asymptotic considerations the algorithm is tested on a series of well known test problems and a comparison is made against the algorithms c somga and donlp2

partitioned variable metric updates for large structured optimization problems this paper presents a minimization method based on the idea of partitioned updating of the hessian matrix in the case where the objective function can be decomposed in a sum of convex element functions this situation occurs in a large class of practical problems including nonlinear finite elements calculations some theoretical and algorithmic properties of the update are discussed and encouraging numerical results are presented

the use of trust regions in kohn sham total energyminimization the self consistent field scf iteration widely used forcomputing the ground state energy and the corresponding single particlewave functions associated with a many electronatomistic system is viewedin this paper as an optimization procedure that minimizes the kohn shamtotal energy indirectly by minimizing a sequence of quadratic surrogatefunctions we point out the similarity and difference between the totalenergy and the surrogate and show how the scf iteration can fail whenthe minimizer of the surrogate produces an increase in the ks totalenergy a trust region technique is introduced as a way to restrict theupdate of the wave functions within a small neighborhood of anapproximate solution at which the gradient of the total energy agreeswith that of the surrogate the use of trust region in scf is not new however it has been observed that directly applying a trust region basedscf trscf to the kohn sham total energy often leads to slowconvergence we propose to use trscf within a direct constrainedminimization dcm algorithm we developed in cite dcm the keyingredients of thedcm algorithm involve projecting the total energyfunction into a sequence of subspaces of small dimensions and seeking theminimizerof the total energy function within each subspace theminimizer of a subspace energy function which is computed by trscf notonly provides amore search direction along which the ks total energy functiondecreases but also gives an optimal step length that yields asufficient decrease in total energy a numerical example is provided todemonstrate that the combination of trscf and dcm is more efficient thanscf less

renewable estimation and incremental inference in generalized linear models with streaming data sets the paper presents an incremental updating algorithm to analyse streaming data sets using generalized linear models the method proposed is formulated within a new framework of renewable estimation and incremental inference in which the maximum likelihood estimator is renewed with current data and summary statistics of historical data our framework can be implemented within a popular distributed computing environment known as apache spark to scale up computation consisting of two data processing layers the rho architecture enables us to accommodate inference related statistics and to facilitate sequential updating of the statistics used in both estimation and inference we establish estimation consistency and asymptotic normality of the proposed renewable estimator in which the wald test is utilized for an incremental inference our methods are examined and illustrated by various numerical examples from both simulation experiments and a real world data analysis

inverse eigenvalue problems a collection of inverse eigenvalue problems are identified and classified according to their characteristics current developments in both the theoretic and the algorithmic aspects are summarized and reviewed in this paper this exposition also reveals many open questions that deserve further study an extensive bibliography of pertinent literature is attached

adaptive regularized self consistent field iteration with exact hessian for electronic structure calculation the self consistent field scf iteration has been used ubiquitously for solving the kohn sham ks equation or the minimization of the ks total energy functional with respect to orthogonality constraints in electronic structure calculations although scf with heuristics such as charge mixing often works remarkably well on many problems it is well known that its convergence can be unpredictable and there is no general theoretical analysis on its performance we regularize the scf iteration and establish rigorous global convergence to the first order optimality conditions the hessian of the total energy functional is further exploited by adding the part of the hessian which is not considered in scf our methods can always achieve a highly accurate solution on problems for which scf fails and exhibit a better convergence rate than scf in the kssolv toolbox under the matlab environment

adjoint based acoustic inversion for the physical characterization of a shallow water environment recently the concept of adjoint modeling has been introduced in shallow water acoustics for solving inverse problems analytical adjoints have been derived for normal modes and for both the standard parabolic equation and claerbout s wide angle approximation wape this paper proposes the application of a semiautomatic adjoint approach that has been successfully applied in the past for multidimensional variational data assimilation in meteorological and climate modeling starting from a modular graph representation of the underlying forward model a programming tool facilitates the generation and coding of both the tangent linear and the adjoint models the potential of this numerical adjoint approach for the physical characterization of a shallow water environment is illustrated with two applications for geoacoustic inversion and ocean acoustic tomography using claerbout s wape in combination with nonlocal boundary conditions furthermore the adjoint optimization is extended to multiple frequencies and

global convergence of online limited memory bfgs global convergence of an online stochastic limited memory version of the broyden fletcher goldfarb shanno bfgs quasi newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established lower and upper bounds on the hessian eigenvalues of the sample functions are shown to suffice to guarantee that the curvature approximation matrices have bounded determinants and traces which in turn permits establishing convergence to optimal arguments with probability 1 experimental evaluation on a search engine advertising problem showcase reductions in convergence time relative to stochastic gradient descent algorithms

accelerated conjugate gradient algorithm with finite difference hessian vector product approximation for unconstrained optimization in this paper we propose a fundamentally different conjugate gradient method in which the well known parameter b k is computed by an approximation of the hessian vector product through finite differences for search direction computation the method uses a forward difference approximation to the hessian vector product in combination with a careful choice of the finite difference interval for the step length computation we suggest an acceleration scheme able to improve the efficiency of the algorithm under common assumptions the method is proved to be globally convergent it is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear but the reduction in function values is significantly improved numerical comparisons with conjugate gradient algorithms including conmin by shanno and phua d f shanno k h phua algorithm 500 minimization of unconstrained multivariate functions acm trans math softw 2 1976 87 94 scalcg by andrei n andrei scaled conjugate gradient algorithms for unconstrained optimization comput optim appl 38 2007 401 416 n andrei scaled memoryless bfgs preconditioned conjugate gradient algorithm for unconstrained optimization optim methods softw 22 2007 561 571 n andrei a scaled bfgs preconditioned conjugate gradient algorithm for unconstrained optimization appl math lett 20 2007 645 650 and new conjugacy condition and related new conjugate gradient by li tang and wei g li c tang z wei new conjugacy condition and related new conjugate gradient methods for unconstrained optimization j comput appl math 202 2007 523 539 or truncated newton tn by nash s g nash preconditioning of truncated newton methods siam j on scientific and statistical computing 6 1985 599 616 using a set of 750 unconstrained optimization test problems show that the suggested algorithm outperforms these conjugate gradient algorithms as well as tn

quadratic projection based feature extraction with its application to biometric recognition this paper presents a novel quadratic projection based feature extraction framework where a set of quadratic matrices is learned to distinguish each class from all other classes we formulate quadratic matrix learning qml as a standard semidefinite programming sdp problem however the conventional interior point sdp solvers do not scale well to the problem of qml for high dimensional data to solve the scalability of qml we develop an efficient algorithm termed dualqml based on the lagrange duality theory to extract nonlinear features to evaluate the feasibility and effectiveness of the proposed framework we conduct extensive experiments on biometric recognition experimental results on three representative biometric recognition tasks including face palmprint and ear recognition demonstrate the superiority of the dualqml based feature extraction algorithm compared to the current state of the art algorithms highlightsa novel quadratic projection based feature extraction framework is developed a set of quadratic matrices is learnt to distinguish each class from other classes quadratic matrix learning qml is formulated as an sdp problem an efficient algorithm is developed to solve qml based on lagrange duality theory experiments on biometric recognition show the effectiveness of our algorithm

a limited memory quasi newton trust region method for box constrained optimization by means of wolfe conditions strategy we propose a quasi newton trust region method to solve box constrained optimization problems this method is an adequate combination of the compact limited memory bfgs and the trust region direction while the generated point satisfies the wolfe conditions and therefore maintains a positive definite approximation to the hessian of the objective function the global convergence and the quadratic convergence of this method are established under suitable conditions finally we compare our algorithms iwtral and ibwtral with an active set trust region algorithm astral xu and burke 2007 on the cutest box constrained test problems presented by gould et al 2015 numerical results show that the presented method is competitive and totally interesting for solving box constrained optimization

bfgs with update skipping and varying memory we give conditions under which limited memory quasi newton methods with exact line searches will terminate in n steps when minimizing n dimensional quadratic functions we show that although all broyden family methods terminate in n steps in their full memory versions only bfgs does so with limited memory additionally we show that full memory broyden family methods with exact line searches terminate in at most n p steps when p matrix updates are skipped we introduce new limited memory bfgs variants and test them on nonquadratic minimization problems

learning harmonium models with infinite latent features undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks one common challenge in learning such models is to determine the number of hidden units that are unknown a priori although bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed bayesian networks very little effort has been made toward applying bayesian nonparametrics to learn undirected latent variable models in this paper we present the infinite exponential family harmonium iefh a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool we also present two important extensions of iefh to 1 multiview iefh for dealing with heterogeneous data and 2 infinite maximum margin harmonium immh for incorporating supervising side information to learn predictive latent features we develop variational inference algorithms to learn model parameters our methods are computationally competitive because of the avoidance of selecting the number of latent units our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iefh and immh inherited from bayesian nonparametrics and max margin learning such results were not available until now and contribute to expanding the scope of bayesian nonparametrics to learn the structures of undirected latent variable models