a time parallel approach to strong constraint four dimensional variational data assimilation a parallel in time algorithm based on an augmented lagrangian approach is proposed to solve four dimensional variational 4d var data assimilation problems the assimilation window is divided into multiple sub intervals that allows parallelization of cost function and gradient computations the solutions to the continuity equations across interval boundaries are added as constraints the augmented lagrangian approach leads to a different formulation of the variational data assimilation problem than the weakly constrained 4d var a combination of serial and parallel 4d vars to increase performance is also explored the methodology is illustrated on data assimilation problems involving the lorenz 96 and the shallow water models

iterative scaling and coordinate descent methods for maximum entropy models maximum entropy maxent is useful in natural language processing and many other areas iterative scaling is methods are one of the most popular approaches to solve maxent with many variants of is methods it is difficult to understand them and see the differences in this paper we create a general and unified framework for iterative scaling methods this framework also connects iterative scaling and coordinate descent methods we prove general convergence results for is methods and analyze their computational complexity based on the proposed framework we extend a coordinate descent method for linear svm to maxent results show that it is faster than existing iterative scaling methods

identification of boundary conditions in a nonlinear shallow water flow abstract in this paper we present sufficient smoothness results on the solution of a nonlinear shallow water problem in order to obtain an existence result for the associated adjoint problem we introduce a necessary but not sufficient existence condition for the functional minimum the control is made on the velocity boundary condition in considering observations at isolated points moreover we give a set of numerical results obtained in a real situation

the limited memory conjugate gradient method in theory the successive gradients generated by the conjugate gradient method applied to a quadratic should be orthogonal however for some ill conditioned problems orthogonality is quickly lost due to rounding errors and convergence is much slower than expected a limited memory version of the nonlinear conjugate gradient method is developed the memory is used to both detect the loss of orthogonality and to restore orthogonality an implementation of the algorithm is presented based on the cg descent nonlinear conjugate gradient method limited memory cg descent l cg descent possesses a global convergence property similar to that of the memoryless algorithm but has much better practical performance numerical comparisons to the limited memory bfgs method l bfgs are given using the cuter test problems

limited memory bfgs systems with diagonal updates abstract in this paper we investigate a formula to solve systems of the form b i x y where b is a limited memory bfgs quasi newton matrix and is a positive constant these types of systems arise naturally in large scale optimization such as trust region methods as well as doubly augmented lagrangian methods we show that provided a simple condition holds on b 0 and the system b i x y can be solved via a recursion formula that requires only vector inner products this formula has complexity m 2 n where m is the number of l bfgs updates and n m is the dimension of x

penalty barrier multiplier methods for convex programming problems we study a class of methods for solving convex programs which are based on nonquadratic augmented lagrangians for which the penalty parameters are functions of the multipliers this gives rise to lagrangians which are nonlinear in the multipliers each augmented lagrangian is specified by a choice of a penalty function varphi and a penalty updating function pi the requirements on varphi are mild and allow for the inclusion of most of the previously suggested augmented lagrangians more importantly a new type of penalty barrier function having a logarithmic branch glued to a quadratic branch is introduced and used to construct an efficient algorithm convergence of the algorithms is proved for the case of pi being a sublinear function of the dual multipliers the algorithms are tested on large scale quadratically constrained problems arising in structural optimization

a tool for the analysis of quasi newton methods with application to unconstrained minimization the bfgs update formula is shown to have an important property that is inde pendent of the algorithmic context of the update and that is relevant to both constrained and unconstrained optimization the bfgs method for unconstrained optimization using a variety of line searches including backtracking is shown to be globally and superlinearly convergent on uniformly convex problems the analysis is particularly simple due to the use of some new tools introduced in this paper

entropy like proximal methods in convex programming we study an extension of the proximal method for convex programming where the quadratic regularization kernel is substituted by a class of convex statistical distances called divergences which are typically entropy like in form after establishing several basic properties of these quasi distances we present a convergence analysis of the resulting entropy like proximal algorithm applying this algorithm to the dual of a convex program we recover a wide class of nonquadratic multiplier methods and prove their convergence

action recognition based on overcomplete independent components analysis existing works on action recognition rely on two separate stages 1 designing hand designed features or learning features from video data 2 classifying features using a classifier such as svm or adaboost motivated by two observations 1 independent component analysis ica is capable of encoding intrinsic features underlying video data and 2 videos of different actions can be easily distinguished by their intrinsic features we propose a simple but effective action recognition framework based on the recently proposed overcomplete ica model after a set of overcomplete ica basis functions are learned from the densely sampled 3d patches from training videos for each action a test video is classified as the class whose basis functions can reconstruct the sampled 3d patches from the test video with the smallest reconstruction error the experimental results on five benchmark datasets demonstrate that the proposed approach outperforms several state of the art works

some numerical experiments with variable storage quasi newton algorithms this paper describes some numerical experiments with variable storage quasi newton methods for the optimization of some large scale models coming from fluid mechanics and molecular biology in addition to assessing these kinds of methods in real life situations we compare an algorithm of a buckley with a proposal by j nocedal the latter seems generally superior provided that careful attention is given to some nontrivial implementation aspects which concern the general question of properly initializing a quasi newton matrix in this context we find it appropriate to use a diagonal matrix generated by an update of the identity matrix so as to fit the rayleigh ellipsoid of the local hessian in the direction of the change in the gradient also a variational derivation of some rank one and rank two updates in hilbert spaces is given

modeling textured images using generalized long correlation models the long correlation lc models are a general class of random field rf models which are able to model correlations extending over large image regions with few model parameters the lc models have seen limited use due to lack of an effective method for estimating the model parameters in this work we develop an estimation scheme for a very general form of this model and demonstrate its applicability to texture modeling applications the relationship of the generalized lc models to other classes of rf models namely the simultaneous autoregressive sar and markov random field mrf models is shown while it is known that the sar model is a special case of the lc model we show that the mrf model is also encompassed by this model consequently the lc model may be considered as a generalization of the sar and mrf models

hypergeometric family and item overlap rates in computerized adaptive testing a computerized adaptive test cat is usually administered to small groups of examinees at frequent time intervals it is often the case that examinees who take the test earlier share information with examinees who will take the test later thus increasing the risk that many items may become known item overlap rate for a group of examinees refers to the number of overlapping items encountered by these examinees divided by the test length for a specific item pool different item selection algorithms may yield different item overlap rates an important issue in designing a good cat item selection algorithm is to keep item overlap rate below a preset level in doing so it is important to investigate what the lowest rate could be for all possible item selection algorithms in this paper we rigorously prove that if every item has an equal possibility to be selected from the pool in a fixed length cat the number of overlapping items among any randomly sampled examinees follows the hypergeometric distribution family for 1 thus the expected values of the number of overlapping items among any randomly sampled examinees can be calculated precisely these values may serve as benchmarks in controlling item overlap rates for fixed length adaptive tests

when cognitive diagnosis meets computerized adaptive testing cd cat computerized adaptive testing cat is a mode of testing which enables more efficient and accurate recovery of one or more latent traits traditionally cat is built upon item response theory irt models that assume unidimensionality however the problem of how to build cat upon latent class models lcm has not been investigated until recently when tatsuoka j r stat soc ser c appl stat 51 337 350 2002 and tatsuoka and ferguson j r stat ser b 65 143 157 2003 established a general theorem on the asymptotically optimal sequential selection of experiments to classify finite partially ordered sets xu chang and douglas paper presented at the annual meeting of national council on measurement in education montreal canada 2003 then tested two heuristics in a simulation study based on tatsuoka s theoretical work in the context of computerized adaptive testing one of the heuristics was developed based on kullback leibler information and the other based on shannon entropy in this paper we showcase the application of the optimal sequential selection methodology in item selection of cat that is built upon cognitive diagnostic models two new heuristics are proposed and are compared against the randomized item selection method and the two heuristics investigated in xu et al paper presented at the annual meeting of national council on measurement in education montreal canada 2003 finally we show the connection between the kullback leibler information based approaches and the shannon entropy based approach as well as the connection between algorithms built upon lcm and those built upon irt models

an implementation of karmarkar s algorithm for linear programming this paper describes the implementation of power series dual affine scaling variants of karmarkar s algorithm for linear programming based on a continuous version of karmarkar s algorithm two variants resulting from first and second order approximations of the continuous trajectory are implemented and tested linear programs are expressed in an inequality form which allows for the inexact computation of the algorithm s direction of improvement resulting in a significant computational advantage implementation issues particular to this family of algorithms such as treatment of dense columns are discussed the code is tested on several standard linear programming problems and compares favorably with the simplex codeminos 4 0

does standard deviation matter using standard deviation to quantify security of multistage testing with the advent of web based technology online testing is becoming a mainstream mode in large scale educational assessments most online tests are administered continuously in a testing window which may post test security problems because examinees who take the test earlier may share information with those who take the test later researchers have proposed various statistical indices to assess the test security and one most often used index is the average test overlap rate which was further generalized to the item pooling index chang zhang 2002 2003 these indices however are all defined as the means that is the expected proportion of common items among examinees and they were originally proposed for computerized adaptive testing cat recently multistage testing mst has become a popular alternative to cat the unique features of mst make it important to report not only the mean but also the standard deviation sd of test overlap rate as we advocate in this paper the standard deviation of test overlap rate adds important information to the test security profile because for the same mean a large sd reflects that certain groups of examinees share more common items than other groups in this study we analytically derived the lower bounds of the sd under mst with the results under cat as a benchmark it is shown that when the mean overlap rate is the same between mst and cat the sd of test overlap tends to be larger in mst a simulation study was conducted to provide empirical evidence we also compared the security of mst under the single pool versus the multiple pool designs both analytical and simulation studies show that the non overlapping multiple pool design will slightly increase the security risk

a new conjugate gradient method with guaranteed descent and an efficient line search a new nonlinear conjugate gradient method and an associated implementation based on an inexact line search are proposed and analyzed with exact line search our method reduces to a nonlinear version of the hestenes stiefel conjugate gradient scheme for any inexact line search our scheme satisfies the descent condition bf g k sf t bf d k le frac 7 8 bf g k 2 moreover a global convergence result is established when the line search fulfills the wolfe conditions a new line search scheme is developed that is efficient and highly accurate efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer high accuracy is achieved by using a convergence criterion which we call the approximate wolfe conditions obtained by replacing the sufficient decrease criterion in the wolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion numerical comparisons are given with both l bfgs and conjugate gradient methods using the unconstrained optimization problems in the cute library

a algorithm for convex minimization for convex minimization we introduce an algorithm based on space decomposition the method uses a bundle subroutine to generate a sequence of approximate proximal points when a primal dual track leading to a solution and zero subgradient pair exists these points approximate the primal track points and give the algorithm s or corrector steps the subroutine also approximates dual track points that are gradients needed for the method s newton predictor steps with the inclusion of a simple line search the resulting algorithm is proved to be globally convergent the convergence is superlinear if the primal dual track points and the objective s hessian are approximated well enough

on the use of stochastic hessian information in optimization methods for machine learning this paper describes how to incorporate sampled curvature information in a newton cg method and in a limited memory quasi newton method for statistical learning the motivation for this work stems from supervised machine learning applications involving a very large number of training points we follow a batch approach also known in the stochastic optimization literature as a sample average approximation approach curvature information is incorporated in two subsampled hessian algorithms one based on a matrix free inexact newton iteration and one on a preconditioned limited memory bfgs iteration a crucial feature of our technique is that hessian vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient the efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition

proximal minimization methods with generalized bregman functions we consider methods for minimizing a convex function f that generate a sequence xk by taking xk 1 to be an approximate minimizer of f x dh x xk ck where ck 0 and dh is the d function of a bregman function h extensions are made to b functions that generalize bregman functions and cover more applications convergence is established under criteria amenable to implementation applications are made to nonquadratic multiplier methods for nonlinear programs

hessian riemannian gradient flows in convex programming in view of solving theoretically constrained minimization problems we investigate the properties of the gradient flows with respect to hessian riemannian metrics induced by legendre functions the first result characterizes hessian riemannian structures on convex sets as metrics that have a specific integration property with respect to variational inequalities giving a new motivation for the introduction of bregman type distances then the general evolution problem is introduced and global convergence is established under quasi convexity conditions with interesting refinements in the case of convex minimization some explicit examples of these gradient flows are discussed dual trajectories are identified and sufficient conditions for dual convergence are examined for a convex program with positivity and equality constraints some convergence rate results are established in the case of a linear objective function several optimality characterizations of the orbits are given optimal path of viscosi

parameter and state model reduction for large scale statistical inverse problems a greedy algorithm for the construction of a reduced model with reduction in both parameter and state is developed for an efficient solution of statistical inverse problems governed by partial differential equations with distributed parameters large scale models are too costly to evaluate repeatedly as is required in the statistical setting furthermore these models often have high dimensional parametric input spaces which compounds the difficulty of effectively exploring the uncertainty space we simultaneously address both challenges by constructing a projection based reduced model that accepts low dimensional parameter inputs and whose model evaluations are inexpensive the associated parameter and state bases are obtained through a greedy procedure that targets the governing equations model outputs and prior information the methodology and results are presented for groundwater inverse problems in one and two dimensions

a practical pr conjugate gradient method only using gradient abstract in this paper we consider the polak ribiere or polak ribiere plus conjugate gradient method for solving optimality condition of an unconstrained minimization problem we give two new steplength rules only using gradient and under gradient lipschitz assumption prove this method s global convergence correspondingly then we develop a practical polak ribiere plus method whose steplength is located by one inequality only using gradient and report promising numerical results on high accuracy solution for some standard test problems when compared to the state of art methods in this research direction importantly our work provides a new idea of devising a practical version of the celebrated polak ribiere or polak ribiere plus method

adjoint implementation of rosenbrock methods applied to variational data assimilation problems abstract in the past decade the variational method has been successfully applied in data assimilation problems for atmospheric chemistry models in 4d var data assimilation a minimization algorithm is used to find the set of control variables which minimizes the weighted least squares distance between model predictions and observations over the assimilation window using the adjoint method the gradient of the cost function can be computed fast at the expense of few function evaluations making the optimization process very efficient for large scale models the high storage requirements and the difficulty of implementing the adjoint code when sophisticated integrators are used to solve the stiff chemistry make the assimilation a very intensive computational process if the sparse structure of the chemical models is carefully exploited rosenbrock methods have been proved to be reliable chemistry solvers because of their outstanding stability properties and conservation of the linear invariants of the system in this paper we present an efficient implementation of the adjoint code for the rosenbrock type methods which can reduce the storage requirements of the forward model and is suitable for automatization the adjoint code is completely generated using symbolic preprocessing and automatic differentiation tools which allow flexibility and require minimal user intervention

mixing linear svms for nonlinear classification in this paper we address the problem of combining linear support vector machines svms for classification of large scale nonlinear datasets the motivation is to exploit both the efficiency of linear svms lsvms in learning and prediction and the power of nonlinear svms in classification to this end we develop a lsvm mixture model that exploits a divide and conquer strategy by partitioning the feature space into subregions of linearly separable datapoints and learning a lsvm for each of these regions we do this implicitly by deriving a generative model over the joint data and label distributions consequently we can impose priors on the mixing coefficients and do implicit model selection in a top down manner during the parameter estimation process this guarantees the sparsity of the learned model experimental results show that the proposed method can achieve the efficiency of lsvms in the prediction phase while still providing a classification performance comparable to nonlinear svms

convergence and boundary behavior of the projective scaling trajectories for linear programming we analyze the convergence and boundary behavior of the continuous trajectories of the vector field induced by the projective scaling algorithm as applied to possibly degenerate linear programming problems in karmarkar s standard form we show that a projective scaling trajectory tends to an optimal solution which in general depends on the starting point when the optimal solution is unique we show that all projective scaling trajectories approach the optimal solution through the same asymptotic direction our analysis is based on the affine scaling trajectories for the homogeneous standard form linear program that arises from karmarkar s standard form linear program by removing the unique nonhomogeneous constraint

on the continuous trajectories for a potential reduction algorithm for linear programming for a possibly degenerate linear program minx ctx ax b x 0 a is an m n real matrix b rm and c rn whose optimal value is 0 we study the limiting behavior of the trajectories of the family of vector fields i qx i xpax xc q i1ctx1 for all values of q n where x is the diagonal matrix associated with x and pax is the projection operator onto the null space of ax a polynomial algorithm based on the directions i qx has been presented by gonzaga 6 when q n or q n n we show that all trajectories of i qx converge to the unique center of the optimal face of the given linear program when this face consists of a unique vertex it is shown that any trajectory of i qx approaches this vertex along the same direction when the optimal face consists of more than one point we show that there is a threshold value i 0 such that for q i most of the trajectories of i qx converge to the center tangentially to the optimal face and that the direction of approach of a trajectory of i qx depends on the initial condition for q i q i the trajectories of i qx converge to the center along a unique direction along several directions which depend on the initial condition forming a positive angle with the optimal face

numerical methods for electronic structure calculations of materials the goal of this article is to give an overview of numerical problems encountered when determining the electronic structure of materials and the rich variety of techniques used to solve these problems the paper is intended for a diverse scientific computing audience for this reason we assume the reader does not have an extensive background in the related physics our overview focuses on the nature of the numerical problems to be solved their origin and the methods used to solve the resulting linear algebra or nonlinear optimization problems it is common knowledge that the behavior of matter at the nanoscale is in principle entirely determined by the schrodinger equation in practice this equation in its original form is not tractable successful but approximate versions of this equation which allow one to study nontrivial systems took about five or six decades to develop in particular the last two decades saw a flurry of activity in developing effective software one of the main practical variants of the schrodinger equation is based on what is referred to as density functional theory dft the combination of dft with pseudopotentials allows one to obtain in an efficient way the ground state configuration for many materials this article will emphasize pseudopotential density functional theory but other techniques will be discussed as well

a fast learning algorithm for deep belief nets we show how to use complementary priors to eliminate the explaining away effects that make inference difficult in densely connected belief nets that have many hidden layers using complementary priors we derive a fast greedy algorithm that can learn deep directed belief networks one layer at a time provided the top two layers form an undirected associative memory the fast greedy algorithm is used to initialize a slower learning procedure that fine tunes the weights using a contrastive version of the wake sleep algorithm after fine tuning a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels this generative model gives better digit classification than the best discriminative learning algorithms the low dimensional manifolds on which the digits lie are modeled by long ravines in the free energy landscape of the top level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind

on the limited memory bfgs method for large scale optimization we study the numerical performance of a limited memory quasi newton method for large scale optimization which we call the l bfgs method we compare its performance with that of the method developed by buckley and lenir 1985 which combines cycles of bfgs steps and conjugate direction steps our numerical tests indicate that the l bfgs method is faster than the method of buckley and lenir and is better able to use additional storage to accelerate convergence we show that the l bfgs method can be greatly accelerated by means of a simple scaling we then compare the l bfgs method with the partitioned quasi newton method of griewank and toint 1982a the results show that for some problems the partitioned quasi newton method is clearly superior to the l bfgs method however we find that for other problems the l bfgs method is very competitive due to its low iteration cost we also study the convergence properties of the l bfgs method and prove global convergence on uniformly convex problems

accelerated scaled memoryless bfgs preconditioned conjugate gradient algorithm for unconstrained optimization abstract an accelerated scaled memoryless bfgs preconditioned conjugate gradient algorithm for solving unconstrained optimization problems is presented the basic idea is to combine the scaled memoryless bfgs method and the preconditioning technique in the frame of the conjugate gradient method the preconditioner which is also a scaled memoryless bfgs matrix is reset when the beale powell restart criterion holds the parameter scaling the gradient is selected as a spectral gradient for the steplength computation the method has the advantage that in conjugate gradient algorithms the step lengths may differ from 1 by two order of magnitude and tend to vary unpredictably thus we suggest an acceleration scheme able to improve the efficiency of the algorithm under common assumptions the method is proved to be globally convergent it is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear but the reduction in the function values is significantly improved in mild conditions the algorithm is globally convergent for strongly convex functions computational results for a set consisting of 750 unconstrained optimization test problems show that this new accelerated scaled conjugate gradient algorithm substantially outperforms known conjugate gradient methods scalcg 3 4 5 6 conmin by shanno and phua 1976 1978 42 43 hestenes and stiefel 1952 25 polak ribiere polyak 1969 32 33 dai and yuan 2001 17 dai and liao 2001 t 1 14 conjugate gradient with sufficient descent condition 7 hybrid dai and yuan 2001 17 hybrid dai and yuan zero 2001 17 cg descent by hager and zhang 2005 2006 22 23 as well as quasi newton lbfgs method 26 and truncated newton method by nash 1985 27

qn like variable storage conjugate gradients both conjugate gradient and quasi newton methods are quite successful at minimizing smooth nonlinear functions of several variables and each has its advantages in particular conjugate gradient methods require much less storage to implement than a quasi newton code and therefore find application when storage limitations occur they are however slower so there have recently been attempts to combine cg and qn algorithms so as to obtain an algorithm with good convergence properties and low storage requirements one such method is the code conmin due to shanno and phua it has proven quite successful but it has one limitation it has no middle ground in that it either operates as a quasi newton code using o n 2 storage locations or as a conjugate gradient code using 7n locations but it cannot take advantage of the not unusual situation where more than 7n locations are available but a quasi newton code requires an excessive amount of storage

some convergence properties of the modified log barrier method for linear programming on each iteration of the modified log barrier method for linear programming a vector of variables is calculated by the unconstrained minimization of a fixed positive multiple of the objective function minus a linear combination of the logarithms of the constraint residuals plus positive constants thus in contrast to interior point methods all the logarithms are finite whenever the vector of variables is feasible the parameters of the calculation have the property that the coefficients of the linear combination of logarithms can be viewed as estimates of lagrange multipliers of karush kuhn tucker conditions of the given problem initially these coefficients have any positive values and then their adjustment is derived from the zero gradient that occurs at the end of each unconstrained minimization calculation these techniques provide a simple and interesting algorithm for linear programming that was proposed by polyak we study its convergence properties finding that the sequence of calculated vari

limiting behavior of trajectories generated by a continuation method for monotone complementarity problems defining the mapping f from the 2n dimensional euclidean space r2n into itself by fx y x1y1 xn yn y1 f1x yn fnx for every x y r2n we write the cp the complementarity problem with a mapping f rn rn as the system of equations fx y 0 and x y r 2n where r 2n denotes the nonnegative orthant of r2n under the assumption that f is a monotone function on r n we show that f maps the positive orthant r 2n of r2n homeomorphically this result then ensures the existence of a trajectory consisting of the solutions x y t of the system of equations fx y wt and x y t r 2n 0 1 where w is a continuous mapping from the unit interval 0 1 into fr 2n such that w0 0 if t 0 the system coincides with the cp we study limiting behavior of the trajectory as t 0 and give some sufficient conditions for the trajectory to lead to a solution of the cp this approach is an extension of the one used in the polynomially bounded algorithm recently given by kojima mizuno and yoshise for solving positive semidefinite linear complementarity problems

articulated and generalized gaussian kernel correlation for human pose estimation in this paper we propose an articulated and generalized gaussian kernel correlation gkc based framework for human pose estimation we first derive a unified gkc representation that generalizes the previous sum of gaussians sog based methods for the similarity measure between a template and an observation both of which are represented by various sog variants then we develop an articulated gkc agkc by integrating a kinematic skeleton in a multivariate sog template that supports subject specific shape modeling and articulated pose estimation for both the full body and the hands we further propose a sequential body hand pose tracking algorithm by incorporating three regularization terms in the agkc function including visibility intersection penalty and pose continuity our tracking algorithm is simple yet effective and computationally efficient we evaluate our algorithm on two benchmark depth data sets the experimental results are promising and competitive when compared with the state of the art algorithms

the asymptotic distribution of average test overlap rate in computerized adaptive testing the average test overlap rate is often computed and reported as a measure of test security risk or item pool utilization of a computerized adaptive test cat despite the prevalent use of this sample statistic in both literature and operations its sampling distribution has never been known nor studied in earnest in response a proof is presented for the asymptotic distribution of a linear transformation of the average test overlap rate in fixed length cat the theoretical results enable the estimation of standard error and construction of confidence intervals moreover a practical simulation study demonstrates the statistical comparison of average test overlap rates between two cat designs with different exposure control methods

discrete second order adjoints in atmospheric chemical transport modeling atmospheric chemical transport models ctms are essential tools for the study of air pollution for environmental policy decisions for the interpretation of observational data and for producing air quality forecasts many air quality studies require sensitivity analyses i e the computation of derivatives of the model output with respect to model parameters the derivatives of a cost functional defined on the model output with respect to a large number of model parameters can be calculated efficiently through adjoint sensitivity analysis while the traditional first order adjoint models give the gradient of the cost functional with respect to parameters second order adjoint models give second derivative information in the form of products between the hessian of the cost functional and a vector representing a perturbation in sensitivity analysis a search direction in optimization an eigenvector etc in this paper we discuss the mathematical foundations of the discrete second order adjoint sensitivity method and present a complete set of computational tools for performing second order sensitivity studies in three dimensional atmospheric ctms the tools include discrete second order adjoints of runge kutta and of rosenbrock time stepping methods for stiff equations together with efficient implementation strategies numerical examples illustrate the use of these computational tools in important applications like sensitivity analysis optimization uncertainty quantification and the calculation of directions of maximal error growth in three dimensional atmospheric ctms

feature space interpretation of svms with indefinite kernels kernel methods are becoming increasingly popular for various kinds of machine learning tasks the most famous being the support vector machine svm for classification the svm is well understood when using conditionally positive definite cpd kernel functions however in practice non cpd kernels arise and demand application in svm the procedure of plugging these indefinite kernels in svm often yields good empirical classification results however they are hard to interpret due to missing geometrical and theoretical understanding in this paper we provide a step toward the comprehension of svm classifiers in these situations we give a geometric interpretation of svm with indefinite kernel functions we show that such svm are optimal hyperplane classifiers not by margin maximization but by minimization of distances between convex hulls in pseudo euclidean spaces by this we obtain a sound framework and motivation for indefinite svm this interpretation is the basis for further theoretical analysis e g investigating uniqueness and for the derivation of practical guidelines like characterizing the suitability of indefinite svm

limiting behavior of the affine scaling continuous trajectories for linear programming problems we consider the continuous trajectories of the vector field induced by the primal affine scaling algorithm as applied to linear programming problems in standard form by characterizing these trajectories as solutions of certain parametrized logarithmic barrier families of problems we show that these trajectories tend to an optimal solution which in general depends on the starting point by considering the trajectories that arise from the lagrangian multipliers of the above mentioned logarithmic barrier families of problems we show that the trajectories of the dual estimates associated with the affine scaling trajectories converge to the so called centered optimal solution of the dual problem we also present results related to asymptotic direction of the affine scaling trajectories we briefly discuss how to apply our results to linear programs formulated in formats different from the standard form finally we extend the results to the primal dual amne scaling algorithm

kullback leibler information and its applications in multi dimensional adaptive testing this paper first discusses the relationship between kullback leibler information kl and fisher information in the context of multi dimensional item response theory and is further interpreted for the two dimensional case from a geometric perspective this explication should allow for a better understanding of the various item selection methods in multi dimensional adaptive tests mat which are based on these two information measures the kl information index ki method is then discussed and two theorems are derived to quantify the relationship between ki and item parameters due to the fact that most of the existing item selection algorithms for mat bear severe computational complexity which substantially lowers the applicability of mat two versions of simplified kl index ski built from the analytical results are proposed to mimic the behavior of ki while reducing the overall computational intensity

on the smoothness constraints for four dimensional data assimilation an algorithm for determination of the weights of smoothness penalty constraints for the four dimensional variational data assimilation technique is proposed and evaluated to study the nature of smoothness penalty constraints a simple nonlinear harmonic oscillator problem is first considered penalizing smoothness constraints is found to make the modified hessian matrix of the cost function more positive definite akin to the idea behind the modified line search newton s methods however the use of the derivative smoothness constraints with a fixed coefficient does not warrant uniform imposition of these constraints at every iteration a remedy is to control the ratio of the smoothness penalty function over the cost function which can dramatically increase the positive definite area on the other hand the large smoothness coefficients obtained from this approach can deteriorate the convergence property of the minimization problem based on these observations an algorithm for tuning the weights of smoothness constraints is proposed to overcome the aforementioned problems the algorithm is first applied to a simple dynamic problem it is then tested on the retrieval of microscale turbulent structures in a simulated convective boundary layer this method is further evaluated on the retrieval of a strong meso scale thunderstorm outflow from doppler radar data the results show that the algorithm yields efficient retrieval

reduced hessian quasi newton methods for unconstrained optimization quasi newton methods are reliable and efficient on a wide range of problems but they can require many iterations if the problem is ill conditioned or if a poor initial estimate of the hessian is used in this paper we discuss methods designed to be more efficient in these situations all the methods to be considered exploit the fact that quasi newton methods accumulate approximate second derivative information in a sequence of expanding subspaces associated with each of these subspaces is a certain reduced approximate hessian that provides a complete but compact representation of the second derivative information approximated up to that point algorithms that compute an explicit reduced hessian approximation have two important advantages over conventional quasi newton methods first the amount of computation for each iteration is significantly less during the early stages this advantage is increased by forcing the iterates to linger on a manifold whose dimension can be significantly smaller than the subspace in which curvature has been accumulated second approximate curvature along directions that lie off the manifold can be reinitialized as the iterations proceed thereby reducing the influence of a poor initial estimate of the hessian these advantages are illustrated by extensive numerical results from problems in the cute test set our experiments provide strong evidence that reduced hessian quasi newton methods are more efficient and robust than conventional bfgs methods and some recently proposed extensions

convergence analysis of a proximal like minimization algorithm using bregman functions an alternative convergence proof of a proximal like minimization algorithm using bregman functions recently proposed by censor and zenios is presented the analysis allows the establishment of a global convergence rate of the algorithm expressed in terms of function values

machine learning of linear differential equations using gaussian processes this work leverages recent advances in probabilistic machine learning to discover governing equations expressed by parametric linear operators such equations involve but are not limited to ordinary and partial differential integro differential and fractional order operators here gaussian process priors are modified according to the particular form of such operators and are employed to infer parameters of the linear equations from scarce and possibly noisy observations such observations may come from experiments or black box computer simulations as demonstrated in several synthetic examples and a realistic application in functional genomics employ probabilistic machine learning to discover governing equations expressed by parametric linear operators proper placement of gaussian process priors allows one to efficiently infer model parameters via maximum likelihood estimation a general treatment of inverse problems governed by linear operators leading to model discovery from just a handful of noisy measurements

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

trust region newton method for logistic regression large scale logistic regression arises in many applications such as document classification and natural language processing in this paper we apply a trust region newton method to maximize the log likelihood of the logistic regression model the proposed method uses only approximate newton steps in the beginning but achieves fast convergence in the end experiments show that it is faster than the commonly used quasi newton approach for logistic regression we also extend the proposed method to large scale l2 loss linear support vector machines svm

central paths generalized proximal point methods and cauchy trajectories in riemannian manifolds we study the relationships between three concepts which arise in connection with variational inequality problems central paths defined by arbitrary barriers generalized proximal point methods where a bregman distance substitutes for the euclidean one and cauchy trajectory in riemannian manifolds first we prove that under rather general hypotheses the central path defined by a general barrier for a monotone variational inequality problem is well defined bounded and continuous and converges to the analytic center of the solution set with respect to the given barrier thus generalizing results which deal only with complementarity problems and with the logarithmic barrier next we prove that a sequence generated by the proximal point method with the bregman distance naturally induced by the barrier function converges precisely to the same point furthermore for a certain class of problems including linear programming such a sequence is contained in the central path making the concepts of central path and generalized proximal point sequence virtually equivalent finally we prove that for this class of problems the central path also coincides with the cauchy trajectory in the riemannian manifold defined on the positive orthant by a metric given by the hessian of the barrier i e a curve whose direction at each point is the negative gradient of the objective function at that point in the riemannian metric

iterated tabu search for the circular open dimension problem abstract this paper investigates the circular open dimension problem codp which consists of packing a set of circles of known radii into a strip of fixed width and unlimited length without overlapping the objective is to minimize the length of the strip in this paper codp is solved by a series of sub problems each corresponding to a fixed strip length for each sub problem an iterated tabu search approach named its is proposed its starts from a randomly generated solution and attempts to gain improvements by a tabu search procedure after that if the obtained solution is not feasible a perturbation operator is subsequently employed to reconstruct the incumbent solution and an acceptance criterion is implemented to determine whether or not accept the perturbed solution as a supplementary method the length of the strip is determined in monotonously decreasing way with the aid of some post processing techniques the search terminates and returns the best found solution after the allowed computation time has been elapsed computational experiments based on numerous well known benchmark instances show that its produces quite competitive results with respect to the best known results while the computational time remains reasonable for each instance

an active set quasi newton method with projected search for bound constrained minimization we analyze an active set quasi newton method for large scale bound constrained problems our approach combines the accurate active set identification function and the projected search both of these strategies permit fast change in the working set the limited memory method is employed to update the inactive variables while the active variables are updated by simple rules a further division of the active set enables the global convergence of the new algorithm numerical tests demonstrate the efficiency and performance of the present strategy and its comparison with some existing active set strategies

a discrete newton algorithm for minimizing a function of many variables a newton like method is presented for minimizing a function ofn variables it uses only function and gradient values and is a variant of the discrete newton algorithm this variant requires fewer operations than the standard method whenn 39 and storage is proportional ton rather thann2

a maximum a posteriori estimation framework for robust high dynamic range video synthesis high dynamic range hdr image synthesis from multiple low dynamic range exposures continues to be actively researched the extension to hdr video synthesis is a topic of significant current interest due to potential cost benefits for hdr video a stiff practical challenge presents itself in the form of accurate correspondence estimation of objects between video frames in particular loss of data resulting from poor exposures and varying intensity makes conventional optical flow methods highly inaccurate we avoid exact correspondence estimation by proposing a statistical approach via maximum a posterior estimation and under appropriate statistical assumptions and choice of priors and models we reduce it to an optimization problem of solving for the foreground and background of the target frame we obtain the background through rank minimization and estimate the foreground via a novel multiscale adaptive kernel regression technique which implicitly captures local structure and temporal motion by solving an unconstrained optimization problem extensive experimental results on both real and synthetic data sets demonstrate that our algorithm is more capable of delivering high quality hdr videos than current state of the art methods under both subjective and objective assessments furthermore a thorough complexity analysis reveals that our algorithm achieves better complexity performance tradeoff than conventional methods

a novel variational algorithmic formulation for wrinkling at finite strains based on energy minimization application to mesh adaption this paper is concerned with an efficient novel algorithmic formulation for wrinkling at finite strains in contrast to previously published numerical implementations the advocated method is fully variational more precisely the parameters describing wrinkles or slacks together with the unknown deformation mapping are computed jointly by minimizing the potential energy of the considered mechanical system furthermore the wrinkling criteria are naturally included within the presented variational framework the proposed method allows to employ three dimensional constitutive models without any additional modification i e a projection in plane stress space is not required analogously to the wrinkling parameters the non vanishing out of plane component of the strain tensor results conveniently from relaxing the respective helmholtz energy of the membrane the proposed framework is very general and does not rely on any assumption regarding the symmetry of the material i e arbitrary anisotropic hyperelastic models can be consistently taken into account the advantages associated with such a variational method are manifold for instance it opens up the possibility of applying standard optimization algorithms to the numerical implementation this is especially important for highly non linear or singular problems such as wrinkling on the other hand minimization principles provide a suitable basis for a posteriori error estimation and thus for adaptive finite element formulations as a prototype a variational error indicator leading to an efficient h adaption for wrinkling is briefly discussed the performance of the wrinkling approach is demonstrated by selected finite element analyses

convergence theorems for generalized alternating minimization procedures the em algorithm is widely used to develop iterative parameter estimation procedures for statistical models in cases where these procedures strictly follow the em formulation the convergence properties of the estimation procedures are well understood in some instances there are practical reasons to develop procedures that do not strictly fall within the em framework we study em variants in which the e step is not performed exactly either to obtain improved rates of convergence or due to approximations needed to compute statistics under a model family over which e steps cannot be realized since these variants are not em procedures the standard g em convergence results do not apply to them we present an information geometric framework for describing such algorithms and analyzing their convergence properties we apply this framework to analyze the convergence properties of incremental em and variational em for incremental em we discuss conditions under these algorithms converge in likelihood for variational em we show how the e step approximation prevents convergence to local maxima in likelihood

a dantzig wolfe like variant of karmarkar s interior point linear programming algorithm we show that a variant of karmarkar s projective algorithm for linear programming can be viewed as following the approach of dantzig wolfe decomposition at each iteration the current primal feasible solution generates prices which are used to form a simple subproblem the solution to the subproblem is then incorporated into the current feasible solution with a suitable choice of stepsize a constant reduction in potential function is achieved at each iteration we also use our analysis to motivate a new primal simplex pivot rule that is closely related to rules used by e klotz and l schrage

global convergence properties of the two new dependent fletcher reeves conjugate gradient methods in this paper we propose two new dependent fletcher reeves conjugate gradient methods arising from different choice for the scalar b k we make two different kinds of estimations of upper bounds of b k with respect to b k f r which are based on abel theorem of non convergent series of positive items with several different line searches global convergence results are established for the two new methods which extend the previous dependent fletcher reeves conjugate gradient methods

a nonmonotone line search technique and its application to unconstrained optimization a new nonmonotone line search algorithm is proposed and analyzed in our scheme we require that an average of the successive function values decreases while the traditional nonmonotone approach of grippo lampariello and lucidi siam j numer anal 23 1986 pp 707 716 requires that a maximum of recent function values decreases we prove global convergence for nonconvex smooth functions and r linear convergence for strongly convex functions for the l bfgs method and the unconstrained optimization problems in the cute library the new nonmonotone line search algorithm used fewer function and gradient evaluations on average than either the monotone or the traditional nonmonotone scheme

nonlinear rescaling and proximal like methods in convex optimization the nonlinear rescaling principle nrp consists of transforming the objective function and or the constraints of a given constrained optimization problem into another problem which is equivalent to the original one in the sense that their optimal set of solutions coincides a nonlinear transformation parameterized by a positive scalar parameter and based on a smooth sealing function is used to transform the constraints the methods based on nrp consist of sequential unconstrained minimization of the classical lagrangian for the equivalent problem followed by an explicit formula updating the lagrange multipliers we first show that the nrp leads naturally to proximal methods with an entropy like kernel which is defined by the conjugate of the scaling function and establish that the two methods are dually equivalent for convex constrained minimization problems we then study the convergence properties of the nonlinear rescaling algorithm and the corresponding entropy like proximal methods for convex constrained optimization problems special cases of the nonlinear rescaling algorithm are presented in particular a new class of exponential penalty modified barrier functions methods is introduced

integrated fisher linear discriminants this paper studies fisher linear discriminants flds based on classification accuracies for imbalanced datasets an optimal threshold is found out from a series of empirical formulas developed which is related not only to sample sizes but also to distribution regions a mixed binary decimal coding system is suggested to make the very dense datasets sparse and enlarge the class margins on condition that the neighborhood relationships of samples are nearly preserved the within class scatter matrices being or approximately singular should be moderately reduced in dimensionality but not added with tiny perturbations the weight vectors can be further updated by a kind of epoch limited three at most iterative learning strategy provided that the current training error rates come down accordingly putting the above ideas together this paper proposes a type of integrated flds the extensive experimental results over real world datasets have demonstrated that the integrated flds have obvious advantages over the conventional flds in the aspects of learning and generalization performances for the imbalanced datasets an optimal threshold is found out from a series of empirical threshold formulas developed for fisher linear discriminants based on classification accuracies weight vectors and thresholds are updated by an epoch limited iterative learning strategy the singular within class scatter matrices are reduced in dimensionality but not added with perturbations a coding system enlarges class margins and approximately preserves neighborhood relationships an integrated learning algorithm improves the learning and generalization performances of fisher linear discriminants

principal component analysis combined with truncated newton minimization for dimensionality reduction of chemical databases the similarity and diversity sampling problems are two challenging optimization tasks that arise in the analysis of chemical databases as a first step to their solution we propose an efficient projection refinement protocol based on the principal component analysis pca and the truncated newton minimization method implemented by our package tnpack pca tnpack we show that pca can provide the same initial guess as the singular value decomposition svd for the optimization task of solving the distance geometry optimization problem if each column of a database matrix has a mean of zero hence our pca tnpack approach is analogous to the svd tnpack projection refinement protocol that we developed recently for visualizing large chemical databases using pca tnpack and the merck mddr database mdl drug data report we further investigate the projection refinement procedure with regards to the preservation of the original clusters of chemical compounds the accuracy of similarity and diversity sampling of chemical compounds and the potential application in the study of structure activity relationships we also explore by simple experiments accuracy and efficiency aspects of the pca tnpack procedure compared to those of a global optimization algorithm simulated annealing as implemented by the program package simann in terms of producing the projection mapping of a database numerical results show that the 2d pca tnpack mapping can preserve the distance relationships of the original database and is thus valuable as a first step in similarity and diversity applications of course the generation of a global rather than local minimizer and its interpretation in terms of pharameceutical applications remains a challenge since all numerical tests are performed on the merck mddr database results are representative of realistic cases encountered in the field of drug design and may help analyze properties of medicinal compounds

partitioned variable metric updates for large structured optimization problems this paper presents a minimization method based on the idea of partitioned updating of the hessian matrix in the case where the objective function can be decomposed in a sum of convex element functions this situation occurs in a large class of practical problems including nonlinear finite elements calculations some theoretical and algorithmic properties of the update are discussed and encouraging numerical results are presented

asymptotic analysis for penalty and barrier methods in convex and linear programming we consider a wide class of penalty and barrier methods for convex programming which includes a number of specific functions proposed in the literature we provide a systematic way to generate penalty and barrier functions in this class and we analyze the existence of primal and dual optimal paths generated by these penalty methods as well as their convergence to the primal and dual optimal sets for linear programming we prove that these optimal paths converge to single points

coupling general penalty schemes for convex programming with the steepest descent and the proximal point algorithm we study the asymptotic behavior of the continuous flow generated by coupling the steepest descent method with a general class of penalty schemes for convex programming we establish convergence of the trajectories towards primal optimal solutions as well as convergence of some naturally associated dual paths the results are extended to the sequences generated by an implicit discretization scheme which corresponds to the coupling of an inexact proximal point iteration with the penalty schemes

renewable estimation and incremental inference in generalized linear models with streaming data sets the paper presents an incremental updating algorithm to analyse streaming data sets using generalized linear models the method proposed is formulated within a new framework of renewable estimation and incremental inference in which the maximum likelihood estimator is renewed with current data and summary statistics of historical data our framework can be implemented within a popular distributed computing environment known as apache spark to scale up computation consisting of two data processing layers the rho architecture enables us to accommodate inference related statistics and to facilitate sequential updating of the statistics used in both estimation and inference we establish estimation consistency and asymptotic normality of the proposed renewable estimator in which the wald test is utilized for an incremental inference our methods are examined and illustrated by various numerical examples from both simulation experiments and a real world data analysis

adjoint based acoustic inversion for the physical characterization of a shallow water environment recently the concept of adjoint modeling has been introduced in shallow water acoustics for solving inverse problems analytical adjoints have been derived for normal modes and for both the standard parabolic equation and claerbout s wide angle approximation wape this paper proposes the application of a semiautomatic adjoint approach that has been successfully applied in the past for multidimensional variational data assimilation in meteorological and climate modeling starting from a modular graph representation of the underlying forward model a programming tool facilitates the generation and coding of both the tangent linear and the adjoint models the potential of this numerical adjoint approach for the physical characterization of a shallow water environment is illustrated with two applications for geoacoustic inversion and ocean acoustic tomography using claerbout s wape in combination with nonlocal boundary conditions furthermore the adjoint optimization is extended to multiple frequencies and

on dual convergence of the generalized proximal point method with bregman distances the use of generalized distances e g bregman distances instead of the euclidean one in the proximal point method for convex optimization allows for elimination of the inequality constraints from the subproblems in this paper we consider the proximal point method with bregman distances applied to linearly constrained convex optimization problems and study the behavior of the dual sequence obtained from the optimal multipliers of the linear constraints of each subproblem under rather general assumptions which cover most bregman distances of interest we obtain an ergodic convergence result namely that a sequence of weighted averages of the dual sequence converges to a specific point of the dual optimal set as an intermediate result we prove under the same assumptions that the dual central path generated by a large class of barriers including the generalized bregman distances converges to the same point

global convergence of online limited memory bfgs global convergence of an online stochastic limited memory version of the broyden fletcher goldfarb shanno bfgs quasi newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established lower and upper bounds on the hessian eigenvalues of the sample functions are shown to suffice to guarantee that the curvature approximation matrices have bounded determinants and traces which in turn permits establishing convergence to optimal arguments with probability 1 experimental evaluation on a search engine advertising problem showcase reductions in convergence time relative to stochastic gradient descent algorithms

latent log linear models for handwritten digit classification we present latent log linear models an extension of log linear models incorporating latent variables and we propose two applications thereof log linear mixture models and image deformation aware log linear models the resulting models are fully discriminative can be trained efficiently and the model complexity can be controlled log linear mixture models offer additional flexibility within the log linear modeling framework unlike previous approaches the image deformation aware model directly considers image deformations and allows for a discriminative training of the deformation parameters both are trained using alternating optimization for certain variants convergence to a stationary point is guaranteed and in practice even variants without this guarantee converge and find models that perform well we tune the methods on the usps data set and evaluate on the mnist data set demonstrating the generalization capabilities of our proposed models our models although using significantly fewer parameters are able to obtain competitive results with models proposed in the literature

accelerated conjugate gradient algorithm with finite difference hessian vector product approximation for unconstrained optimization in this paper we propose a fundamentally different conjugate gradient method in which the well known parameter b k is computed by an approximation of the hessian vector product through finite differences for search direction computation the method uses a forward difference approximation to the hessian vector product in combination with a careful choice of the finite difference interval for the step length computation we suggest an acceleration scheme able to improve the efficiency of the algorithm under common assumptions the method is proved to be globally convergent it is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear but the reduction in function values is significantly improved numerical comparisons with conjugate gradient algorithms including conmin by shanno and phua d f shanno k h phua algorithm 500 minimization of unconstrained multivariate functions acm trans math softw 2 1976 87 94 scalcg by andrei n andrei scaled conjugate gradient algorithms for unconstrained optimization comput optim appl 38 2007 401 416 n andrei scaled memoryless bfgs preconditioned conjugate gradient algorithm for unconstrained optimization optim methods softw 22 2007 561 571 n andrei a scaled bfgs preconditioned conjugate gradient algorithm for unconstrained optimization appl math lett 20 2007 645 650 and new conjugacy condition and related new conjugate gradient by li tang and wei g li c tang z wei new conjugacy condition and related new conjugate gradient methods for unconstrained optimization j comput appl math 202 2007 523 539 or truncated newton tn by nash s g nash preconditioning of truncated newton methods siam j on scientific and statistical computing 6 1985 599 616 using a set of 750 unconstrained optimization test problems show that the suggested algorithm outperforms these conjugate gradient algorithms as well as tn

quadratic projection based feature extraction with its application to biometric recognition this paper presents a novel quadratic projection based feature extraction framework where a set of quadratic matrices is learned to distinguish each class from all other classes we formulate quadratic matrix learning qml as a standard semidefinite programming sdp problem however the conventional interior point sdp solvers do not scale well to the problem of qml for high dimensional data to solve the scalability of qml we develop an efficient algorithm termed dualqml based on the lagrange duality theory to extract nonlinear features to evaluate the feasibility and effectiveness of the proposed framework we conduct extensive experiments on biometric recognition experimental results on three representative biometric recognition tasks including face palmprint and ear recognition demonstrate the superiority of the dualqml based feature extraction algorithm compared to the current state of the art algorithms highlightsa novel quadratic projection based feature extraction framework is developed a set of quadratic matrices is learnt to distinguish each class from other classes quadratic matrix learning qml is formulated as an sdp problem an efficient algorithm is developed to solve qml based on lagrange duality theory experiments on biometric recognition show the effectiveness of our algorithm

on the convergence of the exponential multiplier method for convex programming in this paper we analyze the exponential method of multipliers for convex constrained minimization problems which operates like the usual augmented lagrangian method except that it uses an exponential penalty function in place of the usual quadratic we also analyze a dual counterpart the entropy minimization algorithm which operates like the proximal minimization algorithm except that it uses a logarithmic entropy proximal term in place of a quadratic we strengthen substantially the available convergence results for these methods and we derive the convergence rate of these methods when applied to linear programs

a limited memory quasi newton trust region method for box constrained optimization by means of wolfe conditions strategy we propose a quasi newton trust region method to solve box constrained optimization problems this method is an adequate combination of the compact limited memory bfgs and the trust region direction while the generated point satisfies the wolfe conditions and therefore maintains a positive definite approximation to the hessian of the objective function the global convergence and the quadratic convergence of this method are established under suitable conditions finally we compare our algorithms iwtral and ibwtral with an active set trust region algorithm astral xu and burke 2007 on the cutest box constrained test problems presented by gould et al 2015 numerical results show that the presented method is competitive and totally interesting for solving box constrained optimization

bfgs with update skipping and varying memory we give conditions under which limited memory quasi newton methods with exact line searches will terminate in n steps when minimizing n dimensional quadratic functions we show that although all broyden family methods terminate in n steps in their full memory versions only bfgs does so with limited memory additionally we show that full memory broyden family methods with exact line searches terminate in at most n p steps when p matrix updates are skipped we introduce new limited memory bfgs variants and test them on nonquadratic minimization problems

learning harmonium models with infinite latent features undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks one common challenge in learning such models is to determine the number of hidden units that are unknown a priori although bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed bayesian networks very little effort has been made toward applying bayesian nonparametrics to learn undirected latent variable models in this paper we present the infinite exponential family harmonium iefh a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool we also present two important extensions of iefh to 1 multiview iefh for dealing with heterogeneous data and 2 infinite maximum margin harmonium immh for incorporating supervising side information to learn predictive latent features we develop variational inference algorithms to learn model parameters our methods are computationally competitive because of the avoidance of selecting the number of latent units our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iefh and immh inherited from bayesian nonparametrics and max margin learning such results were not available until now and contribute to expanding the scope of bayesian nonparametrics to learn the structures of undirected latent variable models