a time parallel approach to strong constraint four dimensional variational data assimilation a parallel in time algorithm based on an augmented lagrangian approach is proposed to solve four dimensional variational 4d var data assimilation problems the assimilation window is divided into multiple sub intervals that allows parallelization of cost function and gradient computations the solutions to the continuity equations across interval boundaries are added as constraints the augmented lagrangian approach leads to a different formulation of the variational data assimilation problem than the weakly constrained 4d var a combination of serial and parallel 4d vars to increase performance is also explored the methodology is illustrated on data assimilation problems involving the lorenz 96 and the shallow water models

limited memory bfgs systems with diagonal updates abstract in this paper we investigate a formula to solve systems of the form b i x y where b is a limited memory bfgs quasi newton matrix and is a positive constant these types of systems arise naturally in large scale optimization such as trust region methods as well as doubly augmented lagrangian methods we show that provided a simple condition holds on b 0 and the system b i x y can be solved via a recursion formula that requires only vector inner products this formula has complexity m 2 n where m is the number of l bfgs updates and n m is the dimension of x

a tool for the analysis of quasi newton methods with application to unconstrained minimization the bfgs update formula is shown to have an important property that is inde pendent of the algorithmic context of the update and that is relevant to both constrained and unconstrained optimization the bfgs method for unconstrained optimization using a variety of line searches including backtracking is shown to be globally and superlinearly convergent on uniformly convex problems the analysis is particularly simple due to the use of some new tools introduced in this paper

action recognition based on overcomplete independent components analysis existing works on action recognition rely on two separate stages 1 designing hand designed features or learning features from video data 2 classifying features using a classifier such as svm or adaboost motivated by two observations 1 independent component analysis ica is capable of encoding intrinsic features underlying video data and 2 videos of different actions can be easily distinguished by their intrinsic features we propose a simple but effective action recognition framework based on the recently proposed overcomplete ica model after a set of overcomplete ica basis functions are learned from the densely sampled 3d patches from training videos for each action a test video is classified as the class whose basis functions can reconstruct the sampled 3d patches from the test video with the smallest reconstruction error the experimental results on five benchmark datasets demonstrate that the proposed approach outperforms several state of the art works

a network design problem for a distribution system with uncertain demands a class of production distribution planning problems with nonstochastic uncertain demands is modeled as a dynamic game between two players who control flows on a network with node and arc capacity constraints simple conditions are derived for determining which player wins the game these conditions are then used to design a minimum cost network with the property that its feasible control strategies are allowed to meet the demand without violating the capacity constraints

some numerical experiments with variable storage quasi newton algorithms this paper describes some numerical experiments with variable storage quasi newton methods for the optimization of some large scale models coming from fluid mechanics and molecular biology in addition to assessing these kinds of methods in real life situations we compare an algorithm of a buckley with a proposal by j nocedal the latter seems generally superior provided that careful attention is given to some nontrivial implementation aspects which concern the general question of properly initializing a quasi newton matrix in this context we find it appropriate to use a diagonal matrix generated by an update of the identity matrix so as to fit the rayleigh ellipsoid of the local hessian in the direction of the change in the gradient also a variational derivation of some rank one and rank two updates in hilbert spaces is given

reduced order modeling for time dependent optimization problems with initial value controls this paper presents a new reduced order model rom hessian approximation for linear quadratic optimal control problems where the optimal control is the initial value such problems arise in parameter identification and data assimilation where the parameters to be identified appear in the initial data and also as subproblems in multiple shooting formulations of more general optimal control problems the new rom hessians can provide a substantially better approximation than the underlying basic rom approximation and thus can substantially reduce the computing time needed to solve these optimal control problems the computation of a hessian vector product requires the solution of the linearized state equation with initial value given by the vector to which the hessian is applied followed by the solution of the second order adjoint equation projection based roms of these two linear differential equations are used to generate the hessian approximation while the objective function and gradient are compute

parallel lagrange newton krylov schur methods for pde constrained optimization part i the krylov schur solver large scale optimization of systems governed by partial differential equations pdes is a frontier problem in scientific computation reduced quasi newton sequential quadratic programming sqp methods are state of the art approaches for such problems these methods take full advantage of existing pde solver technology and parallelize well however their algorithmic scalability is questionable for certain problem classes they can be very slow to converge in this two part article we propose a new method for steady state pde constrained optimization based on the idea of using a full space newton solver combined with an approximate reduced space quasi newton sqp preconditioner the basic components of the method are newton solution of the first order optimality conditions that characterize stationarity of the lagrangian function krylov solution of the karush kuhn tucker kkt linear systems arising at each newton iteration using a symmetric quasi minimum residual method preconditioning of the kkt system using an approximate state decision variable decomposition that replaces the forward pde jacobians by their own preconditioners and the decision space schur complement the reduced hessian by a bfgs approximation initialized by a two step stationary method accordingly we term the new method it lagrange newton krylov schur lnks it is fully parallelizable exploits the structure of available parallel algorithms for the pde forward problem and is locally quadratically convergent in part i of this two part article we investigate the effectiveness of the kkt linear system solver we test our method on two optimal control problems in which the state constraints are described by the steady state stokes equations the objective is to minimize dissipation or the deviation from a given velocity field the control variables are the boundary velocities numerical experiments on up to 256 cray t3e processors and on an sgi origin 2000 include scalability and performance assessment of the lnks algorithm and comparisons with reduced sqp for up to 1 000 000 state and 50 000 decision variables in part ii of the article we address globalization and inexactness issues and apply lnks to the optimal control of the steady incompressible navier stokes equations

a algorithm for convex minimization for convex minimization we introduce an algorithm based on space decomposition the method uses a bundle subroutine to generate a sequence of approximate proximal points when a primal dual track leading to a solution and zero subgradient pair exists these points approximate the primal track points and give the algorithm s or corrector steps the subroutine also approximates dual track points that are gradients needed for the method s newton predictor steps with the inclusion of a simple line search the resulting algorithm is proved to be globally convergent the convergence is superlinear if the primal dual track points and the objective s hessian are approximated well enough

adjoint sensitivity analysis of regional air quality models the task of providing an optimal analysis of the state of the atmosphere requires the development of efficient computational tools that facilitate an efficient integration of observational data into models in a variational approach the data assimilation problem is posed as a minimization problem which requires the sensitivity derivatives of a cost functional with respect to problem parameters the direct decoupled method has been extensively applied for sensitivity studies of air pollution adjoint sensitivity is a complementary approach which efficiently calculates the derivatives of a functional with respect to a large number of parameters in this paper we discuss the mathematical foundations of the adjoint sensitivity method applied to air pollution models and present a complete set of computational tools for performing three dimensional adjoint sensitivity studies numerical examples show that three dimensional adjoint sensitivity analysis provides information on influence areas which cannot be obtained solely by an inverse analysis of the meteorological fields several illustrative data assimilation results in a twin experiments framework as well as the assimilation of a real data set are also presented

flows on hypergraphs we consider the capacitated minimum cost flow problem on directed hypergraphs we define spanning hypertrees so generalizing the spanning tree of a standard graph and show that like in the standard and in the generalized minimum cost flow problems a correspondence exists between bases and spanning hypertrees then we show that like for the network simplex algorithms for the standard and for the generalized minimum cost flow problems most of the computations performed at each pivot operation have direct hypergraph interpretations

parameter and state model reduction for large scale statistical inverse problems a greedy algorithm for the construction of a reduced model with reduction in both parameter and state is developed for an efficient solution of statistical inverse problems governed by partial differential equations with distributed parameters large scale models are too costly to evaluate repeatedly as is required in the statistical setting furthermore these models often have high dimensional parametric input spaces which compounds the difficulty of effectively exploring the uncertainty space we simultaneously address both challenges by constructing a projection based reduced model that accepts low dimensional parameter inputs and whose model evaluations are inexpensive the associated parameter and state bases are obtained through a greedy procedure that targets the governing equations model outputs and prior information the methodology and results are presented for groundwater inverse problems in one and two dimensions

adjoint implementation of rosenbrock methods applied to variational data assimilation problems abstract in the past decade the variational method has been successfully applied in data assimilation problems for atmospheric chemistry models in 4d var data assimilation a minimization algorithm is used to find the set of control variables which minimizes the weighted least squares distance between model predictions and observations over the assimilation window using the adjoint method the gradient of the cost function can be computed fast at the expense of few function evaluations making the optimization process very efficient for large scale models the high storage requirements and the difficulty of implementing the adjoint code when sophisticated integrators are used to solve the stiff chemistry make the assimilation a very intensive computational process if the sparse structure of the chemical models is carefully exploited rosenbrock methods have been proved to be reliable chemistry solvers because of their outstanding stability properties and conservation of the linear invariants of the system in this paper we present an efficient implementation of the adjoint code for the rosenbrock type methods which can reduce the storage requirements of the forward model and is suitable for automatization the adjoint code is completely generated using symbolic preprocessing and automatic differentiation tools which allow flexibility and require minimal user intervention

the cone of nonnegative c numerical range and its preservers abstract let h be a complex hilbert space and b h the banach space of all bounded linear operators on h let c c 1 c k where c 1 c k 0 for each a b h the c numerical range of a is the set w c a j 1 k c j a x j x j x 1 x k is an orthonormal set in h in this note the cone p c a b h w c a 0 of operators with nonnegative c numerical ranges is considered we first determine the extreme directions of p c then we show that all bijective linear mappings t b h b h such that t p c p c are positive multiples of jordan isomorphisms as an application the result is used to obtain a description of surjective mappings on b h that preserve the c numerical radius distance

fatode a library for forward adjoint and tangent linear integration of odes fatode is a fortran library for the integration of ordinary differential equations with direct and adjoint sensitivity analysis capabilities the paper describes the capabilities implementation code organization and usage of this package fatode implements four families of methods explicit runge kutta for nonstiff problems fully implicit runge kutta singly diagonally implicit runge kutta and rosenbrock for stiff problems each family contains several methods with different orders of accuracy users can add new methods by simply providing their coefficients for each family the forward adjoint and tangent linear models are implemented general purpose solvers for dense and sparse linear algebra are used users can easily incorporate problem tailored linear algebra routines the performance of the package is demonstrated on several test problems to the best of our knowledge fatode is the first publicly available general purpose package that offers forward and adjoint sensitivity analysis capabi

runge kutta methods in optimal control and the transformed adjoint system the convergence rate is determined for runge kutta discretizations of nonlinear control problems the analysis utilizes a connection between the kuhn tucker multipliers for the discrete problem and the adjoint variables associated with the continuous minimum principle this connection can also be exploited in numerical solution techniques that require the gradient of the discrete cost function

on the limited memory bfgs method for large scale optimization we study the numerical performance of a limited memory quasi newton method for large scale optimization which we call the l bfgs method we compare its performance with that of the method developed by buckley and lenir 1985 which combines cycles of bfgs steps and conjugate direction steps our numerical tests indicate that the l bfgs method is faster than the method of buckley and lenir and is better able to use additional storage to accelerate convergence we show that the l bfgs method can be greatly accelerated by means of a simple scaling we then compare the l bfgs method with the partitioned quasi newton method of griewank and toint 1982a the results show that for some problems the partitioned quasi newton method is clearly superior to the l bfgs method however we find that for other problems the l bfgs method is very competitive due to its low iteration cost we also study the convergence properties of the l bfgs method and prove global convergence on uniformly convex problems

qn like variable storage conjugate gradients both conjugate gradient and quasi newton methods are quite successful at minimizing smooth nonlinear functions of several variables and each has its advantages in particular conjugate gradient methods require much less storage to implement than a quasi newton code and therefore find application when storage limitations occur they are however slower so there have recently been attempts to combine cg and qn algorithms so as to obtain an algorithm with good convergence properties and low storage requirements one such method is the code conmin due to shanno and phua it has proven quite successful but it has one limitation it has no middle ground in that it either operates as a quasi newton code using o n 2 storage locations or as a conjugate gradient code using 7n locations but it cannot take advantage of the not unusual situation where more than 7n locations are available but a quasi newton code requires an excessive amount of storage

discrete second order adjoints in atmospheric chemical transport modeling atmospheric chemical transport models ctms are essential tools for the study of air pollution for environmental policy decisions for the interpretation of observational data and for producing air quality forecasts many air quality studies require sensitivity analyses i e the computation of derivatives of the model output with respect to model parameters the derivatives of a cost functional defined on the model output with respect to a large number of model parameters can be calculated efficiently through adjoint sensitivity analysis while the traditional first order adjoint models give the gradient of the cost functional with respect to parameters second order adjoint models give second derivative information in the form of products between the hessian of the cost functional and a vector representing a perturbation in sensitivity analysis a search direction in optimization an eigenvector etc in this paper we discuss the mathematical foundations of the discrete second order adjoint sensitivity method and present a complete set of computational tools for performing second order sensitivity studies in three dimensional atmospheric ctms the tools include discrete second order adjoints of runge kutta and of rosenbrock time stepping methods for stiff equations together with efficient implementation strategies numerical examples illustrate the use of these computational tools in important applications like sensitivity analysis optimization uncertainty quantification and the calculation of directions of maximal error growth in three dimensional atmospheric ctms

a derivative free trust region framework for variational data assimilation this study develops a hybrid ensemble variational approach for solving data assimilation problems the method called tr 4d enkf is based on a trust region framework and consists of three computational steps first an ensemble of model runs is propagated forward in time and snapshots of the state are stored next a sequence of basis vectors is built and a low dimensional representation of the data assimilation system is obtained by projecting the model state onto the space spanned by the ensemble perturbations finally the low dimensional optimization problem is solved in the reduced space using a trust region approach the size of the trust region is updated according to the relative decrease of the reduced order surrogate cost function the analysis state is projected back onto the full space and the process is repeated with the current analysis serving as a new background a heuristic approach based on the trust region size is proposed in order to adjust the background error statistics from one iteration to the next experimental simulations are carried out using the atmospheric general circulation model speedy the results show that tr 4d enkf is an efficient computational approach and is more accurate than the current state of the art 4d enkf implementations such as the pod 4d enkf and the iterative subspace minimization methods

a discrete newton algorithm for minimizing a function of many variables a newton like method is presented for minimizing a function ofn variables it uses only function and gradient values and is a variant of the discrete newton algorithm this variant requires fewer operations than the standard method whenn 39 and storage is proportional ton rather thann2

conditions for the generalized numerical range to be real abstract necessary and sufficient conditions are given for the c numerical range of a matrix a to be a subset of the real axis in particular it is shown that both a and c must be translates of hermitian matrices

on operators preserving the numerical range abstract let f be a surjective linear mapping between the algebras l h and l k of all bounded operators on nontrivial complex hilbert spaces h and k respectively for any positive integer k let wk a denote the kth numerical range of an operator a on h if k is strictly less than one half the dimension of h and wk f a wk a for all a from l h then there is a unitary mapping u h k such that either f a uau or f a uau t for every a l h where the transposition is taken in any basis of k fixed in advance this generalizes the result of s pierce and w watkins on finite dimensional spaces the case of k greater than or equal to one half of the dimension of h is also treated using our method our proofs depend on a characterization of those linear operators preserving projections of rank one which is of independent interest

the image containment problem and some classes of polynomial instances the image containment problem icp is a minimum cost design problem concerning the containment of particular polyhedra called zonotopes that are images of boxes through linear transformations the icp is np hard here we study a family of nontrivial icp instances called worst case demand wcd instances we prove that such instances can be recognized and solved in polynomial time via linear programming then we characterize the classes of instances that are wcd independently on the choice of the cost vector structurally worst case demand classes swcd and we show that recognizing whether a class of instances is swcd is a conp complete problem finally we describe two families of swcd classes that are interesting from an applicative point of view the classes defined by the incidence matrices of particular directed graphs and those defined by pre leontief matrices

partitioned variable metric updates for large structured optimization problems this paper presents a minimization method based on the idea of partitioned updating of the hessian matrix in the case where the objective function can be decomposed in a sum of convex element functions this situation occurs in a large class of practical problems including nonlinear finite elements calculations some theoretical and algorithmic properties of the update are discussed and encouraging numerical results are presented

forward and adjoint sensitivity analysis with continuous explicit runge kutta schemes abstract we study the numerical solution of tangent linear first and second order adjoint models with high order explicit continuous runge kutta pairs the approaches currently implemented in popular packages such as sundials or daspkadjoint are based on linear multistep methods for adaptive time integration of nonlinear models interpolation of the forward model solution is required during the adjoint model simulation we propose to use the dense output mechanism built in the continuous runge kutta schemes as a highly accurate and cost efficient interpolation method in the inverse problem run we implement our approach in a fortran library called denserks which is found to compare well to other similar software on a number of test problems

quadratic projection based feature extraction with its application to biometric recognition this paper presents a novel quadratic projection based feature extraction framework where a set of quadratic matrices is learned to distinguish each class from all other classes we formulate quadratic matrix learning qml as a standard semidefinite programming sdp problem however the conventional interior point sdp solvers do not scale well to the problem of qml for high dimensional data to solve the scalability of qml we develop an efficient algorithm termed dualqml based on the lagrange duality theory to extract nonlinear features to evaluate the feasibility and effectiveness of the proposed framework we conduct extensive experiments on biometric recognition experimental results on three representative biometric recognition tasks including face palmprint and ear recognition demonstrate the superiority of the dualqml based feature extraction algorithm compared to the current state of the art algorithms highlightsa novel quadratic projection based feature extraction framework is developed a set of quadratic matrices is learnt to distinguish each class from other classes quadratic matrix learning qml is formulated as an sdp problem an efficient algorithm is developed to solve qml based on lagrange duality theory experiments on biometric recognition show the effectiveness of our algorithm

a limited memory quasi newton trust region method for box constrained optimization by means of wolfe conditions strategy we propose a quasi newton trust region method to solve box constrained optimization problems this method is an adequate combination of the compact limited memory bfgs and the trust region direction while the generated point satisfies the wolfe conditions and therefore maintains a positive definite approximation to the hessian of the objective function the global convergence and the quadratic convergence of this method are established under suitable conditions finally we compare our algorithms iwtral and ibwtral with an active set trust region algorithm astral xu and burke 2007 on the cutest box constrained test problems presented by gould et al 2015 numerical results show that the presented method is competitive and totally interesting for solving box constrained optimization

bfgs with update skipping and varying memory we give conditions under which limited memory quasi newton methods with exact line searches will terminate in n steps when minimizing n dimensional quadratic functions we show that although all broyden family methods terminate in n steps in their full memory versions only bfgs does so with limited memory additionally we show that full memory broyden family methods with exact line searches terminate in at most n p steps when p matrix updates are skipped we introduce new limited memory bfgs variants and test them on nonquadratic minimization problems

learning harmonium models with infinite latent features undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks one common challenge in learning such models is to determine the number of hidden units that are unknown a priori although bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed bayesian networks very little effort has been made toward applying bayesian nonparametrics to learn undirected latent variable models in this paper we present the infinite exponential family harmonium iefh a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool we also present two important extensions of iefh to 1 multiview iefh for dealing with heterogeneous data and 2 infinite maximum margin harmonium immh for incorporating supervising side information to learn predictive latent features we develop variational inference algorithms to learn model parameters our methods are computationally competitive because of the avoidance of selecting the number of latent units our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iefh and immh inherited from bayesian nonparametrics and max margin learning such results were not available until now and contribute to expanding the scope of bayesian nonparametrics to learn the structures of undirected latent variable models