on computing the distance to stability for matrices using linear dissipative hamiltonian systems abstract in this paper we consider the problem of computing the nearest stable matrix to an unstable one we propose new algorithms to solve this problem based on a reformulation using linear dissipative hamiltonian systems we show that a matrix a is stable if and only if it can be written as a j r q where j j t r 0 and q 0 that is r is positive semidefinite and q is positive definite this reformulation results in an equivalent optimization problem with a simple convex feasible set we propose three strategies to solve the problem in variables j r q i a block coordinate descent method ii a projected gradient descent method and iii a fast gradient method inspired from smooth convex optimization these methods require o n 3 operations per iteration where n is the size of a we show the effectiveness of the fast gradient method compared to the other approaches and to several state of the art algorithms

stochastic first and zeroth order methods for nonconvex stochastic programming in this paper we introduce a new stochastic approximation type algorithm namely the randomized stochastic gradient rsg method for solving an important class of nonlinear possibly nonconvex stochastic programming problems we establish the complexity of this method for computing an approximate stationary point of a nonlinear programming problem we also show that this method possesses a nearly optimal rate of convergence if the problem is convex we discuss a variant of the algorithm which consists of applying a postoptimization phase to evaluate a short list of solutions generated by several independent runs of the rsg method and we show that such modification allows us to improve significantly the large deviation properties of the algorithm these methods are then specialized for solving a class of simulation based optimization problems in which only stochastic zeroth order information is available

variance based extragradient methods with line search for stochastic variational inequalities in this paper we propose dynamic sampled stochastic approximated ds sa extragradient methods for stochastic variational inequalities svis that are robust with respect to an unknown lipschitz c

convergence rates of inertial forward backward algorithms in a hilbert space mathcal h given alpha k a general sequence of nonnegative numbers we analyze the convergence properties of the inertial forward backward algorithm ifb quad left begin array l y k x k alpha k x k x k 1 x k 1 mbox prox s psi y k s nabla phi y k end array right where psi mathcal h to mathbb r cup lbrace infty rbrace is a proper lower semicontinuous convex function and phi mathcal h to mathbb r is a differentiable convex function whose gradient is lipschitz continuous various options for the sequence alpha k are considered in the literature among them the nesterov choice leads to the fista algorithm and accelerates convergence from mathcal o 1 k to mathcal o 1 k 2 for the values several variants are used to guarantee the convergence of the iterates or also to improve the rate of convergence for the values for the design of fast optimization methods the tuning of the sequence alpha k is a subtle issue which we deal with in this paper in general we show that the convergence rate of the algorithm can be obtained simply by analyzing the sequence of positive real numbers alpha k in addition to the case alpha k 1 frac alpha k with alpha geq 3 our results apply equally well to alpha k 1 frac alpha k r with an exponent r in 0 1 and to polyak s heavy ball method thus we unify most of the existing results based on the accelerated gradient method of nesterov in the process we improve some of them and discover new ones

robust stochastic approximation approach to stochastic programming in this paper we consider optimization problems where the objective function is given in a form of the expectation a basic difficulty of solving such stochastic optimization problems is that the involved multidimensional integrals expectations cannot be computed with high accuracy the aim of this paper is to compare two computational approaches based on monte carlo sampling techniques namely the stochastic approximation sa and the sample average approximation saa methods both approaches the sa and saa methods have a long history current opinion is that the saa method can efficiently use a specific say linear structure of the considered problem while the sa approach is a crude subgradient method which often performs poorly in practice we intend to demonstrate that a properly modified sa approach can be competitive and even significantly outperform the saa method for a certain class of convex stochastic problems we extend the analysis to the case of convex concave stochastic saddle point problems and present in our opinion highly encouraging results of numerical experiments

an optimal method for stochastic composite optimization this paper considers an important class of convex programming cp problems namely the stochastic composite optimization sco whose objective function is given by the summation of general nonsmooth and smooth stochastic components since sco covers non smooth smooth and stochastic cp as certain special cases a valid lower bound on the rate of convergence for solving these problems is known from the classic complexity theory of convex programming note however that the optimization algorithms that can achieve this lower bound had never been developed in this paper we show that the simple mirror descent stochastic approximation method exhibits the best known rate of convergence for solving these problems our major contribution is to introduce the accelerated stochastic approximation ac sa algorithm based on nesterov s optimal method for smooth cp nesterov in doklady an sssr 269 543 547 1983 nesterov in math program 103 127 152 2005 and show that the ac sa algorithm can achieve the aforementioned lower bound on the rate of convergence for sco to the best of our knowledge it is also the first universally optimal algorithm in the literature for solving non smooth smooth and stochastic cp problems we illustrate the significant advantages of the ac sa algorithm over existing methods in the context of solving a special but broad class of stochastic programming problems

on weighted integrability of double sine series abstract we introduce a new class of double sequences d g m r called double general monotone and some new class of weight functions to study the weighted integrability of double sine series some results of d yu p zhou and s zhou are also generalized

accelerated gradient methods for nonconvex nonlinear and stochastic programming in this paper we generalize the well known nesterov s accelerated gradient ag method originally designed for convex smooth optimization to solve nonconvex and possibly stochastic optimization problems we demonstrate that by properly specifying the stepsize policy the ag method exhibits the best known rate of convergence for solving general nonconvex smooth optimization problems by using first order information similarly to the gradient descent method we then consider an important class of composite optimization problems and show that the ag method can solve them uniformly i e by using the same aggressive stepsize policy as in the convex case even if the problem turns out to be nonconvex we demonstrate that the ag method exhibits an optimal rate of convergence if the composite problem is convex and improves the best known rate of convergence if the problem is nonconvex based on the ag method we also present new nonconvex stochastic approximation methods and show that they can improve a few existing rates of convergence for nonconvex stochastic optimization to the best of our knowledge this is the first time that the convergence of the ag method has been established for solving nonconvex nonlinear programming in the literature

iteration complexity of first order augmented lagrangian methods for convex programming this paper considers a special class of convex programming cp problems whose feasible regions consist of a simple compact convex set intersected with an affine manifold we present first order methods for this class of problems based on an inexact version of the classical augmented lagrangian al approach where the subproblems are approximately solved by means of nesterov s optimal method we then establish a bound on the total number of nesterov s optimal iterations i e the inner iterations performed throughout the entire inexact al method to obtain a near primal dual optimal solution we also present variants with possibly better iteration complexity bounds than the original inexact al method which consist of applying the original approach directly to a perturbed problem obtained by adding a strongly convex component to the objective function of the cp problem

trigonometric series with general monotone coefficients we study trigonometric series with general monotone coefficients convergence results in the different metrics are obtained also we prove a hardy type result on the behavior of the series near the origin

misspecified nonconvex statistical optimization for sparse phase retrieval existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying true statistical models to address this issue we take a first step towards taming model misspecification by studying the high dimensional sparse phase retrieval problem with misspecified link functions in particular we propose a simple variant of the thresholded wirtinger flow algorithm that given a proper initialization linearly converges to an estimator with optimal statistical accuracy for a broad family of unknown link functions we further provide extensive numerical experiments to support our theoretical findings

stochastic solutions to evolution equations of non local branching processes abstract we give a probabilistic representation for the solution to a nonlinear evolution equation induced by a measure valued branching process we first construct the involved branching processes on the set of all finite configurations of a given set with a killing rate induced by a continuous additive functional and with a non local branching procedure given by a sequence of markovian kernels the main application is to prove stochastic aspects for a nonlinear evolution equation related to the neumann problem and the surface measure on the boundary which corresponds to the reflecting brownian motion as base movement taking the killing rate given by the local time on the boundary we use specific potential theoretical tools

a proximal method for composite minimization we consider minimization of functions that are compositions of convex or prox regular functions possibly extended valued with smooth vector functions a wide variety of important optimization problems fall into this framework we describe an algorithmic framework based on a subproblem constructed from a linearized approximation to the objective and a regularization term properties of local solutions of this subproblem underlie both a global convergence result and an identification property of the active manifold containing the solution of the original problem preliminary computational results on both convex and nonconvex examples are promising

approximating the nearest stable discrete time system abstract in this paper we consider the problem of stabilizing discrete time linear systems by computing a nearby stable matrix to an unstable one to do so we provide a new characterization for the set of stable matrices we show that a matrix a is stable if and only if it can be written as a s 1 u b s where s is positive definite u is orthogonal and b is a positive semidefinite contraction that is the singular values of b are less or equal to 1 this characterization results in an equivalent non convex optimization problem with a feasible set on which it is easy to project we propose a very efficient fast projected gradient method to tackle the problem in variables s u b and generate locally optimal solutions we show the effectiveness of the proposed method compared to other approaches

markov processes on the lipschitz boundary for the neumann and robin problems abstract we investigate the markov process on the boundary of a bounded lipschitz domain associated to the neumann and robin boundary value problems we first construct l p semigroups of sub markovian contractions on the boundary generated by the boundary conditions and we show that they are induced by the transition functions of the forthcoming processes as in the smooth boundary case the process on the boundary is obtained by the time change with the inverse of a continuous additive functional of the reflected brownian motion the robin problem is treated with a kato type l p perturbation method using the revuz correspondence an exceptional polar set occurs on the boundary we make the link with the dirichlet forms approach

optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i a generic algorithmic framework in this paper we present a generic algorithmic framework namely the accelerated stochastic approximation ac sa algorithm for solving strongly convex stochastic composite optimization sco problems while the classical stochastic approximation algorithms are asymptotically optimal for solving differentiable and strongly convex problems the ac sa algorithm when employed with proper stepsize policies can achieve optimal or nearly optimal rates of convergence for solving different classes of sco problems during a given number of iterations moreover we investigate these ac sa algorithms in more detail such as by establishing the large deviation results associated with the convergence rates and introducing an efficient validation procedure to check the accuracy of the generated solutions

on the complexity of steepest descent newton s and regularized newton s methods for nonconvex unconstrained optimization problems it is shown that the steepest descent and newton s methods for unconstrained nonconvex optimization under standard assumptions may both require a number of iterations and function evaluations arbitrarily close to o epsilon 2 to drive the norm of the gradient below epsilon this shows that the upper bound of o epsilon 2 evaluations known for the steepest descent is tight and that newton s method may be as slow as the steepest descent method in the worst case the improved evaluation complexity bound of o epsilon 3 2 evaluations known for cubically regularized newton s methods is also shown to be tight

nonlinear pdes and measure valued branching type processes abstract we deal with the probabilistic approach to a nonlinear operator of the form u u k 1 q k u k in connection with the works of m nagasawa n ikeda s watanabe and m l silverstein on the discrete branching processes instead of the laplace operator we may consider the generator of a right markov process called base process with a general not necessarily locally compact state space it turns out that solutions of the nonlinear equation u 0 are produced by the harmonic functions with respect to the linear generator of a discrete branching type process the consideration of the general state space allows to take as base process a measure valued superprocess in the sense of e b dynkin the probabilistic counterpart is a markov process which is a combination between a continuous branching process e g associated with a nonlinear operator of the form u u 1 2 and a discrete branching type one on a space of configurations of finite measures our approach uses probabilistic and analytic potential theoretical tools like the potential kernel of a continuous additive functional and the subordination operators

behavior of accelerated gradient methods near critical points of nonconvex functions we examine the behavior of accelerated gradient methods in smooth nonconvex unconstrained optimization focusing in particular on their behavior near strict saddle points accelerated methods are iterative methods that typically step along a direction that is a linear combination of the previous step and the gradient of the function evaluated at a point at or near the current iterate the previous step encodes gradient information from earlier stages in the iterative process we show by means of the stable manifold theorem that the heavy ball method is unlikely to converge to strict saddle points which are points at which the gradient of the objective is zero but the hessian has at least one negative eigenvalue we then examine the behavior of the heavy ball method and other accelerated gradient methods in the vicinity of a strict saddle point of a nonconvex quadratic function showing that both methods can diverge from this point more rapidly than the steepest descent method

measure valued branching processes associated with neumann nonlinear semiflows abstract we construct a measure valued branching markov process associated with a nonlinear boundary value problem where the boundary condition has a nonlinear pseudo monotone branching mechanism term which includes the case u u m with 0 m 1 the process is then used in the probabilistic representation of the solution of the parabolic problem associated with a nonlinear neumann boundary value problem in this way the classical association of the superprocesses to the dirichlet boundary value problems also holds for the nonlinear neumann boundary value problems it turns out that the obtained branching process behaves on the measures carried by the given open set like the linear continuous semiflow induced by the reflected brownian motion while the branching occurs on the measures having non zero traces on the boundary of the open set with the behavior of the superprocess having as spatial motion the process on the boundary associated to the reflected brownian motion

statistical inference for the population landscape via moment adjusted stochastic gradients modern statistical inference tasks often require iterative optimization methods to compute the solution convergence analysis from an optimization viewpoint informs us only how well the solution is approximated numerically but overlooks the sampling nature of the data in contrast recognizing the randomness in the data statisticians are keen to provide uncertainty quantification or confidence for the solution obtained by using iterative optimization methods the paper makes progress along this direction by introducing moment adjusted stochastic gradient descent a new stochastic optimization method for statistical inference we establish non asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees on the statistical front the theory allows for model misspecification with very mild conditions on the data for optimization the theory is flexible for both convex and non convex cases remarkably the moment adjusting idea motivated from error standardization in statistics achieves a similar effect to acceleration in first order optimization methods that are used to fit generalized linear models we also demonstrate this acceleration effect in the non convex setting through numerical experiments

optimal stochastic approximation algorithms for strongly convex stochastic composite optimization ii shrinking procedures and optimal algorithms in this paper we study new stochastic approximation sa type algorithms namely the accelerated sa ac sa for solving strongly convex stochastic composite optimization sco problems specifically by introducing a domain shrinking procedure we significantly improve the large deviation results associated with the convergence rate of a nearly optimal ac sa algorithm presented by ghadimi and lan in siam j optim 22 2012 pp 1469 1492 moreover we introduce a multistage ac sa algorithm which possesses an optimal rate of convergence for solving strongly convex sco problems in terms of the dependence on not only the target accuracy but also a number of problem parameters and the selection of initial points to the best of our knowledge this is the first time that such an optimal method has been presented in the literature from our computational results these ac sa algorithms can substantially outperform the classical sa and some other sa type algorithms for solving certain classes of strongly

on summability of fourier coefficients of functions from lebesgue space abstract in this paper we state new theorems of hardy littlewood type for functions with general monotone fourier coefficients sharpness of stated results is discusses

bounded and lp weak solutions for nonlinear equations of measure valued branching processes abstract we study the nonlinear schrodinger type equation c u c k 1 q k u k f where c is a bounded positive function we may replace the laplace operator with the generator of a borel right markov process with state space e and consider a nonlinear term induced by a perturbing kernel we solve this type of equation in an l p weak sense involving positive definite extensions of the solution from e to the set of all finite positive measures on e the extensions to the set of measures are related to the classical connection from the works of m nagasawa n ikeda s watanabe and m l silverstein between this nonlinear equation and the discrete branching markov processes we also investigate the existence of the bounded solutions for the dirichlet problem associated to the above equation with bounded boundary data our approach is based on probabilistic and analytic potential theoretical methods used on both spaces e and the set of all finite configurations of e like the revuz formula for the continuous additive functionals and the perturbation with kernels of the transition functions and resolvents

acceleration of stochastic approximation by averaging a new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated convergence with probability one is proved for a variety of classical optimization and identification problems it is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence

recursive trust region methods for multiscale nonlinear optimization a class of trust region methods is presented for solving unconstrained nonlinear and possibly nonconvex discretized optimization problems like those arising in systems governed by partial differential equations the algorithms in this class make use of the discretization level as a means of speeding up the computation of the step this use is recursive leading to true multilevel multiscale optimization methods reminiscent of multigrid methods in linear algebra and the solution of partial differential equations a simple algorithm of the class is then described and its numerical performance is shown to be numerically promising this observation then motivates a proof of global convergence to first order stationary points on the fine grid that is valid for all algorithms in the class