information theoretic determination of minimax rates of convergence we present some general results determining minimax bounds on statistical risk for density estimation based on certain information theoretic considerations these bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence

erratum swarming for faster convergence in stochastic optimization in this erratum we correct the statements and or proofs of lemma 4 6 theorem 4 7 and theorem 4 16 in s pu and a garcia siam j control optim 56 2018 pp 2997 3020 the main conclusion

stochastic first and zeroth order methods for nonconvex stochastic programming in this paper we introduce a new stochastic approximation type algorithm namely the randomized stochastic gradient rsg method for solving an important class of nonlinear possibly nonconvex stochastic programming problems we establish the complexity of this method for computing an approximate stationary point of a nonlinear programming problem we also show that this method possesses a nearly optimal rate of convergence if the problem is convex we discuss a variant of the algorithm which consists of applying a postoptimization phase to evaluate a short list of solutions generated by several independent runs of the rsg method and we show that such modification allows us to improve significantly the large deviation properties of the algorithm these methods are then specialized for solving a class of simulation based optimization problems in which only stochastic zeroth order information is available

on the motion of non homogeneous fluids in the presence of diffusion in this paper we study the motion of a non viscous fluid consisting of two components both incompressible say saturated salt water and water taking into consideration the diffusion among these components the equations of the model are obtained for example in frank and kamenetskii 6 and the viscous case is studied in ignat ev and kuznetsov 8 and kazhikhov and smagulov 9 lo for a presentation of this kind of problems see also kuznetsov 111 observe that in s lo the problem is solved only for fluids whose viscosity d is sufficiently large i e

on the fundamental limits of adaptive sensing suppose we can sequentially acquire arbitrary linear measurements of an n dimensional vector x resulting in the linear model y a x z where z represents measurement noise if the signal is known to be sparse one would expect the following folk theorem to be true choosing an adaptive strategy which cleverly selects the next row of a based on what has been previously observed should do far better than a nonadaptive strategy which sets the rows of a ahead of time thus not trying to learn anything about the signal in between observations this paper shows that the folk theorem is false we prove that the advantages offered by clever adaptive strategies and sophisticated estimation procedures no matter how intractable over classical compressed acquisition recovery schemes are in general minimal

sample complexity of solving non cooperative games this paper studies the complexity of solving two classes of non cooperative games in a distributed manner in which the players communicate with a set of system nodes over noisy communication channels the complexity of solving each game class is defined as the minimum number of iterations required to find a nash equilibrium ne of any game in that class with epsilon accuracy first we consider the class mathcal g of all n player non cooperative games with a continuous action space that admit at least one ne using information theoretic inequalities a lower bound on the complexity of solving mathcal g is derived which depends on the kolmogorov 2 epsilon capacity of the constraint set and the total capacity of the communication channels our results indicate that the game class mathcal g can be solved at most exponentially fast we next consider the class of all n player non cooperative games with at least one ne such that the players utility functions satisfy a certain differential constraint we derive lower bounds on the complexity of solving this game class under both gaussian and non gaussian noise models finally we derive upper and lower bounds on the sample complexity of a class of quadratic games it is shown that the complexity of solving this game class scales according to theta left frac 1 epsilon 2 right where epsilon is the accuracy parameter

a note on the realignment criterion for a quantum state in a bipartite system represented as a density matrix researchers used the realignment matrix and functions on its singular values to study the separability of the quantum state we obtain bounds for elementary symmetric functions of singular values of realignment matrices this answers some open problems proposed by lupo aniello and scardicchio as a consequence we show that the proposed scheme by these authors for testing separability would not work if the two subsystems of the bipartite system have the same dimension

information theoretic lower bounds on the oracle complexity of stochastic convex optimization relative to the large literature on upper bounds on complexity of convex optimization lesser attention has been paid to the fundamental hardn4516420ess of these problems given the extensive use of convex optimization in machine learning and statistics gaining an understanding of these complexity theoretic issues is important in this paper we study the complexity of stochastic convex optimization in an oracle model of computation we introduce a new notion of discrepancy between functions and use it to reduce problems of stochastic convex optimization to statistical parameter estimation which can be lower bounded using information theoretic methods using this approach we improve upon known results and obtain tight minimax complexity estimates for various function classes

robust stochastic approximation approach to stochastic programming in this paper we consider optimization problems where the objective function is given in a form of the expectation a basic difficulty of solving such stochastic optimization problems is that the involved multidimensional integrals expectations cannot be computed with high accuracy the aim of this paper is to compare two computational approaches based on monte carlo sampling techniques namely the stochastic approximation sa and the sample average approximation saa methods both approaches the sa and saa methods have a long history current opinion is that the saa method can efficiently use a specific say linear structure of the considered problem while the sa approach is a crude subgradient method which often performs poorly in practice we intend to demonstrate that a properly modified sa approach can be competitive and even significantly outperform the saa method for a certain class of convex stochastic problems we extend the analysis to the case of convex concave stochastic saddle point problems and present in our opinion highly encouraging results of numerical experiments

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

stochastic block mirror descent methods for nonsmooth and stochastic optimization in this paper we present a new stochastic algorithm namely the stochastic block mirror descent sbmd method for solving large scale nonsmooth and stochastic optimization problems the basic idea of this algorithm is to incorporate block coordinate decomposition and an incremental block averaging scheme into the classic stochastic mirror descent method in order to significantly reduce the cost per iteration of the latter algorithm we establish the rate of convergence of the sbmd method along with its associated large deviation results for solving general nonsmooth and stochastic optimization problems we also introduce variants of this method and establish their rate of convergence for solving strongly convex smooth and composite optimization problems as well as certain nonconvex optimization problems to the best of our knowledge all these developments related to the sbmd methods are new in the stochastic optimization literature moreover some of our results seem to be new for block coordinate descent methods for deterministic optimization

existence of global weak solutions for a 2d viscous shallow water equations and convergence to the quasi geostrophic model we consider a two dimensional viscous shallow water model with friction term existence of global weak solutions is obtained and convergence to the strong solution of the viscous quasi geostrophic equation with free surface term is proven in the well prepared case the ill prepared data case is also discussed

incompressible viscous flows in borderline besov spaces we establish two new estimates for a transport diffusion equation as an application we treat the problem of global persistence of the besov regularity b p 1 frac 2 p 1 with p in 2 infty for the two dimensional navier stokes equations with uniform bounds on the viscosity we provide also an inviscid global result

accelerated gradient methods for nonconvex nonlinear and stochastic programming in this paper we generalize the well known nesterov s accelerated gradient ag method originally designed for convex smooth optimization to solve nonconvex and possibly stochastic optimization problems we demonstrate that by properly specifying the stepsize policy the ag method exhibits the best known rate of convergence for solving general nonconvex smooth optimization problems by using first order information similarly to the gradient descent method we then consider an important class of composite optimization problems and show that the ag method can solve them uniformly i e by using the same aggressive stepsize policy as in the convex case even if the problem turns out to be nonconvex we demonstrate that the ag method exhibits an optimal rate of convergence if the composite problem is convex and improves the best known rate of convergence if the problem is nonconvex based on the ag method we also present new nonconvex stochastic approximation methods and show that they can improve a few existing rates of convergence for nonconvex stochastic optimization to the best of our knowledge this is the first time that the convergence of the ag method has been established for solving nonconvex nonlinear programming in the literature

exponentiated gradient versus gradient descent for linear predictors we consider two algorithm for on line prediction based on a linear model the algorithms are the well known gradient descent gd algorithm and a new algorithm which we call eg they both maintain a weight vector using simple updates for the gd algorithm the update is based on subtracting the gradient of the squared error made on a prediction the eg algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively we present worst case loss bounds for eg and compare them to previously known bounds for the gd algorithm the bounds suggest that the losses of the algorithms are in general incomparable but eg has a much smaller loss if only a few components of the input are relevant for the predictions we have performed experiments which show that our worst case upper bounds are quite tight already on simple artificial data

misspecified nonconvex statistical optimization for sparse phase retrieval existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying true statistical models to address this issue we take a first step towards taming model misspecification by studying the high dimensional sparse phase retrieval problem with misspecified link functions in particular we propose a simple variant of the thresholded wirtinger flow algorithm that given a proper initialization linearly converges to an estimator with optimal statistical accuracy for a broad family of unknown link functions we further provide extensive numerical experiments to support our theoretical findings

dynamic stochastic approximation for multi stage stochastic optimization in this paper we consider multi stage stochastic optimization problems with convex objectives and conic constraints at each stage we present a new stochastic first order method namely the dynamic stochastic approximation dsa algorithm for solving these types of stochastic optimization problems we show that dsa can achieve an optimal mathcal o 1 epsilon 4 rate of convergence in terms of the total number of required scenarios when applied to a three stage stochastic optimization problem we further show that this rate of convergence can be improved to mathcal o 1 epsilon 2 when the objective function is strongly convex we also discuss variants of dsa for solving more general multi stage stochastic optimization problems with the number of stages t 3 the developed dsa algorithms only need to go through the scenario tree once in order to compute an epsilon solution of the multi stage stochastic optimization problem as a result the memory required by dsa only grows linearly with respect to the number of stages to the best of our knowledge this is the first time that stochastic approximation type methods are generalized for multi stage stochastic optimization with t ge 3

low mach number limit of the full navier stokes equations the low mach number limit for classical solutions of the full navier stokes equations is here studied the combined effects of large temperature variations and thermal conduction are taken into account in particular we consider general initial data the equations lead to a singular problem depending on a small scaling parameter whose linearized system is not uniformly well posed yet it is proved that solutions exist and they are uniformly bounded for a time interval which is independent of the mach number ma 0 1 the reynolds number re 1 and the peclet number pe 1 based on uniform estimates in sobolev spaces and using a theorem of g metivier s schochet 30 we next prove that the penalized terms converge strongly to zero this allows us to rigorously justify at least in the whole space case the well known computations given in the introduction of p l lions book 26

conditional gradient type methods for composite nonlinear and stochastic optimization in this paper we present a conditional gradient type cgt method for solving a class of composite optimization problems where the objective function consists of a weakly smooth term and a strongly convex regularization term while including a strongly convex term in the subproblems of the classical conditional gradient method improves its rate of convergence it does not cost per iteration as much as general proximal type algorithms more specifically we present a unified analysis for the cgt method in the sense that it achieves the best known rate of convergence when the weakly smooth term is nonconvex and possesses nearly optimal complexity if it turns out to be convex while implementation of the cgt method requires explicitly estimating problem parameters like the level of smoothness of the first term in the objective function we also present a few variants of this method which relax such estimation unlike general proximal type parameter free methods these variants of the cgt method do not require any additional effort for computing sub gradients of the objective function and or solving extra subproblems at each iteration we then generalize these methods under stochastic setting and present a few new complexity results to the best of our knowledge this is the first time that such complexity results are presented for solving stochastic weakly smooth nonconvex and strongly convex optimization problems

on the well posedness of the incompressible density dependent euler equations in the lp framework abstract the present paper is devoted to the study of the well posedness issue for the density dependent euler equations in the whole space we establish local in time results for the cauchy problem pertaining to data in the besov spaces embedded in the set of lipschitz functions including the borderline case b p 1 n p 1 r n a continuation criterion in the spirit of the celebrated one by beale kato and majda 1984 in 2 for the classical euler equations is also proved in contrast with the previous work dedicated to this system in the whole space our approach is not restricted to the l 2 framework or to small perturbations of a constant density state we just need the density to be bounded away from zero the key to that improvement is a new a priori estimate in besov spaces for an elliptic equation with nonconstant coefficients

a separability criterion for density operators we give a necessary and sufficient condition for a mixed quantum mechanical state to be separable the criterion is formulated as a boundedness condition in terms of the greatest cross norm on the tensor product of trace class operators

on sampling rates in simulation based recursions we consider the context of simulation based recursions that is recursions that involve quantities needing to be estimated using a stochastic simulation examples include stochastic adaptations of fixed point and gradient descent recursions obtained by replacing function and derivative values appearing within the recursion by their monte carlo counterparts the primary motivating settings are simulation optimization and stochastic root finding problems where the low point and the zero of a function are sought respectively with only monte carlo estimates of the functions appearing within the problem we ask how much monte carlo sampling needs to be performed within simulation based recursions in order that the resulting iterates remain consistent and more importantly efficient where efficient implies convergence at the fastest possible rate answering these questions involves trading off two types of error inherent in the iterates the deterministic error due to recursion and the stochastic er

an alternative to em for gaussian mixture models batch and stochastic riemannian optimization we consider maximum likelihood estimation for gaussian mixture models gmm s this task is almost invariably solved in theory and practice via the expectation maximization em algorithm em owes its success to various factors of which is its ability to fulfill positive definiteness constraints in closed form is of key importance we propose an alternative to em grounded in the riemannian geometry of positive definite matrices using which we cast gmm parameter estimation as a riemannian optimization problem surprisingly such an out of the box riemannian formulation completely fails and proves much inferior to em this motivates us to take a closer look at the problem geometry and derive a better formulation that is much more amenable to riemannian optimization we then develop riemannian batch and stochastic gradient algorithms that outperform em often substantially we provide a non asymptotic convergence analysis for our stochastic method which is also the first to our knowledge such global analysis for riemannian stochastic gradient numerous empirical results are included to demonstrate the effectiveness of our methods

stochastic optimization using a trust region method and random models in this paper we propose and analyze a trust region model based algorithm for solving unconstrained stochastic optimization problems our framework utilizes random models of an objective function f x obtained from stochastic observations of the function or its gradient our method also utilizes estimates of function values to gauge progress that is being made the convergence analysis relies on requirements that these models and these estimates are sufficiently accurate with high enough but fixed probability beyond these conditions no assumptions are made on how these models and estimates are generated under these general conditions we show an almost sure global convergence of the method to a first order stationary point in the second part of the paper we present examples of generating sufficiently accurate random models under biased or unbiased noise assumptions lastly we present some computational results showing the benefits of the proposed method compared to existing approaches that are based on sample averaging or stochastic gradients

global convergence rate analysis of unconstrained optimization methods based on probabilistic models we present global convergence rates for a line search method which is based on random first order models and directions whose quality is ensured only with certain probability we show that in terms of the order of the accuracy the evaluation complexity of such a method is the same as its counterparts that use deterministic accurate models the use of probabilistic models only increases the complexity by a constant which depends on the probability of the models being good we particularize and improve these results in the convex and strongly convex case we also analyse a probabilistic cubic regularization variant that allows approximate probabilistic second order models and show improved complexity bounds compared to probabilistic first order methods again as a function of the accuracy the probabilistic cubic regularization bounds are of the same optimal order as for the deterministic case

optimal rates for zero order convex optimization the power of two function evaluations we consider derivative free algorithms for stochastic and nonstochastic convex optimization problems that use only function values rather than gradients focusing on nonasymptotic bounds on convergence rates we show that if pairs of function values are available algorithms for d dimensional optimization that use gradient estimates based on random perturbations suffer a factor of at most sqrt d in convergence rate over traditional stochastic gradient methods we establish such results for both smooth and nonsmooth cases sharpening previous analyses that suggested a worse dimension dependence and extend our results to the case of multiple m ge 2 evaluations we complement our algorithmic development with information theoretic lower bounds on the minimax convergence rate of such problems establishing the sharpness of our achievable results up to constant sometimes logarithmic factors

primal dual optimization algorithms over riemannian manifolds an iteration complexity analysis in this paper we study nonconvex and nonsmooth multi block optimization over euclidean embedded smooth riemannian submanifolds with coupled linear constraints such optimization problems naturally arise from machine learning statistical learning compressive sensing image processing and tensor pca among others by utilizing the embedding structure we develop an admm like primal dual approach based on decoupled solvable subroutines such as linearized proximal mappings where the duality is with respect to the embedded euclidean spaces first we introduce the optimality conditions for the afore mentioned optimization models then the notion of epsilon stationary solutions is introduced as a result the main part of the paper is to show that the proposed algorithms possess an iteration complexity of o 1 epsilon 2 to reach an epsilon stationary solution for prohibitively large size tensor or machine learning models we present a sampling based stochastic algorithm with the same iteration complexity bound in expectation in case the subproblems are not analytically solvable a feasible curvilinear line search variant of the algorithm based on retraction operators is proposed finally we show specifically how the algorithms can be implemented to solve a variety of practical problems such as the np hard maximum bisection problem the ell q regularized sparse tensor principal component analysis and the community detection problem our preliminary numerical results show great potentials of the proposed methods

stochastic quasi newton methods for nonconvex stochastic optimization in this paper we study stochastic quasi newton methods for nonconvex stochastic optimization where we assume that noisy information about the gradients of the objective function is available via a

the ordered subsets mirror descent optimization method with applications to tomography we describe an optimization problem arising in reconstructing three dimensional medical images from positron emission tomography pet a mathematical model of the problem based on the maximum likelihood principle is posed as a problem of minimizing a convex function of several million variables over the standard simplex to solve a problem of these characteristics we develop and implement a new algorithm ordered subsets mirror descent and demonstrate theoretically and computationally that it is well suited for solving the pet reconstruction problem

bipartite quantum systems on the realignment criterion and beyond inspired by the computable cross norm or realignment criterion we propose a new point of view about the characterization of the states of bipartite quantum systems we consider a schmidt decomposition of a bipartite density operator the corresponding schmidt coefficients or the associated symmetric polynomials are regarded as quantities that can be used to characterize bipartite quantum states in particular starting from the realignment criterion a family of necessary conditions for the separability of bipartite quantum states are derived we conjecture that these conditions which are weaker than the parent criterion can be strengthened in such a way to obtain a new family of criteria that are independent of the original one this conjecture is supported by numerical examples for the low dimensional cases these ideas can be applied to the study of quantum channels leading to a relation between the rate of contraction of a map and its ability to preserve entanglement

stochastic convex optimization with bandit feedback this paper addresses the problem of minimizing a convex lipschitz function f over a convex compact set mathcal x under a stochastic bandit i e noisy zeroth order feedback model in this model the algorithm is allowed to observe noisy realizations of the function value f x at any query point x in mathcal x the quantity of interest is the regret of the algorithm which is the sum of the function values at algorithm s query points minus the optimal function value we demonstrate a generalization of the ellipsoid algorithm that incurs widetilde mathcal o rm poly d sqrt t regret since any algorithm has regret at least omega sqrt t on this problem our algorithm is optimal in terms of the scaling with t

randomized smoothing for stochastic optimization we analyze convergence rates of stochastic optimization algorithms for nonsmooth convex optimization problems by combining randomized smoothing techniques with accelerated gradient methods we obtain convergence rates of stochastic optimization procedures both in expectation and with high probability that have optimal dependence on the variance of the gradient estimates to the best of our knowledge these are the first variance based rates for nonsmooth optimization we give several applications of our results to statistical estimation problems and provide experimental results that demonstrate the effectiveness of the proposed algorithms we also describe how a combination of our algorithm with recent work on decentralized optimization yields a distributed stochastic optimization algorithm that is order optimal

analysis of an inviscid zero mach number system in endpoint besov spaces for finite energy initial data abstract the present paper is the continuation of work 18 devoted to the study of an inviscid zero mach number system in the framework of endpoint besov spaces of type b r s r d r 1 d 2 which can be embedded in the lipschitz class c 0 1 in particular the largest case b 1 1 and the case of holder spaces c 1 are taken into account the local in time well posedness of this system is proved under an additional finite energy hypothesis on the initial data the key to get this result is new a priori estimates for parabolic equations with variable coefficients in endpoint spaces b r s r d which are of independent interest in the special case of space dimension d 2 we are able to give a lower bound for the lifespan such that the solutions tend to be globally defined when the initial inhomogeneity is small there we will show refined a priori estimates in endpoint besov spaces for transport equations