si admm a stochastic inexact admm framework for stochastic convex programs we consider the structured stochastic convex program requiring the minimization of mathbb e xi tilde f x xi mathbb e xi tilde g y xi subject to the constraint ax by b motivated by the need for decentralized schemes we propose a stochastic inexact alternating direction method of multiplier si admm framework where subproblems are solved inexactly via stochastic approximation schemes we propose a stochastic inexact alternating direction method of multiplier si admm framework where subproblems are solved inexactly via stochastic approximation schemes based on this framework we prove the following 1 under suitable assumptions on the associated batch size of samples utilized at each iteration the si admm scheme produces a sequence that converges to the unique solution almost surely 2 if the number of gradient steps or equivalently the number of sampled gradients utilized for solving the subproblems in each iteration increases at a geometric rate the mean squared error diminishes to zero at a prescribed geometric rate and 3 the overall iteration complexity in terms of gradient steps or equivalently samples is found to be consistent with the canonical level of mathcal o 1 epsilon preliminary applications on lasso and distributed regression suggest that the scheme performs well compared to its competitors

worst case linear discriminant analysis as scalable semidefinite feasibility problems in this paper we propose an efficient semidefinite programming sdp approach to worst case linear discriminant analysis wlda compared with the traditional lda wlda considers the dimensionality reduction problem from the worst case viewpoint which is in general more robust for classification however the original problem of wlda is non convex and difficult to optimize in this paper we reformulate the optimization problem of wlda into a sequence of semidefinite feasibility problems to efficiently solve the semidefinite feasibility problems we design a new scalable optimization method with a quasi newton method and eigen decomposition being the core components the proposed method is orders of magnitude faster than standard interior point sdp solvers experiments on a variety of classification problems demonstrate that our approach achieves better performance than standard lda our method is also much faster and more scalable than standard interior point sdp solvers based wlda the computational complexity for an sdp with m constraints and matrices of size d by d is roughly reduced from mathcal o m 3 md 3 m 2 d 2 to mathcal o d 3 m d in our case

validation analysis of mirror descent stochastic approximation method the main goal of this paper is to develop accuracy estimates for stochastic programming problems by employing stochastic approximation sa type algorithms to this end we show that while running a mirror descent stochastic approximation procedure one can compute with a small additional effort lower and upper statistical bounds for the optimal objective value we demonstrate that for a certain class of convex stochastic programs these bounds are comparable in quality with similar bounds computed by the sample average approximation method while their computational cost is considerably smaller

robust parameter design with computer experiments using orthonormal polynomials robust parameter design with computer experiments is becoming increasingly important for product design existing methodologies for this problem are mostly for finding optimal control factor settings however in some cases the objective of the experimenter may be to understand how the noise and control factors contribute to variation in the response the functional analysis of variance anova and variance decompositions of the response in addition to the mean and variance models help achieve this objective estimation of these quantities is not easy and few methods are able to quantity the estimation uncertainty in this article we show that the use of an orthonormal polynomial model of the simulator leads to simple formulas for functional anova and variance decompositions and the mean and variance models we show that estimation uncertainty can be taken into account in a simple way by first fitting a gaussian process model to experiment data and then approximating it with the orthonormal polynomia

high dimensional nonconvex stochastic optimization by doubly stochastic successive convex approximation in this paper we consider supervised learning problems over training sets in which the number of training examples and the dimension of feature vectors are both large we focus on the case where the loss function defining the quality of the parameter we wish to estimate may be non convex but also has a convex regularization we propose a doubly stochastic successive convex approximation scheme dssc able to handle non convex regularized expected risk minimization the method operates by decomposing the decision variable into blocks and operating on random subsets of blocks at each step fusing the merits of stochastic approximation with block coordinate methods and then implements successive convex approximation in contrast to many stochastic convex methods whose almost sure behavior is not guaranteed in non convex settings dssc attains almost sure convergence to a stationary solution of the problem moreover we show that the proposed dssc algorithm achieves stationarity at a rate of mathcal o log t t 1 4 numerical experiments on a non convex variant of a lasso regression problem show that dssc performs favorably in this setting we then apply this method to the task of dictionary learning from high dimensional visual data collected from a ground robot and observe reliable convergence behavior for a difficult non convex stochastic program

on a multistage discrete stochastic optimization problem with stochastic constraints and nested sampling we consider a multistage stochastic discrete program in which constraints on any stage might involve expectations that cannot be computed easily and are approximated by simulation we study a sample average approximation saa approach that uses nested sampling in which at each stage a number of scenarios are examined and a number of simulation replications are performed for each scenario to estimate the next stage constraints this approach provides an approximate solution to the multistage problem to establish the consistency of the saa approach we first consider a two stage problem and show that in the second stage problem given a scenario the optimal values and solutions of the saa converge to those of the true problem with probability one when the sample sizes go to infinity these convergence results do not hold uniformly over all possible scenarios for the second stage problem we are nevertheless able to prove that the optimal values and solutions of the saa converge to the true ones with probability one when the sample sizes at both stages increase to infinity we also prove exponential convergence of the probability of a large deviation for the optimal value of the saa the true value of an optimal solution of the saa and the probability that any optimal solution to the saa is an optimal solution of the true problem all of these results can be extended to a multistage setting and we explain how to do it our framework and saa results cover a large variety of resource allocation problems for which at each stage after the first one new information becomes available and the allocation can be readjusted under constraints that involve expectations estimated by monte carlo as an illustration we apply this saa method to a staffing problem in a call center in which the goal is to optimize the numbers of agents of each type under some constraints on the quality of service qos the staffing allocation has to be decided under an uncertain arrival rate with a prior distribution in the first stage and can be adjusted at some additional cost when better information on the arrival rate becomes available in later stages

the sample average approximation method for stochastic discrete optimization in this paper we study a monte carlo simulation based approach to stochastic discrete optimization problems the basic idea of such methods is that a random sample is generated and the expected value function is approximated by the corresponding sample average function the obtained sample average optimization problem is solved and the procedure is repeated several times until a stopping criterion is satisfied we discuss convergence rates stopping rules and computational complexity of this procedure and present a numerical example for the stochastic knapsack problem

variance based extragradient methods with line search for stochastic variational inequalities in this paper we propose dynamic sampled stochastic approximated ds sa extragradient methods for stochastic variational inequalities svis that are robust with respect to an unknown lipschitz c

bundle level type methods uniformly optimal for smooth and nonsmooth convex optimization the main goal of this paper is to develop uniformly optimal first order methods for convex programming cp by uniform optimality we mean that the first order methods themselves do not require the input of any problem parameters but can still achieve the best possible iteration complexity bounds by incorporating a multi step acceleration scheme into the well known bundle level method we develop an accelerated bundle level method and show that it can achieve the optimal complexity for solving a general class of black box cp problems without requiring the input of any smoothness information such as whether the problem is smooth nonsmooth or weakly smooth as well as the specific values of lipschitz constant and smoothness level we then develop a more practical restricted memory version of this method namely the accelerated prox level apl method we investigate the generalization of the apl method for solving certain composite cp problems and an important class of saddle point problems recently studied by nesterov math program 103 127 152 2005 we present promising numerical results for these new bundle level methods applied to solve certain classes of semidefinite programming and stochastic programming problems

learning in games with continuous action sets and unknown payoff functions this paper examines the convergence of no regret learning in games with continuous action sets for concreteness we focus on learning via dual averaging a widely used class of no regret learning schemes where players take small steps along their individual payoff gradients and then mirror the output back to their action sets in terms of feedback we assume that players can only estimate their payoff gradients up to a zero mean error with bounded variance to study the convergence of the induced sequence of play we introduce the notion of variational stability and we show that stable equilibria are locally attracting with high probability whereas globally stable equilibria are globally attracting with probability 1 we also discuss some applications to mixed strategy learning in finite games and we provide explicit estimates of the method s convergence speed

an optimal method for stochastic composite optimization this paper considers an important class of convex programming cp problems namely the stochastic composite optimization sco whose objective function is given by the summation of general nonsmooth and smooth stochastic components since sco covers non smooth smooth and stochastic cp as certain special cases a valid lower bound on the rate of convergence for solving these problems is known from the classic complexity theory of convex programming note however that the optimization algorithms that can achieve this lower bound had never been developed in this paper we show that the simple mirror descent stochastic approximation method exhibits the best known rate of convergence for solving these problems our major contribution is to introduce the accelerated stochastic approximation ac sa algorithm based on nesterov s optimal method for smooth cp nesterov in doklady an sssr 269 543 547 1983 nesterov in math program 103 127 152 2005 and show that the ac sa algorithm can achieve the aforementioned lower bound on the rate of convergence for sco to the best of our knowledge it is also the first universally optimal algorithm in the literature for solving non smooth smooth and stochastic cp problems we illustrate the significant advantages of the ac sa algorithm over existing methods in the context of solving a special but broad class of stochastic programming problems

robust stochastic approximation approach to stochastic programming in this paper we consider optimization problems where the objective function is given in a form of the expectation a basic difficulty of solving such stochastic optimization problems is that the involved multidimensional integrals expectations cannot be computed with high accuracy the aim of this paper is to compare two computational approaches based on monte carlo sampling techniques namely the stochastic approximation sa and the sample average approximation saa methods both approaches the sa and saa methods have a long history current opinion is that the saa method can efficiently use a specific say linear structure of the considered problem while the sa approach is a crude subgradient method which often performs poorly in practice we intend to demonstrate that a properly modified sa approach can be competitive and even significantly outperform the saa method for a certain class of convex stochastic problems we extend the analysis to the case of convex concave stochastic saddle point problems and present in our opinion highly encouraging results of numerical experiments

a central limit theorem and hypotheses testing for risk averse stochastic programs we study statistical properties of the optimal value and optimal solutions of the sample average approximation of risk averse stochastic problems central limit theorem type results are derived for the optimal value when the stochastic program is expressed in terms of a law invariant coherent risk measure having a discrete kusuoka representation the obtained results are applied to hypotheses testing problems aiming at comparing the optimal values of several risk averse convex stochastic programs on the basis of samples of the underlying random vectors we also consider nonasymptotic tests based on confidence intervals on the optimal values of the stochastic programs obtained using the stochastic mirror descent algorithm numerical simulations show how to use our developments to choose among different distributions and on the considered class of risk averse stochastic programs the asymptotic tests show better results

optimality functions in stochastic programming optimality functions define stationarity in nonlinear programming semi infinite optimization and optimal control in some sense in this paper we consider optimality functions for stochastic programs with nonlinear possibly nonconvex expected value objective and constraint functions we show that an optimality function directly relates to the difference in function values at a candidate point and a local minimizer we construct confidence intervals for the value of the optimality function at a candidate point and hence provide a quantitative measure of solution quality based on sample average approximations we develop an algorithm for classes of stochastic programs that include cvar problems and utilize optimality functions to select sample sizes

accelerated gradient methods for nonconvex nonlinear and stochastic programming in this paper we generalize the well known nesterov s accelerated gradient ag method originally designed for convex smooth optimization to solve nonconvex and possibly stochastic optimization problems we demonstrate that by properly specifying the stepsize policy the ag method exhibits the best known rate of convergence for solving general nonconvex smooth optimization problems by using first order information similarly to the gradient descent method we then consider an important class of composite optimization problems and show that the ag method can solve them uniformly i e by using the same aggressive stepsize policy as in the convex case even if the problem turns out to be nonconvex we demonstrate that the ag method exhibits an optimal rate of convergence if the composite problem is convex and improves the best known rate of convergence if the problem is nonconvex based on the ag method we also present new nonconvex stochastic approximation methods and show that they can improve a few existing rates of convergence for nonconvex stochastic optimization to the best of our knowledge this is the first time that the convergence of the ag method has been established for solving nonconvex nonlinear programming in the literature

on solving large scale polynomial convex problems by randomized first order algorithms one of the most attractive recent approaches to processing well structured large scale convex optimization problems is based on smooth convex concave saddle point reformulation of the problem of interest and solving the resulting problem by a fast first order saddle point method utilizing smoothness of the saddle point cost function in this paper we demonstrate that when the saddle point cost function is polynomial the precise gradients of the cost function required by deterministic first order saddle point algorithms and becoming prohibitively computationally expensive in the extremely large scale case can be replaced with incomparably cheaper computationally unbiased random estimates of the gradients we show that for large scale problems with favorable geometry this randomization accelerates progressively as the sizes of the problem grow the solution process this extends significantly previous results on acceleration by randomization which to the best of our knowledge dealt solely with bilin

on variance reduction for stochastic smooth convex optimization with multiplicative noise we propose dynamic sampled stochastic approximation sa methods for stochastic optimization with a heavy tailed distribution with finite 2nd moment the objective is the sum of a smooth convex function with a convex regularizer typically it is assumed an oracle with an upper bound sigma 2 on its variance oubv differently we assume an oracle with multiplicative noise this rarely addressed setup is more aggressive but realistic where the variance may not be uniformly bounded our methods achieve optimal iteration complexity and near optimal oracle complexity for the smooth convex class we use an accelerated sa method a la fista which achieves given tolerance varepsilon 0 the optimal iteration complexity of mathscr o varepsilon frac 1 2 with a near optimal oracle complexity of mathscr o varepsilon 2 ln varepsilon frac 1 2 2 this improves upon ghadimi and lan math program 156 59 99 2016 where it is assumed an oubv for the strongly convex class our method achieves optimal iteration complexity of mathscr o ln varepsilon 1 and optimal oracle complexity of mathscr o varepsilon 1 this improves upon byrd et al math program 134 127 155 2012 where it is assumed an oubv in terms of variance our bounds are local they depend on variances sigma x 2 at solutions x and the per unit distance multiplicative variance sigma 2 l for the smooth convex class there exist policies such that our bounds resemble up to absolute constants those obtained in the mentioned papers if it was assumed an oubv with sigma 2 sigma x 2 for the strongly convex class such property is obtained exactly if the condition number is estimated or in the limit for better conditioned problems or for larger initial batch sizes in any case if it is assumed an oubv our bounds are thus sharper since typically max sigma x 2 sigma l 2 ll sigma 2

asymptotic and finite sample properties of estimators based on stochastic gradients stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets however their statistical properties are not well understood in theory and in practice avoiding numerical instability requires careful tuning of key parameters here we introduce implicit stochastic gradient descent procedures which involve parameter updates that are implicitly defined intuitively implicit updates shrink standard stochastic gradient descent updates the amount of shrinkage depends on the observed fisher information matrix which does not need to be explicitly computed thus implicit procedures increase stability without increasing the computational burden our theoretical analysis provides the first full characterization of the asymptotic behavior of both standard and implicit stochastic gradient descent based estimators including finite sample error bounds importantly analytical expressions for the variances of these stochastic gradient based estimators reveal their exact loss of efficiency we also develop new algorithms to compute implicit stochastic gradient descent based estimators for generalized linear models cox proportional hazards m estimators in practice and perform extensive experiments our results suggest that implicit stochastic gradient descent procedures are poised to become a workhorse for approximate inference from large data sets

fixed width sequential stopping rules for a class of stochastic programs monte carlo sampling based methods are frequently used in stochastic programming when exact solution is not possible a critical component of monte carlo sampling based methods is determining when to stop sampling to ensure the desired quality of the solutions in this paper we develop stopping rules for sequential sampling procedures that depend on the width of an optimality gap confidence interval estimator the procedures solve a sequence of sampling approximations with increasing sample size to generate solutions and stop when the width of the confidence interval on the current solution s optimality gap plus an inflation factor fall below a prespecified value epsilon we first present a method that takes the schedule of sample sizes as an input and provide guidelines on the growth of sample sizes then we present a method that increases the sample sizes according to the current estimates of the optimality gap the larger the estimates the larger the increases in sample sizes we provide conditi

a smooth penalty based sample average approximation method for stochastic complementarity problems sample average approximation method is one of the effective methods in the stochastic optimization a smooth penalty based sample average approximation method for stochastic nonlinear complementarity problems is presented in this paper based on a smooth penalty function a reformulation is proposed for the equivalent problem of ev formulation for stochastic complementary problems and it is proven that its solutions are existent under some mild assumptions an implementable sample average approximation method for the reformulation is further established and its convergence is analyzed the numerical results for some test examples are reported at last to show efficiency of the proposed method

introduction to convex optimization in financial markets convexity arises naturally in financial risk management in risk preferences concerning random cash flows convexity corresponds to the fundamental diversification principle convexity is a basic property also of budget constraints both in classical linear models as well as in more realistic models with transaction costs and constraints moreover modern securities markets are based on trading protocols that result in convex trading costs the first part of this paper gives an introduction to certain basic concepts and principles of financial risk management in simple optimization terms the second part reviews some convex optimization techniques used in mathematical and numerical analysis of financial optimization problems

non euclidean restricted memory level method for large scale convex optimization we propose a new subgradient type method for minimizing extremely large scale nonsmooth convex functions over simple domains the characteristic features of the method are a the possibility to adjust the scheme to the geometry of the feasible set thus allowing to get nearly dimension independent and nearly optimal in the large scale case rate of convergence results for minimization of a convex lipschitz continuous function over a euclidean ball a standard simplex and a spectahedron the set of positive semidefinite symmetric matrices of given size with unit trace b flexible handling of accumulated information allowing for tradeoff between the level of utilizing this information and iteration s complexity we present extensions of the scheme for the cases of minimizing non lipschitzian convex objectives finding saddle points of convex concave functions and solving variational inequalities with monotone operators finally we report on encouraging numerical results of experiments with test problems of dimensions up to 66 000

multistep stochastic mirror descent for risk averse convex stochastic programs based on extended polyhedral risk measures we consider risk averse convex stochastic programs expressed in terms of extended polyhedral risk measures we derive computable confidence intervals on the optimal value of such stochastic programs using the robust stochastic approximation and the stochastic mirror descent smd algorithms when the objective functions are uniformly convex we also propose a multistep extension of the stochastic mirror descent algorithm and obtain confidence intervals on both the optimal values and optimal solutions numerical simulations show that our confidence intervals are much less conservative and are quicker to compute than previously obtained confidence intervals for smd and that the multistep stochastic mirror descent algorithm can obtain a good approximate solution much quicker than its nonmultistep counterpart

smoothing techniques for computing nash equilibria of sequential games we develop first order smoothing techniques for saddle point problems that arise in finding a nash equilibrium of sequential games the crux of our work is a construction of suitable prox functions for a certain class of polytopes that encode the sequential nature of the game we also introduce heuristics that significantly speed up the algorithm and decomposed game representations that reduce the memory requirements enabling the application of the techniques to drastically larger games an implementation based on our smoothing techniques computes approximate nash equilibria for games that are more than four orders of magnitude larger than what prior approaches can handle finally we show near linear further speedups from parallelization

asymptotic results of stochastic decomposition for two stage stochastic quadratic programming this paper presents the stochastic decomposition sd algorithms for two classes of stochastic programming problems 1 two stage stochastic quadratic linear programming sqlp in which a quadrati

optimization with reference based robust preference constraints we propose an optimization model using a novel robust preference relationship reference based almost stochastic dominance rsd the concept of rsd addresses the two problems in utility based dec

hybrid deterministic stochastic methods for data fitting many structured data fitting applications require the solution of an optimization problem involving a sum over a potentially large number of measurements incremental gradient algorithms offer inexpensive iterations by sampling a subset of the terms in the sum these methods can make great progress initially but often slow as they approach a solution in contrast full gradient methods achieve steady convergence at the expense of evaluating the full objective and gradient on each iteration we explore hybrid methods that exhibit the benefits of both approaches rate of convergence analysis shows that by controlling the sample size in an incremental gradient algorithm it is possible to maintain the steady convergence rates of full gradient methods we detail a practical quasi newton implementation based on this approach numerical experiments illustrate its potential benefits

a comparison of three methods for selecting values of input variables in the analysis of output from a computer code two types of sampling plans are examined as alternatives to simple random sampling in monte carlo studies these plans are shown to be improvements over simple random sampling with respect to variance for a class of estimators which includes the sample mean and the empirical distribution function

distributed stochastic gradient tracking methods in this paper we study the problem of distributed multi agent optimization over a network where each agent possesses a local cost function that is smooth and strongly convex the global objective is to find a common solution that minimizes the average of all cost functions assuming agents only have access to unbiased estimates of the gradients of their local cost functions we consider a distributed stochastic gradient tracking method dsgt and a gossip like stochastic gradient tracking method gsgt we show that in expectation the iterates generated by each agent are attracted to a neighborhood of the optimal solution where they accumulate exponentially fast under a constant stepsize choice under dsgt the limiting expected error bounds on the distance of the iterates from the optimal solution decrease with the network size n which is a comparable performance to a centralized stochastic gradient algorithm moreover we show that when the network is well connected gsgt incurs lower communication cost than dsgt while maintaining a similar computational cost numerical example further demonstrates the effectiveness of the proposed methods

a stochastic successive minimization method for nonsmooth nonconvex optimization with applications to transceiver design in wireless communication networks consider the problem of minimizing the expected value of a cost function parameterized by a random variable the classical sample average approximation method for solving this problem requires minimization of an ensemble average of the objective at each step which can be expensive in this paper we propose a stochastic successive upper bound minimization method ssum which minimizes an approximate ensemble average at each iteration to ensure convergence and to facilitate computation we require the approximate ensemble average to be a locally tight upper bound of the expected cost function and be easily optimized the main contributions of this work include the development and analysis of the ssum method as well as its applications in linear transceiver design for wireless communication networks and online dictionary learning moreover using the ssum framework we extend the classical stochastic sub gradient method to the case of minimizing a nonsmooth nonconvex objective function and establish its convergence

min max generalization for deterministic batch mode reinforcement learning relaxation schemes we study the min max optimization problem introduced in fonteneau et al towards min max reinforcement learning icaart 2010 springer heidelberg 2011 pp 61 77 for computing policies for batch mode reinforcement learning in a deterministic setting with fixed finite time horizon first we show that the min part of this problem is np hard we then provide two relaxation schemes the first relaxation scheme works by dropping some constraints in order to obtain a problem that is solvable in polynomial time the second relaxation scheme based on a lagrangian relaxation where all constraints are dualized can also be solved in polynomial time we also theoretically prove and empirically illustrate that both relaxation schemes provide better results than those given in fonteneau et al 2011 as cited above

on sampling rates in simulation based recursions we consider the context of simulation based recursions that is recursions that involve quantities needing to be estimated using a stochastic simulation examples include stochastic adaptations of fixed point and gradient descent recursions obtained by replacing function and derivative values appearing within the recursion by their monte carlo counterparts the primary motivating settings are simulation optimization and stochastic root finding problems where the low point and the zero of a function are sought respectively with only monte carlo estimates of the functions appearing within the problem we ask how much monte carlo sampling needs to be performed within simulation based recursions in order that the resulting iterates remain consistent and more importantly efficient where efficient implies convergence at the fastest possible rate answering these questions involves trading off two types of error inherent in the iterates the deterministic error due to recursion and the stochastic er

a positivity preserving semi implicit discontinuous galerkin scheme for solving extended magnetohydrodynamics equations a positivity preserving discontinuous galerkin dg scheme 42 is used to solve the extended magnetohydrodynamics xmhd model which is a two fluid model expressed with a center of mass formulation we prove that dg scheme with a positivity preserving limiter is stable for the system governed by the xmhd model or the resistive mhd model we use the relaxation system formulation 28 for describing the xmhd model and solve the equations using a split level implicit explicit time advance scheme stepping over the time step constraint imposed by the stiff source terms the magnetic field is represented in an exact locally divergence free form of dg 23 which greatly improves the accuracy and stability of mhd simulations as presently constructed the method is able to handle a wide range of density variations solve xmhd model on mhd time scales and provide greatly improved accuracy over a finite volume implementation of the same model

cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents abstract for random matrix models the parameter estimation based on the traditional likelihood functions is not straightforward in particular when we have only one sample matrix we introduce a new parameter optimization method for random matrix models which works even in such a case the method is based on the spectral distribution instead of the traditional likelihood in the method the cauchy noise has an essential role because the free deterministic equivalent which is a tool in free probability theory allows us to approximate the spectral distribution perturbed by cauchy noises by a smooth and accessible density function moreover we study an asymptotic property of determination gap which has a similar role as generalization gap besides we propose a new dimensionality recovery method for the signal plus noise model and experimentally demonstrate that it recovers the rank of the signal part even if the true rank is not small it is a simultaneous rank selection and parameter estimation procedure

online nonnegative matrix factorization with outliers we propose a unified and systematic framework for performing online nonnegative matrix factorization in the presence of outliers our framework is particularly suited to large scale data we propose two solvers based on projected gradient descent and the alternating direction method of multipliers we prove that the sequence of objective values converges almost surely by appealing to the quasi martingale convergence theorem we also show the sequence of learned dictionaries converges to the set of stationary points of the expected loss function almost surely in addition we extend our basic problem formulation to various settings with different constraints and regularizers we also adapt the solvers and analyses to each setting we perform extensive experiments on both synthetic and real datasets these experiments demonstrate the computational efficiency and efficacy of our algorithms on tasks such as parts based basis learning image denoising shadow removal and foreground background separation

sample average approximation method for compound stochastic optimization problems the paper studies stochastic optimization programming problems with compound functions containing expectations and extreme values of other random functions as arguments compound functions arise in various applications a typical example is a variance function of nonlinear outcomes other examples include stochastic minimax problems econometric models with latent variables and multilevel and multicriteria stochastic optimization problems as a solution technique a sample average approximation saa method also known as statistical or empirical sample mean method is used the method consists in approximation of all expectation functions by their empirical means and solving the resulting approximate deterministic optimization problems in stochastic optimization this method is widely used for optimization of standard expectation functions under constraints in this paper saa method is extended to general compound stochastic optimization problems the conditions for convergence in mean almost sure

accelerated schemes for a class of variational inequalities we propose a novel stochastic method namely the stochastic accelerated mirror prox samp method for solving a class of monotone stochastic variational inequalities svi the main idea of the proposed algorithm is to incorporate a multi step acceleration scheme into the stochastic mirror prox method the developed samp method computes weak solutions with the optimal iteration complexity for svis in particular if the operator in svi consists of the stochastic gradient of a smooth function the iteration complexity of the samp method can be accelerated in terms of their dependence on the lipschitz constant of the smooth function for svis with bounded feasible sets the bound of the iteration complexity of the samp method depends on the diameter of the feasible set for unbounded svis we adopt the modified gap function introduced by monteiro and svaiter for solving monotone inclusion and show that the iteration complexity of the samp method depends on the distance from the initial point to the set of strong solutions it is worth noting that our study also significantly improves a few existing complexity results for solving deterministic variational inequality problems we demonstrate the advantages of the samp method over some existing algorithms through our preliminary numerical experiments

an evolving construction scheme for fuzzy systems this paper proposes an evolving construction scheme for fuzzy systems ecsfs ecsfs begins with a simple fuzzy system and evolves its structure by adding more fuzzy terms and rules to achieve a better accuracy in a greedy way an interesting feature of ecsfs is that it is able to automatically locate mathematically meaningful points such as the extremum and inflexion points of the approximated function one by one and then adds fuzzy terms based on these points fuzzy systems with such extreme points like their fuzzy terms are more efficient than other fuzzy systems by using the same number of fuzzy rules as a result ecsfs often achieves a better accuracy for fuzzy system identification compared with the previous methods when using the same number of fuzzy rules a number of simulation results are given to illustrate the advantages of the proposed scheme

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

optimal primal dual methods for a class of saddle point problems we present a novel accelerated primal dual apd method for solving a class of deterministic and stochastic saddle point problems spps the basic idea of this algorithm is to incorporate a multistep acceleration scheme into the primal dual method without smoothing the objective function for deterministic spp the apd method achieves the same optimal rate of convergence as nesterov s smoothing technique our stochastic apd method exhibits an optimal rate of convergence for stochastic spp not only in terms of its dependence on the number of the iteration but also on a variety of problem parameters to the best of our knowledge this is the first time that such an optimal algorithm has been developed for stochastic spp in the literature furthermore for both deterministic and stochastic spp the developed apd algorithms can deal with the situation when the feasible region is unbounded as long as a saddle point exists in the unbounded case we incorporate the modified termination criterion introduced b

primal dual subgradient methods for convex problems in this paper we present a new approach for constructing subgradient schemes for different types of nonsmooth problems with convex structure our methods are primal dual since they are always able to generate a feasible approximation to the optimum of an appropriately formulated dual problem besides other advantages this useful feature provides the methods with a reliable stopping criterion the proposed schemes differ from the classical approaches divergent series methods mirror descent methods by presence of two control sequences the first sequence is responsible for aggregating the support functions in the dual space and the second one establishes a dynamically updated scale between the primal and dual spaces this additional flexibility allows to guarantee a boundedness of the sequence of primal test points even in the case of unbounded feasible set however we always assume the uniform boundedness of subgradients we present the variants of subgradient schemes for nonsmooth convex minimization minimax problems saddle point problems variational inequalities and stochastic optimization in all situations our methods are proved to be optimal from the view point of worst case black box lower complexity bounds

stochastic optimization using a trust region method and random models in this paper we propose and analyze a trust region model based algorithm for solving unconstrained stochastic optimization problems our framework utilizes random models of an objective function f x obtained from stochastic observations of the function or its gradient our method also utilizes estimates of function values to gauge progress that is being made the convergence analysis relies on requirements that these models and these estimates are sufficiently accurate with high enough but fixed probability beyond these conditions no assumptions are made on how these models and estimates are generated under these general conditions we show an almost sure global convergence of the method to a first order stationary point in the second part of the paper we present examples of generating sufficiently accurate random models under biased or unbiased noise assumptions lastly we present some computational results showing the benefits of the proposed method compared to existing approaches that are based on sample averaging or stochastic gradients

optimal stochastic approximation algorithms for strongly convex stochastic composite optimization ii shrinking procedures and optimal algorithms in this paper we study new stochastic approximation sa type algorithms namely the accelerated sa ac sa for solving strongly convex stochastic composite optimization sco problems specifically by introducing a domain shrinking procedure we significantly improve the large deviation results associated with the convergence rate of a nearly optimal ac sa algorithm presented by ghadimi and lan in siam j optim 22 2012 pp 1469 1492 moreover we introduce a multistage ac sa algorithm which possesses an optimal rate of convergence for solving strongly convex sco problems in terms of the dependence on not only the target accuracy but also a number of problem parameters and the selection of initial points to the best of our knowledge this is the first time that such an optimal method has been presented in the literature from our computational results these ac sa algorithms can substantially outperform the classical sa and some other sa type algorithms for solving certain classes of strongly

a parallel decomposition method for nonconvex stochastic multi agent optimization problems this paper considers the problem of minimizing the expected value of a possibly nonconvex cost function parameterized by a random vector variable when the expectation cannot be computed accurately e g because the statistics of the random variables are unknown and or the computational complexity is prohibitive classical stochastic gradient methods for solving this problem may suffer from slow convergence in this paper we propose a stochastic parallel successive convex approximation based best response algorithm for general nonconvex stochastic sum utility optimization problems which arise naturally in the design of multi agent networks the proposed novel decomposition approach enables all users to update their optimization variables in parallel by solving a sequence of strongly convex subproblems one for each user almost sure convergence to stationary points is proved we then customize the algorithmic framework to solve the stochastic sum rate maximization problem over single input single output siso frequency selective interference channels multiple input multiple output mimo interference channels and mimo multiple access channels numerical results corroborate that the proposed algorithms can converge faster than state of the art stochastic gradient schemes while achieving the same or better sum rates

bayesian inference with optimal maps we present a new approach to bayesian inference that entirely avoids markov chain simulation by constructing a map that pushes forward the prior measure to the posterior measure existence and uniqueness of a suitable measure preserving map is established by formulating the problem in the context of optimal transport theory we discuss various means of explicitly parameterizing the map and computing it efficiently through solution of an optimization problem exploiting gradient information from the forward model when possible the resulting algorithm overcomes many of the computational bottlenecks associated with markov chain monte carlo advantages of a map based representation of the posterior include analytical expressions for posterior moments and the ability to generate arbitrary numbers of independent posterior samples without additional likelihood evaluations or forward solves the optimization approach also provides clear convergence criteria for posterior approximation and facilitates model selection through automatic evaluation of the marginal likelihood we demonstrate the accuracy and efficiency of the approach on nonlinear inverse problems of varying dimension involving the inference of parameters appearing in ordinary and partial differential equations

on perturbed proximal gradient algorithms we study a version of the proximal gradient algorithm for which the gradient is intractable and is approximated by monte carlo methods and in particular markov chain monte carlo we derive conditions on the step size and the monte carlo batch size under which convergence is guaranteed both increasing batch size and constant batch size are considered we also derive non asymptotic bounds for an averaged version our results cover both the cases of biased and unbiased monte carlo approximation to support our findings we discuss the inference of a sparse generalized linear model with random effect and the problem of learning the edge structure and parameters of sparse undirected graphical models

optimal rates for zero order convex optimization the power of two function evaluations we consider derivative free algorithms for stochastic and nonstochastic convex optimization problems that use only function values rather than gradients focusing on nonasymptotic bounds on convergence rates we show that if pairs of function values are available algorithms for d dimensional optimization that use gradient estimates based on random perturbations suffer a factor of at most sqrt d in convergence rate over traditional stochastic gradient methods we establish such results for both smooth and nonsmooth cases sharpening previous analyses that suggested a worse dimension dependence and extend our results to the case of multiple m ge 2 evaluations we complement our algorithmic development with information theoretic lower bounds on the minimax convergence rate of such problems establishing the sharpness of our achievable results up to constant sometimes logarithmic factors

mini batch stochastic approximation methods for nonconvex stochastic composite optimization this paper considers a class of constrained stochastic composite optimization problems whose objective function is given by the summation of a differentiable possibly nonconvex component together with a certain non differentiable but convex component in order to solve these problems we propose a randomized stochastic projected gradient rspg algorithm in which proper mini batch of samples are taken at each iteration depending on the total budget of stochastic samples allowed the rspg algorithm also employs a general distance function to allow taking advantage of the geometry of the feasible region complexity of this algorithm is established in a unified setting which shows nearly optimal complexity of the algorithm for convex stochastic programming a post optimization phase is also proposed to significantly reduce the variance of the solutions returned by the algorithm in addition based on the rspg algorithm a stochastic gradient free algorithm which only uses the stochastic zeroth order information has been also discussed some preliminary numerical results are also provided

on stochastic gradient and subgradient methods with adaptive steplength sequences traditionally stochastic approximation sa schemes have been popular choices for solving stochastic optimization problems however the performance of standard sa implementations can vary significantly based on the choice of the steplength sequence and in general little guidance is provided about good choices motivated by this gap we present two adaptive steplength schemes for strongly convex differentiable stochastic optimization problems equipped with convergence theory that aim to overcome some of the reliance on user specific parameters the first scheme referred to as a recursive steplength stochastic approximation rsa scheme optimizes the error bounds to derive a rule that expresses the steplength at a given iteration as a simple function of the steplength at the previous iteration and certain problem parameters the second scheme termed as a cascading steplength stochastic approximation csa scheme maintains the steplength sequence as a piecewise constant decreasing function with the reduction in the steplength occurring when a suitable error threshold is met then we allow for nondifferentiable objectives but with bounded subgradients over a certain domain in such a regime we propose a local smoothing technique based on random local perturbations of the objective function that leads to a differentiable approximation of the function assuming a uniform distribution on the local randomness we establish a lipschitzian property for the gradient of the approximation and prove that the obtained lipschitz bound grows at a modest rate with problem size this facilitates the development of an adaptive steplength stochastic approximation framework which now requires sampling in the product space of the original measure and the artificially introduced distribution

privacy aware learning we study statistical risk minimization problems under a privacy model in which the data is kept confidential even from the learner in this local privacy framework we establish sharp upper and lower bounds on the convergence rates of statistical estimation procedures as a consequence we exhibit a precise tradeoff between the amount of privacy the data preserves and the utility as measured by convergence rate of any statistical estimator or learning procedure

statistical guarantees for the em algorithm from population to sample based analysis we develop a general framework for proving rigorous guarantees on the performance of the em algorithm and a variant known as gradient em our analysis is divided into two parts a treatment of these algorithms at the population level in the limit of infinite data followed by results that apply to updates based on a finite set of samples first we characterize the domain of attraction of any global maximizer of the population likelihood this characterization is based on a novel view of the em updates as a perturbed form of likelihood ascent or in parallel of the gradient em updates as a perturbed form of standard gradient ascent leveraging this characterization we then provide non asymptotic guarantees on the em and gradient em algorithms when applied to a finite set of samples we develop consequences of our general theory for three canonical examples of incompletedata problems mixture of gaussians mixture of regressions and linear regression with covariates missing completely at random in each case our theory guarantees that with a suitable initialization a relatively small number of em or gradient em steps will yield with high probability an estimate that is within statistical error of the mle we provide simulations to confirm this theoretically predicted behavior

on projected stochastic gradient descent algorithm with weighted averaging for least squares regression the problem of least squares regression of a d dimensional unknown parameter is considered a stochastic gradient descent based algorithm with weighted iterate averaging that uses a single pass over the data is studied and its convergence rate is analyzed we first consider a bounded constraint set of the unknown parameter under some standard regularity assumptions we provide an explicit o 1 k upper bound on the convergence rate depending on the variance due to the additive noise in the measurements and the size of the constraint set we show that the variance term dominates the error and decreases with rate 1 k while the term that is related to the size of the constraint set decreases with rate log k k 2 we then compare the asymptotic ratio rho between the convergence rate of the proposed scheme and the empirical risk minimizer erm as the number of iterations approaches infinity we show that rho leq 4 for all d geq 1 when the random entries of the sensing vector are uncorrelated and identically distributed we further improve the upper bound by showing that rho leq 4 3 for the case of d 1 and unbounded parameter set when the random sensing entries are equal across time simulation results demonstrate strong performance of the algorithm as compared to existing methods and coincide with rho leq 4 3 even for large d in practice

optimal budget allocation for sample average approximation the sample average approximation approach to solving stochastic programs induces a sampling error caused by replacing an expectation by a sample average as well as an optimization error due to approximating the solution of the resulting sample average problem we obtain estimators of an optimal solution and the optimal value of the original stochastic program after executing a finite number of iterations of an optimization algorithm applied to the sample average problem we examine the convergence rate of the estimators as the computing budget tends to infinity and we characterize the allocation policies that maximize the convergence rate in the case of sublinear linear and superlinear convergence regimes for the optimization algorithm

monotonic calibrated interpolated look up tables real world machine learning applications may have requirements beyond accuracy such as fast evaluation times and interpretability in particular guaranteed monotonicity of the learned function with respect to some of the inputs can be critical for user confidence we propose meeting these goals for low dimensional machine learning problems by learning flexible monotonic functions using calibrated interpolated look up tables we extend the structural risk minimization framework of lattice regression to monotonic functions by adding linear inequality constraints in addition we propose jointly learning interpretable calibrations of each feature to normalize continuous features and handle categorical or missing data at the cost of making the objective non convex we address large scale learning through parallelization mini batching and random sampling of additive regularizer terms case studies on real world problems with up to sixteen features and up to hundreds of millions of training samples demonstrate the proposed monotonic functions can achieve state of the art accuracy in practice while providing greater transparency to users

online pairwise learning algorithms pairwise learning usually refers to a learning task that involves a loss function depending on pairs of examples among which the most notable ones are bipartite ranking metric learning and auc maximization in this letter we study an online algorithm for pairwise learning with a least square loss function in an unconstrained setting of a reproducing kernel hilbert space rkhs that we refer to as the online pairwise learning algorithm opera in contrast to existing works kar sriperumbudur jain karnick 2013 wang khardon pechyony jones 2012 which require that the iterates are restricted to a bounded domain or the loss function is strongly convex opera is associated with a non strongly convex objective function and learns the target function in an unconstrained rkhs specifically we establish a general theorem that guarantees the almost sure convergence for the last iterate of opera without any assumptions on the underlying distribution explicit convergence rates are derived under the condition of polynomially decaying step sizes we also establish an interesting property for a family of widely used kernels in the setting of pairwise learning and illustrate the convergence results using such kernels our methodology mainly depends on the characterization of rkhss using its associated integral operators and probability inequalities for random variables with values in a hilbert space

robust inversion dimensionality reduction and randomized sampling we consider a class of inverse problems in which the forward model is the solution operator to linear odes or pdes this class admits several dimensionality reduction techniques based on data averaging or sampling which are especially useful for large scale problems we survey these approaches and their connection to stochastic optimization the data averaging approach is only viable however for a least squares misfit which is sensitive to outliers in the data and artifacts unexplained by the forward model this motivates us to propose a robust formulation based on the student s t distribution of the error we demonstrate how the corresponding penalty function together with the sampling approach can obtain good results for a large scale seismic inverse problem with 50 corrupted data

self concordant inclusions a unified framework for path following generalized newton type algorithms we study a class of monotone inclusions called self concordant inclusion which covers three fundamental convex optimization formulations as special cases we develop a new generalized newton type framework to solve this inclusion our framework subsumes three schemes full step damped step and path following methods as specific instances while allows one to use inexact computation to form generalized newton directions we prove the local quadratic convergence of both full step and damped step algorithms then we propose a new two phase inexact path following scheme for solving this monotone inclusion which possesses an mathcal o sqrt nu log 1 varepsilon worst case iteration complexity to achieve an varepsilon solution where nu is the barrier parameter and varepsilon is a desired accuracy as byproducts we customize our scheme to solve three convex problems the convex concave saddle point problem the nonsmooth constrained convex program and the nonsmooth convex program with linear constraints we also provide three numerical examples to illustrate our theory and compare with existing methods

the subdifferential of measurable composite max integrands and smoothing approximation the subdifferential calculus for the expectation of nonsmooth random integrands involves many fundamental and challenging problems in stochastic optimization it is known that for clarke regular integrands the clarke subdifferential of the expectation equals the expectation of their clarke subdifferential in particular this holds for convex integrands however little is known about the calculation of clarke subgradients for the expectation of non regular integrands the focus of this contribution is to approximate clarke subgradients for the expectation of random integrands by smoothing methods applied to the integrand a framework for how to proceed along this path is developed and then applied to a class of measurable composite max integrands this class contains non regular integrands from stochastic complementarity problems as well as stochastic optimization problems arising in statistical learning

acceleration of stochastic approximation by averaging a new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated convergence with probability one is proved for a variety of classical optimization and identification problems it is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence

stochastic root finding and efficient estimation of convex risk measures reliable risk measurement is a key problem for financial institutions and regulatory authorities the current industry standard value at risk has several deficiencies improved risk measures have been suggested and analyzed in the recent literature but their computational implementation has largely been neglected so far we propose and investigate stochastic approximation algorithms for the convex risk measure utility based shortfall risk our approach combines stochastic root finding schemes with importance sampling we prove that the resulting shortfall risk estimators are consistent and asymptotically normal and provide formulas for confidence intervals the performance of the proposed algorithms is tested numerically we finally apply our techniques to the normal copula model which is also known as the industry model creditmetrics this provides guidance for future implementations in practice

optimal affine invariant smooth minimization algorithms we formulate an affine invariant implementation of the accelerated first order algorithm in y nesterov dokl math 27 1983 pp 372 376 its complexity bound is proportional to an affine invariant regularity constant defined with respect to the minkowski gauge of the feasible set we extend these results to more general problems optimizing holder smooth functions using p uniformly convex prox terms and derive an algorithm whose complexity better fits the geometry of the feasible set and adapts to both the best holder smoothness parameter and the best gradient lipschitz constant finally we detail matching complexity lower bounds when the feasible set is an ell p ball in this setting our upper bounds on iteration complexity for the algorithm in y nesterov dokl math 27 1983 pp 372 376 are thus optimal in terms of target precision smoothness and problem dimension

a new approach to the maximum flow problem all previously known efficient maximum flow algorithms work by finding augmenting paths either one path at a time as in the original ford and fulkerson algorithm or all shortest length augmenting paths at once using the layered network approach of dinic an alternative method based on the preflow concept of karzanov is introduced a preflow is like a flow except that the total amount flowing into a vertex is allowed to exceed the total amount flowing out the method maintains a preflow in the original network and pushes local flow excess toward the sink along what are estimated to be shortest paths the algorithm and its analysis are simple and intuitive yet the algorithm runs as fast as any other known method on dense graphs achieving an o n 3 time bound on an n vertex graph by incorporating the dynamic tree data structure of sleator and tarjan we obtain a version of the algorithm running in o nm log n 2 m time on an n vertex m edge graph this is as fast as any known method for any graph density and faster on graphs of moderate density the algorithm also admits efficient distributed and parallel implementations a parallel implementation running in o n 2 log n time using n processors and o m space is obtained this time bound matches that of the shiloach vishkin algorithm which also uses n processors but requires o n 2 space

two stage linear decision rules for multi stage stochastic programming multi stage stochastic linear programs mslps are notoriously hard to solve in general linear decision rules ldrs yield an approximation of an mslp by restricting the decisions at each stage to be an affine function of the observed uncertain parameters finding an optimal ldr is a static optimization problem that provides an upper bound on the optimal value of the mslp and under certain assumptions can be formulated as an explicit linear program similarly as proposed by kuhn et al math program 130 1 177 209 2011 a lower bound for an mslp can be obtained by restricting decisions in the dual of the mslp to follow an ldr we propose a new approximation approach for mslps two stage ldrs the idea is to require only the state variables in an mslp to follow an ldr which is sufficient to obtain an approximation of an mslp that is a two stage stochastic linear program 2slp we similarly propose to apply ldr only to a subset of the variables in the dual of the mslp which yields a 2slp approximation of the dual that provides a lower bound on the optimal value of the mslp although solving the corresponding 2slp approximations exactly is intractable in general we investigate how approximate solution approaches that have been developed for solving 2slp can be applied to solve these approximation problems and derive statistical upper and lower bounds on the optimal value of the mslp in addition to potentially yielding better policies and bounds this approach requires many fewer assumptions than are required to obtain an explicit reformulation when using the standard static ldr approach a computational study on two example problems demonstrates that using a two stage ldr can yield significantly better primal policies and modestly better dual policies than using policies based on a static ldr

manifold identification in dual averaging for regularized stochastic online learning iterative methods that calculate their steps from approximate subgradient directions have proved to be useful for stochastic learning problems over large and streaming data sets when the objective consists of a loss function plus a nonsmooth regularization term the solution often lies on a low dimensional manifold of parameter space along which the regularizer is smooth when an l1 regularizer is used to induce sparsity in the solution for example this manifold is defined by the set of nonzero components of the parameter vector this paper shows that a regularized dual averaging algorithm can identify this manifold with high probability before reaching the solution this observation motivates an algorithmic strategy in which once an iterate is suspected of lying on an optimal or near optimal manifold we switch to a local phase that searches in this manifold thus converging rapidly to a near optimal point computational results are presented to verify the identification property and to illustrate the effectiveness of this approach

proximity without consensus in online multiagent optimization we consider stochastic optimization problems in multiagent settings where a network of agents aims to learn parameters that are optimal in terms of a global convex objective while giving preference to locally observed streaming information to do so we depart from the canonical decentralized optimization framework where agreement constraints are enforced and instead formulate a problem where each agent minimizes a global convex objective while enforcing network proximity constraints this formulation includes online consensus optimization as a special case but allows for the more general hypothesis that there is data heterogeneity across the network to solve this problem we propose using a stochastic saddle point algorithm inspired by arrow and hurwicz this method yields a decentralized algorithm for processing observations sequentially received at each node of the network using lagrange multipliers to penalize the discrepancy between them only neighboring nodes exchange model information we establish that under a constant step size regime the time average suboptimality and constraint violation are contained in a neighborhood whose radius vanishes with increasing number of iterations as a consequence we prove that the time average primal vectors converge to the optimal objective while satisfying the network proximity constraints we apply this method to the problem of sequentially estimating a correlated random field in a sensor network as well as an online source localization problem both of which demonstrate the empirical validity of the aforementioned convergence results

bundle methods for sum functions with easy components applications to multicommodity network design we propose a version of the bundle scheme for convex nondifferentiable optimization suitable for the case of a sum function where some of the components are easy that is they are lagrangian functions of explicitly known compact convex programs this corresponds to a stabilized partial dantzig wolfe decomposition where suitably modified representations of the easy convex subproblems are inserted in the master problem as an alternative to iteratively inner approximating them by extreme points thus providing the algorithm with exact information about a part of the dual objective function the resulting master problems are potentially larger and less well structured than the standard ones ruling out the available specialized techniques and requiring the use of general purpose solvers for their solution this strongly favors piecewise linear stabilizing terms as opposed to the more usual quadratic ones which in turn may have an adverse effect on the convergence speed of the algorithm so that the overall performance may depend on appropriate tuning of all these aspects yet very good computational results are obtained in at least one relevant application the computation of tight lower bounds for fixed charge multicommodity min cost flow problems

optimization of convex functions with random pursuit we consider unconstrained randomized optimization of convex objective functions we analyze the random pursuit algorithm which iteratively computes an approximate solution to the optimization problem by repeated optimization over a randomly chosen one dimensional subspace this randomized method only uses zeroth order information about the objective function and does not need any problem specific parametrization we prove convergence and give convergence rates for smooth objectives assuming that the one dimensional optimization can be solved exactly or approximately by an oracle a convenient property of random pursuit is its invariance under strictly monotone transformations of the objective function it thus enjoys identical convergence behavior on a wider function class to support the theoretical results we present extensive numerical performance results of random pursuit two gradient free algorithms recently proposed by nesterov and a classical adaptive step size random search scheme we also pres

incremental majorization minimization optimization with application to large scale machine learning majorization minimization algorithms consist of successively minimizing a sequence of upper bounds of the objective function these upper bounds are tight at the current estimate and each iteration monotonically drives the objective function downhill such a simple principle is widely applicable and has been very popular in various scientific fields especially in signal processing and statistics we propose an incremental majorization minimization scheme for minimizing a large sum of continuous functions a problem of utmost importance in machine learning we present convergence guarantees for nonconvex and convex optimization when the upper bounds approximate the objective up to a smooth error we call such upper bounds first order surrogate functions more precisely we study asymptotic stationary point guarantees for nonconvex problems and for convex ones we provide convergence rates for the expected objective function value we apply our scheme to composite optimization and obtain a new incremental proximal gradient algorithm with linear convergence rate for strongly convex functions our experiments show that our method is competitive with the state of the art for solving machine learning problems such as logistic regression when the number of training samples is large enough and we demonstrate its usefulness for sparse estimation with nonconvex penalties

learning in an uncertain world mimo covariance matrix optimization with imperfect feedback in this paper we present a distributed learning algorithm for the optimization of signal covariance matrices in gaussian multiple input and multiple output mimo multiple access channel with imperfect and possibly delayed feedback the algorithm is based on the method of matrix exponential learning mxl and it has the same information and computation requirements as distributed water filling however unlike water filling the proposed algorithm converges to the system s optimum signal covariance profile even under stochastic uncertainty and imperfect feedback moreover the algorithm also retains its convergence properties in the presence of user update asynchronicities random delays and or ergodically changing channel conditions our theoretical analysis is complemented by extensive numerical simulations which illustrate the robustness and scalability of mxl in realistic network conditions in particular the algorithm retains its convergence speed even for large numbers of users and or antennas per user