mismatch in high rate entropy constrained vector quantization bucklew s 1984 high rate vector quantizer mismatch result is extended from fixed rate coding to variable rate coding using a lagrangian formulation it is shown that if an asymptotically high rate optimal sequence of variable rate codes is designed for a k dimensional probability density function pdf g and then applied to another pdf f for which f g is bounded then the resulting mismatch or loss of performance from the optimal possible is given by the relative entropy or kullback leibler 1968 divergence i f spl par g it is also shown that under the same assumptions an asymptotically optimal code sequence for g can be converted to an asymptotically optimal code sequence for a mismatched source f by modifying only the lossless component of the code applications to quantizer design using uniform and gaussian densities are described including a high rate analog to the shannon rate distortion result of sakrison 1975 and lapidoth 1997 showing that the gaussian is the worst case for lossy compression of a source with known covariance by coupling the mismatch result with composite quantizers the worst case properties of uniform and gaussian densities are extended to conditionally uniform and gaussian densities which provides a lloyd clustering algorithm for fitting mixtures to general densities

dc programming and dca thirty years of developments the year 2015 marks the 30th birthday of dc difference of convex functions programming and dca dc algorithms which constitute the backbone of nonconvex programming and global optimization in this article we offer a short survey on thirty years of developments of these theoretical and algorithmic tools the survey is comprised of three parts in the first part we present a brief history of the field while in the second we summarize the state of the art results and recent advances we focus on main theoretical results and dca solvers for important classes of difficult nonconvex optimization problems and then give an overview of real world applications whose solution methods are based on dca the third part is devoted to new trends and important open issues as well as suggestions for future developments

learning overcomplete representations in an overcomplete basis the number of basis vectors is greater than the dimensionality of the input and the representation of an input is not a unique combination of basis vectors overcomplete representations have been advocated because they have greater robustness in the presence of noise can be sparser and can have greater flexibility in matching structure in the data overcomplete codes have also been proposed as a model of some of the response properties of neurons in primary visual cortex previous work has focused on finding the best representation of a signal using a fixed overcomplete basis or dictionary we present an algorithm for learning an overcomplete basis by viewing it as probabilistic model of the observed data we show that overcomplete bases can yield a better approximation of the underlying statistical distribution of the data and can thus lead to greater coding efficiency this can be viewed as a generalization of the technique of independent component analysis and provides a method for bayesian reconstruction of signals in the presence of noise and for blind source separation when there are more sources than mixtures

on the support of mse optimal fixed rate scalar quantizers this paper determines how the support regions of optimal and asymptotically optimal fixed rate scalar quantizers with respect to mean squared error depend on the number of quantization points n and the probability density of the variable being quantized it shows that for asymptotic optimality it is necessary and sufficient that the support region grow fast enough that the outer or overload distortion decreases as o 1 n sup 2 formulas are derived for the minimal support of asymptotically optimal quantizers for generalized gamma densities including gaussian and laplacian interestingly these turn out to be essentially the same as for the support of optimal fixed rate uniform scalar quantizers heuristic arguments are then used to find closed form estimates for the support of truly optimal quantizers for generalized gamma densities these are found to be more accurate than the best prior estimates as computed by numerical algorithms they demonstrate that the support of an optimal quantizer is larger than the minimal asymptotically optimal support by a factor depending on the density but not n and that the outer distortion of optimal quantizers decreases as 1 n sup 3

variance mismatched fixed rate scalar quantization of laplacian sources the paper investigates the mean squared error mse distortion of a variance mismatched fixed rate scalar quantizer that is optimal or asymptotically optimal in the minimum mse sense for a laplacian density with standard deviation q but is applied to another with standard deviation p a nitadori like formula is discovered for an optimal quantizer when p q 1 2 also it is shown that the distortion expressions derived rigorously for asymptotically optimal quantile quantizers essentially confirm the heuristically obtained previous result that the distortion decreases as 1 n3 for the heavy mismatch of 3 2 as lnn n2 for the critical mismatch of 3 2 and as 1 n2 for the mild mismatch of 1 and the resulting distortion decreases as in n n2 1 p

on the pervasiveness of difference convexity in optimization and statistics with the increasing interest in applying the methodology of difference of convex dc optimization to diverse problems in engineering and statistics this paper establishes the dc property of many functions in various areas of applications not previously known to be of this class motivated by a quadratic programming based recourse function in two stage stochastic programming we show that the optimal value function of a copositive thus not necessarily convex quadratic program is dc on the domain of finiteness of the program when the matrix in the objective function s quadratic term and the constraint matrix are fixed the proof of this result is based on a dc decomposition of a piecewise hbox lc 1 function i e functions with lipschitz gradients armed with these new results and known properties of dc functions existed in the literature we show that many composite statistical functions in risk analysis including the value at risk var conditional value at risk cvar optimized certainty equivalent and the expectation based var based and cvar based random deviation functionals are all dc adding the known class of dc surrogate sparsity functions that are employed as approximations of the ell 0 function in statistical learning our work significantly expands the classes of dc functions and positions them for fruitful applications

a unified convergence analysis of block successive minimization methods for nonsmooth optimization the block coordinate descent bcd method is widely used for minimizing a continuous function f of several block variables at each iteration of this method a single block of variables is optimized while the remaining variables are held fixed to ensure the convergence of the bcd method the subproblem of each block variable needs to be solved to its unique global optimal unfortunately this requirement is often too restrictive for many practical scenarios in this paper we study an alternative inexact bcd approach which updates the variable blocks by successively minimizing a sequence of approximations of f which are either locally tight upper bounds of f or strictly convex local approximations of f the main contributions of this work include the characterizations of the convergence conditions for a fairly wide class of such methods especially for the cases where the objective functions are either nondifferentiable or nonconvex our results unify and extend the existing convergence results

robust stochastic approximation approach to stochastic programming in this paper we consider optimization problems where the objective function is given in a form of the expectation a basic difficulty of solving such stochastic optimization problems is that the involved multidimensional integrals expectations cannot be computed with high accuracy the aim of this paper is to compare two computational approaches based on monte carlo sampling techniques namely the stochastic approximation sa and the sample average approximation saa methods both approaches the sa and saa methods have a long history current opinion is that the saa method can efficiently use a specific say linear structure of the considered problem while the sa approach is a crude subgradient method which often performs poorly in practice we intend to demonstrate that a properly modified sa approach can be competitive and even significantly outperform the saa method for a certain class of convex stochastic problems we extend the analysis to the case of convex concave stochastic saddle point problems and present in our opinion highly encouraging results of numerical experiments

an optimal method for stochastic composite optimization this paper considers an important class of convex programming cp problems namely the stochastic composite optimization sco whose objective function is given by the summation of general nonsmooth and smooth stochastic components since sco covers non smooth smooth and stochastic cp as certain special cases a valid lower bound on the rate of convergence for solving these problems is known from the classic complexity theory of convex programming note however that the optimization algorithms that can achieve this lower bound had never been developed in this paper we show that the simple mirror descent stochastic approximation method exhibits the best known rate of convergence for solving these problems our major contribution is to introduce the accelerated stochastic approximation ac sa algorithm based on nesterov s optimal method for smooth cp nesterov in doklady an sssr 269 543 547 1983 nesterov in math program 103 127 152 2005 and show that the ac sa algorithm can achieve the aforementioned lower bound on the rate of convergence for sco to the best of our knowledge it is also the first universally optimal algorithm in the literature for solving non smooth smooth and stochastic cp problems we illustrate the significant advantages of the ac sa algorithm over existing methods in the context of solving a special but broad class of stochastic programming problems

stochastic primal dual hybrid gradient algorithm with arbitrary sampling and imaging applications we propose a stochastic extension of the primal dual hybrid gradient algorithm studied by chambolle and pock in 2011 to solve saddle point problems that are separable in the dual variable the anal

a stackelberg game approach to distributed spectrum management we consider a cognitive radio system with one primary licensed user and multiple secondary unlicensed users given the interference temperature constraint the secondary users compete for the available spectrum to fulfill their own communication need borrowing the concept of price from market theory we develop a decentralized stackelberg game formulation for power allocation in this scheme the primary user leader announces prices for the available tones such that a system utility is maximized using the announced prices secondary users followers compete for the available bandwidth to maximize their own utilities we show that this stackelberg game is polynomial time solvable under certain channel conditions when the individual power constraints of secondary users are inactive due to strict interference temperature constraint the proposed distributed power control method is decomposable across the tones and unlike normal water filling it respects the interference temperature constraints of the primary user when individual power constraints are active we propose a distributed approach that solves the problem under an aggregate interference temperature constraint moreover we propose a dual decomposition based power control method and show that it solves the stackelberg game asymptotically when the number of tones becomes large

convergence of approximate and incremental subgradient methods for convex optimization we present a unified convergence framework for approximate subgradient methods that covers various stepsize rules including both diminishing and nonvanishing stepsizes convergence in objective values and convergence to a neighborhood of the optimal set we discuss ways of ensuring the boundedness of the iterates and give efficiency estimates our results are extended to incremental subgradient methods for minimizing a sum of convex functions which have recently been shown to be promising for various large scale problems including those arising from lagrangian relaxation

block iterative algorithms with underrelaxed bregman projections the notion of relaxation is well understood for orthogonal projections onto convex sets for general bregman projections it was considered only for hyperplanes and the question of how to relax bregman projections onto convex sets that are not linear i e not hyperplanes or half spaces has remained open a definition of the underrelaxation of bregman projections onto general convex sets is given here which includes as special cases the underrelaxed orthogonal projections and the underrelaxed bregman projections onto linear sets as given by de pierro and iusem j optim theory appl 51 1986 pp 421 440 with this new definition we construct a block iterative projection algorithmic scheme and prove its convergence to a solution of the convex feasibility problem the practical importance of relaxation parameters in the application of such projection algorithms to real world problems is demonstrated on a problem of image reconstruction from projections

a new class of incremental gradient methods for least squares problems the least mean squares lms method for linear least squares problems differs from the steepest descent method in that it processes data blocks one by one with intermediate adjustment of the parameter vector under optimization this mode of operation often leads to faster convergence when far from the eventual limit and to slower sublinear convergence when close to the optimal solution we embed both lms and steepest descent as well as other intermediate methods within a one parameter class of algorithms and we propose a hybrid class of methods that combine the faster early convergence rate of lms with the faster ultimate linear convergence rate of steepest descent these methods are well suited for neural network training problems with large data sets furthermore these methods allow the effective use of scaling based for example on diagonal or other approximations of the hessian matrix

beyond the regret minimization barrier optimal algorithms for stochastic strongly convex optimization we give novel algorithms for stochastic strongly convex optimization in the gradient oracle model which return a o 1 t approximate solution after t iterations the first algorithm is deterministic and achieves this rate via gradient updates and historical averaging the second algorithm is randomized and is based on pure gradient steps with a random step size his rate of convergence is optimal in the gradient oracle model this improves upon the previously known best rate of o log t t which was obtained by applying an online strongly convex optimization algorithm with regret o log t to the batch setting we complement this result by proving that any algorithm has expected regret of log t in the online stochastic strongly convex optimization setting this shows that any online to batch conversion is inherently suboptimal for stochastic strongly convex optimization this is the first formal evidence that online convex optimization is strictly more difficult than batch stochastic convex optimization

optimizing cluster structures with inner product induced norm based dissimilarity measures dissimilarity measures play a key role in exploring the inherent cluster structure of the data for any partitional clustering algorithm commonly used dissimilarity functions for clustering purpose are so far confined to the euclidean exponential and mahalanobish distances in this article we develop generalized algorithms to solve the partitional clustering problems formulated with a general class of inner product induced norm ipin based dissimilarity measures we provide an in depth mathematical analysis of the underlying optimization framework and analytically address the issue of existence of a solution and its uniqueness in absence of a closed form solution we develop a fast stochastic gradient descent algorithm and the minimization by incremental surrogate optimization miso algorithm in case of constrained optimization with exponential convergence rate to obtain the solution we carry out a convergence analysis of the fuzzy and k means clustering algorithms with the ipin based dissimilarity measures and also establish how these algorithms guarantee convergence to a stationary point in addition we investigate the nature of the stationary point novelty of the paper lies in the introduction of a generalized class of divergence measures development of fuzzy and k means clustering algorithms with the general class of divergence measures and a thorough convergence analysis of the developed algorithms

coordinated beamforming for multiuser miso interference channel under rate outage constraints this paper studies the coordinated beamforming design problem for the multiple input single output miso interference channel assuming only channel distribution information cdi known to the transmitters under a given requirement on the rate outage probability for receivers we aim to maximize the system utility e g the weighted sum rate weighted geometric mean rate and the weighed harmonic mean rate subject to the rate outage constraints and individual power constraints the outage constraints however lead to a complicated nonconvex structure for the considered beamforming design problem and render the optimization problem difficult to handle although this nonconvex optimization problem can be solved in an exhaustive search manner this brute force approach is only feasible when the number of transmitter receiver pairs is small for a system with a large number of transmitter receiver pairs computationally efficient alternatives are necessary hence the focus of this paper is the design of such efficient approximation methods in particular by employing semidefinite relaxation sdr and first order approximation techniques we propose an efficient successive convex approximation sca algorithm that provides high quality approximate beamforming solutions via solving a sequence of convex approximation problems the solution thus obtained is further shown to be a stationary point for the sdr of the original outage constrained beamforming design problem furthermore we propose a distributed sca algorithm where each transmitter optimizes its own beamformer using local cdi and information obtained from limited message exchange with the other transmitters our simulation results demonstrate that the proposed sca algorithm and its distributed counterpart indeed converge and promising performance can be achieved for all the considered system utilities

linear transceiver design for interference alignment complexity and computation consider a multiple input multiple output mimo interference channel where each transmitter and receiver are equipped with multiple antennas an effective approach to practically achieving high system throughput is to deploy linear transceivers or beamformers that can optimally exploit the spatial characteristics of the channel the recent work of cadambe and jafar ieee trans inf theory vol 54 no 8 suggests that optimal beamformers should maximize the total degrees of freedom and achieve interference alignment in the high signal to noise ratio snr regime in this paper we first consider the interference alignment problem without channel extension and prove that the problem of maximizing the total achieved degrees of freedom for a given mimo interference channel is np hard furthermore we show that even checking the achievability of a given tuple of degrees of freedom for all receivers is np hard when each receiver is equipped with at least three antennas interestingly the same problem becomes polynomial time solvable when each transmit receive node is equipped with no more than two antennas we also propose a distributed algorithm for transmit covariance matrix design that does not require the dof tuple preassignment under the assumption that each receiver uses a linear minimum mean square error mmse beamformer the simulation results show that the proposed algorithm outperforms the existing interference alignment algorithms in terms of system throughput

sufficient conditions for uniqueness of a locally optimal quantizer for a class of convex error weighting functions sufficient conditions are presented for uniqueness of a locally optimal quantizer being a stationary point of the quantization distortion measure e f x eta the expected value of an error weighting function f x eta where x is a random variable to be quantized where the probability density function p x describing x is continuous and positive on some finite or infinite interval and zero outside it and where eta is the quantization error the function f x eta is assumed convex and symmetric in eta and zero only for eta 0 it is shown that in the cases of f x eta eta 2 and f x eta eta the simple condition of concavity of in p x is sufficient for uniqueness of a locally optimal quantizer

robust guided image filtering using nonconvex potentials filtering images using a guidance signal a process called guided or joint image filtering has been used in various tasks in computer vision and computational photography particularly for noise reduction and joint upsampling this uses an additional guidance signal as a structure prior and transfers the structure of the guidance signal to an input image restoring noisy or altered image structure the main drawbacks of such a data dependent framework are that it does not consider structural differences between guidance and input images and that it is not robust to outliers we propose a novel sd for static dynamic filter to address these problems in a unified framework and jointly leverage structural information from guidance and input images guided image filtering is formulated as a nonconvex optimization problem which is solved by the majorize minimization algorithm the proposed algorithm converges quickly while guaranteeing a local minimum the sd filter effectively controls the underlying image structure at different scales and can handle a variety of types of data from different sensors it is robust to outliers and other artifacts such as gradient reversal and global intensity shift and has good edge preserving smoothing properties we demonstrate the flexibility and effectiveness of the proposed sd filter in a variety of applications including depth upsampling scale space filtering texture removal flash non flash denoising and rgb nir denoising

generalized linear precoder and decoder design for mimo channels using the weighted mmse criterion we address the problem of designing jointly optimum linear precoder and decoder for a mimo channel possibly with delay spread using a weighted minimum mean squared error mmse criterion subject to a transmit power constraint we show that the optimum linear precoder and decoder diagonalize the mimo channel into eigen subchannels for any set of error weights furthermore we derive the optimum linear precoder and decoder as functions of the error weights and consider specialized designs based on specific choices of error weights we show how to obtain 1 the maximum information rate design 2 qos based design we show how to achieve any set of relative snrs across the subchannels and 3 the unweighted mmse and equal error design for fixed rate systems

fixed point properties for nonexpansive representations of topological semigroups in this paper we study a fixed point and a nonlinear ergodic properties for an amenable semigroup of nonexpansive mappings on a nonempty subset of a hilbert space

unraveling the rank one solution mystery of robust miso downlink transmit optimization a verifiable sufficient condition via a new duality result this paper concentrates on a robust transmit optimization problem for the multiuser multi input single output miso downlink scenario and under inaccurate channel state information csi this robust problem deals with a general rank transmit covariance design and follows a safe rate constrained formulation under spherically bounded csi uncertainties curiously simulation results in previous works suggested that the robust problem admits rank one optimal transmit covariances in most cases such a numerical finding is appealing because transmission with rank one covariances can be easily realized by single stream transmit beamforming this gives rise to a fundamentally important question namely whether we can theoretically identify conditions under which the robust problem admits a rank one solution in this paper we identify one such condition simply speaking we show that the robust problem is guaranteed to admit a rank one solution if the csi uncertainties are not too large and the multiuser channel is not too poorly conditioned to establish the aforementioned condition we develop a novel duality framework through which an intimate relationship between the robust problem and a related maximin problem is revealed our condition involves only a simple expression with respect to the multiuser channel and other system parameters in particular unlike other sufficient rank one conditions that have appeared in the literature ours is verifiable the application of our analysis framework to several other csi uncertainty models is also discussed

sample path optimization of convex stochastic performance functions in this paper we propose a method for optimizing convex performance functions in stochastic systems these functions can include expected performance in static systems and steady state performance in discrete event dynamic systems they may be nonsmooth the method is closely related to retrospective simulation optimization it appears to overcome some limitations of stochastic approximation which is often applied to such problems we explain the method and give computational results for two classes of problems tandem production lines with up to 50 machines and stochastic pert program evaluation and review technique problems with up to 70 nodes and 110 arcs

nonconvex nonsmooth optimization via convex nonconvex majorization minimization the class of majorization minimization algorithms is based on the principle of successively minimizing upper bounds of the objective function each upper bound or surrogate function is locally tight at the current estimate and each minimization step decreases the value of the objective function we present a majorization minimization approach based on a novel convex nonconvex upper bounding strategy for the solution of a certain class of nonconvex nonsmooth optimization problems we propose an efficient algorithm for minimizing the convex surrogate function based on the alternating direction method of multipliers a preliminary convergence analysis for the proposed approach is provided numerical experiments show the effectiveness of the proposed method for the solution of nonconvex nonsmooth minimization problems

distributed asynchronous computation of fixed points we present an algorithmic model for distributed computation of fixed points whereby several processors participate simultaneously in the calculations while exchanging information via communication links we place essentially no assumptions on the ordering of computation and communication between processors thereby allowing for completely uncoordinated execution we provide a general convergence theorem for algorithms of this type and demonstrate its applicability to several classes of problems including the calculation of fixed points of contraction and monotone mappings arising in linear and nonlinear systems of equations optimization problems shortest path problems and dynamic programming

asymptotic mse distortion of mismatched uniform scalar quantization asymptotic formulas are derived for the mean squared error mse distortion of n level uniform scalar quantizers designed to be mse optimal for one density function but applied to another as n these formulas which are based on the euler maclaurin formula are then applied with generalized gamma bucklew gallagher and hui neuhoff density functions as the designed for and applied to densities it is found that the mismatch between the designed for and applied to densities can disturb the delicate balance between granular and overload distortions in optimal quantization with the result that generally speaking the granular or overload distortion dominates respectively depending on whether the applied to density function has a lighter or heavier tail than the designed for density specifically in the case of generalized gamma densities a variance mismatch makes overload distortion dominate for an applied to source with a slightly larger variance whereas a shape mismatch can tolerate a wider variance difference while retaining the dominance of the granular distortion in addition for the studied density functions the euler maclaurin approach is used to derive asymptotic formulas for the optimal quantizer step size in a simpler more direct way than previous approaches

analysis of sample path optimization sample path optimization is a method for optimizing limit functions occurring in stochastic modeling problems such as steady state functions in discrete event dynamic systems it is closely related to retrospective optimization techniques and to m estimation the method has been computationally tested elsewhere on problems arising in production and in project planning with apparent success in this paper we provide a mathematical justification for sample path optimization by showing that under certain assumptions which hold for the problems just mentioned the method will almost surely find a point that is in a specified sense sufficiently close to the set of optimizers of the limit function

incremental subgradient methods for nondifferentiable optimization we consider a class of subgradient methods for minimizing a convex function that consists of the sum of a large number of component functions this type of minimization arises in a dual context from lagrangian relaxation of the coupling constraints of large scale separable problems the idea is to perform the subgradient iteration incrementally by sequentially taking steps along the subgradients of the component functions with intermediate adjustment of the variables after processing each component function this incremental approach has been very successful in solving large differentiable least squares problems such as those arising in the training of neural networks and it has resulted in a much better practical rate of convergence than the steepest descent method in this paper we establish the convergence properties of a number of variants of incremental subgradient methods including some that are stochastic based on the analysis and computational experiments the methods appear very promising and effective for important classes of large problems a particularly interesting discovery is that by randomizing the order of selection of component functions for iteration the convergence rate is substantially improved

entropy density and mismatch in high rate scalar quantization with renyi entropy constraint properties of scalar quantization with th power distortion and constrained renyi entropy of order are investigated for an asymptotically high rate optimal sequence of quantizers the contribution to the renyi entropy due to source values in a fixed interval is identified in terms of the entropy density of the quantizer sequence this extends results related to the well known point density concept in optimal fixed rate quantization a dual of the entropy density result quantifies the distortion contribution of a given interval to the overall distortion the distortion loss resulting from a mismatch of source densities in the design of an asymptotically optimal sequence of quantizers is also determined this extends bucklew s fixed rate and gray et al s variable rate mismatch results to general values of the entropy order parameter

an iteratively weighted mmse approach to distributed sum utility maximization for a mimo interfering broadcast channel consider the multiple input multiple output mimo interfering broadcast channel whereby multiple base stations in a cellular network simultaneously transmit signals to a group of users in their own cells while causing interference to each other the basic problem is to design linear beamformers that can maximize the system throughput in this paper we propose a linear transceiver design algorithm for weighted sum rate maximization that is based on iterative minimization of weighted mean square error mse the proposed algorithm only needs local channel knowledge and converges to a stationary point of the weighted sum rate maximization problem furthermore the algorithm and its convergence can be extended to a general class of sum utility maximization problem the effectiveness of the proposed algorithm is validated by numerical experiments

on the support of fixed rate minimum mean squared error scalar quantizers for a laplacian source this correspondence shows that the support growth of a fixed rate optimum minimum mean squared error scalar quantizer for a laplacian density is logarithmic with the number of quantization points specifically it is shown that for a unit variance laplacian density the ratio of the support determining threshold of an optimum quantizer to 3 spl radic 2lnn 2 converges to 1 as the number n of quantization points grows also derived is a limiting upper bound that says that the support determining threshold cannot exceed the logarithmic growth by more than a small constant e g 0 0669 these results confirm the logarithmic growth of the optimum support that has previously been derived heuristically

gradient convergence in gradient methods with errors we consider the gradient method x t 1 x t g t s t w t where s t is a descent direction of a function f rn to re and w t is a deterministic or stochastic error we assume that gr f is lipschitz continuous that the stepsize g t diminishes to 0 and that s t and w t satisfy standard conditions we show that either f x t to infty or f x t converges to a finite value and gr f x t to0 with probability 1 in the stochastic case and in doing so we remove various boundedness conditions that are assumed in existing results such as boundedness from below of f boundedness of gr f x t or boundedness of xt

on the convergence of the lms algorithm with adaptive learning rate for linear feedforward networks we consider the problem of training a linear feedforward neural network by using a gradient descent like lms learning algorithm the objective is to find a weight matrix for the network by repeatedly presenting to it a finite set of examples so that the sum of the squares of the errors is minimized kohonen showed that with a small but fixed learning rate or stepsize some subsequences of the weight matrices generated by the algorithm will converge to certain matrices close to the optimal weight matrix in this paper we show that by dynamically decreasing the learning rate during each training cycle the sequence of matrices generated by the algorithm will converge to the optimal weight matrix we also show that for any given 0 the lms algorithm with decreasing learning rates will generate an optimal weight matrix i e a matrix of distance at most away from the optimal matrix after o 1 training cycles this is in contrast to 1 log 1 training cycles needed to generate an optima

a stochastic successive minimization method for nonsmooth nonconvex optimization with applications to transceiver design in wireless communication networks consider the problem of minimizing the expected value of a cost function parameterized by a random variable the classical sample average approximation method for solving this problem requires minimization of an ensemble average of the objective at each step which can be expensive in this paper we propose a stochastic successive upper bound minimization method ssum which minimizes an approximate ensemble average at each iteration to ensure convergence and to facilitate computation we require the approximate ensemble average to be a locally tight upper bound of the expected cost function and be easily optimized the main contributions of this work include the development and analysis of the ssum method as well as its applications in linear transceiver design for wireless communication networks and online dictionary learning moreover using the ssum framework we extend the classical stochastic sub gradient method to the case of minimizing a nonsmooth nonconvex objective function and establish its convergence

convergence theory for nonconvex stochastic programming with an application to mixed logit monte carlo methods have extensively been used and studied in the area of stochastic programming their convergence properties typically consider global minimizers or first order critical points of the sample average approximation saa problems and minimizers of the true problem and show that the former converge to the latter for increasing sample size however the assumption of global minimization essentially restricts the scope of these results to convex problems we review and extend these results in two directions we allow for local saa minimizers of possibly nonconvex problems and prove under suitable conditions almost sure convergence of local second order solutions of the saa problem to second order critical points of the true problem we also apply this new theory to the estimation of mixed logit models for discrete choice analysis new useful convergence properties are derived in this context both for the constrained and unconstrained cases and associated estimates of the simulation bias and variance are proposed

stochastic quasi gradient methods variance reduction via jacobian sketching we develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions our method jacsketch is motivated by novel developments in randomized numerical linear algebra and operates by maintaining a stochastic estimate of a jacobian matrix composed of the gradients of individual functions in each iteration jacsketch efficiently updates the jacobian matrix by first obtaining a random linear measurement of the true jacobian through cheap sketching and then projecting the previous estimate onto the solution space of a linear matrix equation whose solutions are consistent with the measurement the jacobian estimate is then used to compute a variance reduced unbiased estimator of the gradient our strategy is analogous to the way quasi newton methods maintain an estimate of the hessian and hence our method can be seen as a stochastic quasi gradient method our method can also be seen as stochastic gradient descent applied to a controlled stochastic optimization reformulation of the original problem where the control comes from the jacobian estimates we prove that for smooth and strongly convex functions jacsketch converges linearly with a meaningful rate dictated by a single convergence theorem which applies to general sketches we also provide a refined convergence theorem which applies to a smaller class of sketches featuring a novel proof technique based on a stochastic lyapunov function this enables us to obtain sharper complexity results for variants of jacsketch with importance sampling by specializing our general approach to specific sketching strategies jacsketch reduces to the celebrated stochastic average gradient saga method and its several existing and many new minibatch reduced memory and importance sampling variants our rate for saga with importance sampling is the current best known rate for this method resolving a conjecture by schmidt et al proceedings of the eighteenth international conference on artificial intelligence and statistics aistats 2015 san diego california 2015 the rates we obtain for minibatch saga are also superior to existing rates and are sufficiently tight as to show a decrease in total complexity as the minibatch size increases moreover we obtain the first minibatch saga method with importance sampling

optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i a generic algorithmic framework in this paper we present a generic algorithmic framework namely the accelerated stochastic approximation ac sa algorithm for solving strongly convex stochastic composite optimization sco problems while the classical stochastic approximation algorithms are asymptotically optimal for solving differentiable and strongly convex problems the ac sa algorithm when employed with proper stepsize policies can achieve optimal or nearly optimal rates of convergence for solving different classes of sco problems during a given number of iterations moreover we investigate these ac sa algorithms in more detail such as by establishing the large deviation results associated with the convergence rates and introducing an efficient validation procedure to check the accuracy of the generated solutions

the concave convex procedure the concave convex procedure cccp is a way to construct discrete time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically this procedure can be applied to almost any optimization problem and many existing algorithms can be interpreted in terms of it in particular we prove that all expectation maximization algorithms and classes of legendre minimization and variational bounding algorithms can be reexpressed in terms of cccp we show that many existing neural network and mean field theory algorithms are also examples of cccp the generalized iterative scaling algorithm and sinkhorn s algorithm can also be expressed as cccp by changing variables cccp can be used both as a new way to understand and prove the convergence of existing optimization algorithms and as a procedure for generating new algorithms

online nonnegative matrix factorization with outliers we propose a unified and systematic framework for performing online nonnegative matrix factorization in the presence of outliers our framework is particularly suited to large scale data we propose two solvers based on projected gradient descent and the alternating direction method of multipliers we prove that the sequence of objective values converges almost surely by appealing to the quasi martingale convergence theorem we also show the sequence of learned dictionaries converges to the set of stationary points of the expected loss function almost surely in addition we extend our basic problem formulation to various settings with different constraints and regularizers we also adapt the solvers and analyses to each setting we perform extensive experiments on both synthetic and real datasets these experiments demonstrate the computational efficiency and efficacy of our algorithms on tasks such as parts based basis learning image denoising shadow removal and foreground background separation

mean shift is a bound optimization we build on the current understanding of mean shift as an optimization procedure we demonstrate that in the case of piecewise constant kernels mean shift is equivalent to newton s method further we prove that for all kernels the mean shift procedure is a quadratic bound maximization

fixed point continuation for ell_1 minimization methodology and convergence we present a framework for solving the large scale ell 1 regularized convex minimization problem min x 1 mu f x our approach is based on two powerful algorithmic ideas operator splitting and continuation operator splitting results in a fixed point algorithm for any given scalar mu continuation refers to approximately following the path traced by the optimal value of x as mu increases in this paper we study the structure of optimal solution sets prove finite convergence for important quantities and establish q linear convergence rates for the fixed point algorithm applied to problems with f x convex but not necessarily strictly convex the continuation framework motivated by our convergence results is demonstrated to facilitate the construction of practical algorithms

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

steered sequential projections for the inconsistent convex feasibility problem we study a steered sequential gradient algorithm which minimizes the sum of convex functions by proceeding cyclically in the directions of the negative gradients of the functions and using steered step sizes this algorithm is applied to the convex feasibility problem by minimizing a proximity function which measures the sum of the bregman distances to the members of the family of convex sets the resulting algorithm is a new steered sequential bregman projection method

a convergent incremental gradient method with a constant step size an incremental aggregated gradient method for minimizing a sum of continuously differentiable functions is presented the method requires a single gradient evaluation per iteration and uses a constant step size for the case that the gradient is bounded and lipschitz continuous we show that the method visits infinitely often regions in which the gradient is small under certain unimodality assumptions global convergence is established in the quadratic case a global linear rate of convergence is shown the method is applied to distributed optimization problems arising in wireless sensor networks and numerical experiments compare the new method with other incremental gradient methods

primal dual subgradient methods for convex problems in this paper we present a new approach for constructing subgradient schemes for different types of nonsmooth problems with convex structure our methods are primal dual since they are always able to generate a feasible approximation to the optimum of an appropriately formulated dual problem besides other advantages this useful feature provides the methods with a reliable stopping criterion the proposed schemes differ from the classical approaches divergent series methods mirror descent methods by presence of two control sequences the first sequence is responsible for aggregating the support functions in the dual space and the second one establishes a dynamically updated scale between the primal and dual spaces this additional flexibility allows to guarantee a boundedness of the sequence of primal test points even in the case of unbounded feasible set however we always assume the uniform boundedness of subgradients we present the variants of subgradient schemes for nonsmooth convex minimization minimax problems saddle point problems variational inequalities and stochastic optimization in all situations our methods are proved to be optimal from the view point of worst case black box lower complexity bounds

mutual information and minimum mean square error in gaussian channels this paper deals with arbitrarily distributed finite power input signals observed through an additive gaussian noise channel it shows a new formula that connects the input output mutual information and the minimum mean square error mmse achievable by optimal estimation of the input given the output that is the derivative of the mutual information nats with respect to the signal to noise ratio snr is equal to half the mmse regardless of the input statistics this relationship holds for both scalar and vector signals as well as for discrete time and continuous time noncausal mmse estimation this fundamental information theoretic result has an unexpected consequence in continuous time nonlinear estimation for any input signal with finite power the causal filtering mmse achieved at snr is equal to the average value of the noncausal smoothing mmse achieved with a channel whose snr is chosen uniformly distributed between 0 and snr

convergent incremental optimization transfer algorithms application to tomography no convergent ordered subsets os type image reconstruction algorithms for transmission tomography have been proposed to date in contrast in emission tomography there are two known families of convergent os algorithms methods that use relaxation parameters and methods based on the incremental expectation maximization em approach this paper generalizes the incremental em approach by introducing a general framework incremental optimization transfer the proposed algorithms accelerate convergence speeds and ensure global convergence without requiring relaxation parameters the general optimization transfer framework allows the use of a very broad family of surrogate functions enabling the development of new algorithms this paper provides the first convergent os type algorithm for nonconcave penalized likelihood pl transmission image reconstruction by using separable paraboloidal surrogates sps which yield closed form maximization steps we found it is very effective to achieve fast convergence rates by starting with an os algorithm with a large number of subsets and switching to the new transmission incremental optimization transfer triot algorithm results show that triot is faster in increasing the pl objective than nonincremental ordinary sps and even os sps yet is convergent

asymptotic analysis of optimal fixed rate uniform scalar quantization studies the asymptotic characteristics of uniform scalar quantizers that are optimal with respect to mean squared error mse when a symmetric source density with infinite support is sufficiently well behaved the optimal step size spl delta sub n for symmetric uniform scalar quantization decreases as 2 spl sigma n sup 1 v sup 1 1 6n sup 2 where n is the number of quantization levels spl sigma sup 2 is the source variance and v sup 1 spl middot is the inverse of v y y sup 1 spl int sub y sup spl infin p spl sigma sup 1 x x dx equivalently the optimal support length n spl delta sub n increases as 2 spl sigma v sup 1 1 6n sup 2 granular distortion is asymptotically well approximated by spl delta sub n sup 2 12 and the ratio of overload to granular distortion converges to a function of the limit spl tau spl equiv lim sub y spl rarr spl infin y sup 1 e x x y provided as usually happens that spl tau exists when it does its value is related to the number of finite moments of the source density an asymptotic formula for the overall distortion d sub n is obtained and spl tau 1 is both necessary and sufficient for the overall distortion to be asymptotically well approximated by spl delta sub n sup 2 12 applying these results to the class of two sided densities of the form b x sup spl beta e spl alpha x sup spl alpha which includes gaussian laplacian gamma and generalized gaussian it is found that spl tau 1 that spl delta sub n decreases as ln n sup 1 spl alpha n that d sub n is asymptotically well approximated by spl delta sub n sup 2 12 and decreases as ln n sup 2 spl alpha n sup 2 and that more accurate approximations to spl delta sub n are possible the results also apply to densities with one sided infinite support such as rayleigh and weibull and to densities whose tails are asymptotically similar to those previously mentioned

two results on the asymptotic performance of quantizers a necessary and sufficient condition is presented for the normalized asymptotic r th power distortion of a mismatched multidimensional quantizer to converge to a certain optimum constant known as bennett s integral the dimensionality of our results is allowed to approach infinity in order to make some universal source coding comparisons to quantization theory

a parallel decomposition method for nonconvex stochastic multi agent optimization problems this paper considers the problem of minimizing the expected value of a possibly nonconvex cost function parameterized by a random vector variable when the expectation cannot be computed accurately e g because the statistics of the random variables are unknown and or the computational complexity is prohibitive classical stochastic gradient methods for solving this problem may suffer from slow convergence in this paper we propose a stochastic parallel successive convex approximation based best response algorithm for general nonconvex stochastic sum utility optimization problems which arise naturally in the design of multi agent networks the proposed novel decomposition approach enables all users to update their optimization variables in parallel by solving a sequence of strongly convex subproblems one for each user almost sure convergence to stationary points is proved we then customize the algorithmic framework to solve the stochastic sum rate maximization problem over single input single output siso frequency selective interference channels multiple input multiple output mimo interference channels and mimo multiple access channels numerical results corroborate that the proposed algorithms can converge faster than state of the art stochastic gradient schemes while achieving the same or better sum rates

general bounds and finite time improvement for the kiefer wolfowitz stochastic approximation algorithm we consider the kiefer wolfowitz kw stochastic approximation algorithm and derive general upper bounds on its mean squared error the bounds are established using an elementary induction argument and phrased directly in the terms of tuning sequences of the algorithm from this we deduce the nonnecessity of one of the main assumptions imposed on the tuning sequences by kiefer and wolfowitz kiefer j j wolfowitz 1952 stochastic estimation of the maximum of a regression function ann math statist 23 3 462 466 and essentially all subsequent literature the optimal choice of sequences is derived for various cases of interest and an adaptive version of the kw algorithm scaled and shifted kw or sskw is proposed with the aim of improving its finite time behavior the key idea is to dynamically scale and shift the tuning sequences to better match them with characteristics of the unknown function and noise level and thus improve algorithm performance numerical results are provided that illustrate that the proposed algorithm retains the convergence properties of the original kw algorithm while dramatically improving its performance in some cases

choosing parameters in block iterative or ordered subset reconstruction algorithms viewed abstractly all the algorithms considered here are designed to provide a nonnegative solution x to the system of linear equations y px where y is a vector with positive entries and p a matrix whose entries are nonnegative and with no purely zero columns the expectation maximization maximum likelihood method as it occurs in emission tomography and the simultaneous multiplicative algebraic reconstruction technique are slow to converge on large data sets accelerating convergence through the use of block iterative or ordered subset versions of these algorithms is a topic of considerable interest these block iterative versions involve relaxation and normalization parameters the correct selection of which may not be obvious to all users the algorithms are not faster merely by virtue of being block iterative the correct choice of the parameters is crucial through a detailed discussion of the theoretical foundations of these methods we come to a better understanding of the precise roles these parameters play

incremental least squares methods and the extended kalman filter in this paper we propose and analyze nonlinear least squares methods which process the data incrementally one data block at a time such methods are well suited for large data sets and real time operation and have received much attention in the context of neural network training problems we focus on the extended kalman filter which may be viewed as an incremental version of the gauss newton method we provide a nonstochastic analysis of its convergence properties and we discuss variants aimed at accelerating its convergence

on the existence of fixed points and ergodic retractions for lipschitzian semigroups in hilbert spaces and especially nonexpansive in the case of k 1 let s be a semitopological semigroup and let s c s e s be a continuous representation of s as lipschitzian mappings of c into itself which is said to be a lipschitzian semigroup on c a lipschitzian semigroup s t s e s with lipschitz constants k s e s is called a uniformly k lipschitzian semigroup and a nonexpansive semigroup when k k and k 1 for all s e s respectively goebel and kirk 7 first called a mapping t of c into itself a uniformly k lipschitzian mapping on c in the case when s t n e n is a lipschitzian semigroup with k k for all n e n lifschitz 12 obtained a fixed point theorem for uniformly k lipschitzian mappings on c with k 5 and this result was extended to uniformly k lipschitzian semigroups which are left reversible by downing and ray 6 lau l l showed that a nonexpansive semigroup has a common fixed point if there exists a left invariant mean on the space rug s of bounded right uniformly continuous functions on the semigroup furthermore ishihara and takahashi 9 extended lau s result to uniformly k lipschitzian semigroups and also gave a simpler proof of downing and ray s theorem on the other hand the first nonlinear ergodic theorem for nonexpansive mappings was established by baillon 11 let t be a nonexpansive mapping of c into itself and suppose that the set f t of fixed points of t is nonempty the the cesaro means

on stochastic gradient and subgradient methods with adaptive steplength sequences traditionally stochastic approximation sa schemes have been popular choices for solving stochastic optimization problems however the performance of standard sa implementations can vary significantly based on the choice of the steplength sequence and in general little guidance is provided about good choices motivated by this gap we present two adaptive steplength schemes for strongly convex differentiable stochastic optimization problems equipped with convergence theory that aim to overcome some of the reliance on user specific parameters the first scheme referred to as a recursive steplength stochastic approximation rsa scheme optimizes the error bounds to derive a rule that expresses the steplength at a given iteration as a simple function of the steplength at the previous iteration and certain problem parameters the second scheme termed as a cascading steplength stochastic approximation csa scheme maintains the steplength sequence as a piecewise constant decreasing function with the reduction in the steplength occurring when a suitable error threshold is met then we allow for nondifferentiable objectives but with bounded subgradients over a certain domain in such a regime we propose a local smoothing technique based on random local perturbations of the objective function that leads to a differentiable approximation of the function assuming a uniform distribution on the local randomness we establish a lipschitzian property for the gradient of the approximation and prove that the obtained lipschitz bound grows at a modest rate with problem size this facilitates the development of an adaptive steplength stochastic approximation framework which now requires sampling in the product space of the original measure and the artificially introduced distribution

the iterated kalman smoother as a gauss newton method the kalman smoother is known to be the maximum likelihood estimator when the measurement and transition functions are affine i e a linear function plus a constant a new proof of this result is presented that shows that the kalman smoother decomposes a large least squares problem into a sequence of much smaller problems the iterated kalman smoother is then presented and shown to be a gauss newton method for maximizing the likelihood function in the nonaffine case this method takes advantage of the decomposition obtained with the kalman smoother

a unified convergence analysis of the multiplicative update algorithm for regularized nonnegative matrix factorization the multiplicative update mu algorithm has been extensively used to estimate the basis and coefficient matrices in nonnegative matrix factorization nmf problems under a wide range of divergences and regularizers however theoretical convergence guarantees have only been derived for a few special divergences without regularization in this work we provide a conceptually simple self contained and unified proof for the convergence of the mu algorithm applied on nmf with a wide range of divergences and regularizers our main result shows the sequence of iterates i e pairs of basis and coefficient matrices produced by the mu algorithm converges to the set of stationary points of the nonconvex nmf optimization problem our proof strategy has the potential to open up new avenues for analyzing similar problems in machine learning and signal processing

adaptive subgradient methods for online learning and stochastic optimization we present a new family of subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient based learning metaphorically the adaptation allows us to find needles in haystacks in the form of very predictive but rarely seen features our paradigm stems from recent advances in stochastic optimization and online learning which employ proximal functions to control the gradient steps of the algorithm we describe and analyze an apparatus for adaptively modifying the proximal function which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal function that can be chosen in hindsight we give several efficient algorithms for empirical risk minimization problems with common and important regularization functions and domain constraints we experimentally study our theoretical analysis and show that adaptive subgradient methods outperform state of the art yet non adaptive subgradient algorithms

majorization minimization algorithms in signal processing communications and machine learning this paper gives an overview of the majorization minimization mm algorithmic framework which can provide guidance in deriving problem driven algorithms with low computational cost a general introduction of mm is presented including a description of the basic principle and its convergence results the extensions acceleration schemes and connection to other algorithmic frameworks are also covered to bridge the gap between theory and practice upperbounds for a large number of basic functions derived based on the taylor expansion convexity and special inequalities are provided as ingredients for constructing surrogate functions with the pre requisites established the way of applying mm to solving specific problems is elaborated by a wide range of applications in signal processing communications and machine learning

semigroup of nonexpansive mappings on a hilbert space soit s un semi groupe semi topologique soit h un espace de hilbert et s t s s s une representation continue de s comme applications non dilatation sur un sous ensemble convexe clos c de h dans c soit x c on etudie la convergence de t s x s s vers un point fixe commun z de s dans c

simultaneous variable selection we propose a new method for selecting a common subset of explanatory variables where the aim is to model several response variables the idea is a natural extension of the lasso technique proposed by tibshirani 1996 and is based on the joint residual sum of squares while constraining the parameter estimates to lie within a suitable polyhedral region the properties of the resulting convex programming problem are analyzed for the special case of an orthonormal design for the general case we develop an efficient interior point algorithm the method is illustrated on a dataset with infrared spectrometry measurements on 14 qualitatively different but correlated responses using 770 wavelengths the aim is to select a subset of the wavelengths suitable for use as predictors for as many of the responses as possible

the ordered subsets mirror descent optimization method with applications to tomography we describe an optimization problem arising in reconstructing three dimensional medical images from positron emission tomography pet a mathematical model of the problem based on the maximum likelihood principle is posed as a problem of minimizing a convex function of several million variables over the standard simplex to solve a problem of these characteristics we develop and implement a new algorithm ordered subsets mirror descent and demonstrate theoretically and computationally that it is well suited for solving the pet reconstruction problem

an incremental gradient projection method with momentum term and adaptive stepsize rule we consider an incremental gradient method with momentum term for minimizing the sum of continuously differentiable functions this method uses a new adaptive stepsize rule that decreases the stepsize whenever sufficient progress is not made we show that if the gradients of the functions are bounded and lipschitz continuous over a certain level set then every cluster point of the iterates generated by the method is a stationary point in addition if the gradient of the functions have a certain growth property then the method is either linearly convergent in some sense or the stepsizes are bounded away from zero the new stepsize rule is much in the spirit of heuristic learning rules used in practice for training neural networks via backpropagation as such the new stepsize rule may suggest improvements on existing learning rules finally extension of the method and the convergence results to constrained minimization is discussed as are some implementation issues and numerical experience

incremental proximal methods for large scale convex optimization we consider the minimization of a sum sum i 1 mf i x consisting of a large number of convex component functions f i for this problem incremental methods consisting of gradient or subgradient iterations applied to single components have proved very effective we propose new incremental methods consisting of proximal iterations applied to single components as well as combinations of gradient subgradient and proximal iterations we provide a convergence and rate of convergence analysis of a variety of such methods including some that involve randomization in the selection of components we also discuss applications in a few contexts including signal processing and inference machine learning

incremental majorization minimization optimization with application to large scale machine learning majorization minimization algorithms consist of successively minimizing a sequence of upper bounds of the objective function these upper bounds are tight at the current estimate and each iteration monotonically drives the objective function downhill such a simple principle is widely applicable and has been very popular in various scientific fields especially in signal processing and statistics we propose an incremental majorization minimization scheme for minimizing a large sum of continuous functions a problem of utmost importance in machine learning we present convergence guarantees for nonconvex and convex optimization when the upper bounds approximate the objective up to a smooth error we call such upper bounds first order surrogate functions more precisely we study asymptotic stationary point guarantees for nonconvex problems and for convex ones we provide convergence rates for the expected objective function value we apply our scheme to composite optimization and obtain a new incremental proximal gradient algorithm with linear convergence rate for strongly convex functions our experiments show that our method is competitive with the state of the art for solving machine learning problems such as logistic regression when the number of training samples is large enough and we demonstrate its usefulness for sparse estimation with nonconvex penalties

decomposition by partial linearization parallel optimization of multi agent systems we propose a novel decomposition framework for the distributed optimization of general nonconvex sum utility functions arising naturally in the system design of wireless multi user interfering systems our main contributions are i the development of the first class of inexact jacobi best response algorithms with provable convergence where all the users simultaneously and iteratively solve a suitably convexified version of the original sum utility optimization problem ii the derivation of a general dynamic pricing mechanism that provides a unified view of existing pricing schemes that are based instead on heuristics and iii a framework that can be easily particularized to well known applications giving rise to very efficient practical jacobi or gauss seidel algorithms that outperform existing ad hoc methods proposed for very specific problems interestingly our framework contains as special cases well known gradient algorithms for nonconvex sum utility problems and many block coordinate descent schemes for convex functions