dc programming and dca thirty years of developments the year 2015 marks the 30th birthday of dc difference of convex functions programming and dca dc algorithms which constitute the backbone of nonconvex programming and global optimization in this article we offer a short survey on thirty years of developments of these theoretical and algorithmic tools the survey is comprised of three parts in the first part we present a brief history of the field while in the second we summarize the state of the art results and recent advances we focus on main theoretical results and dca solvers for important classes of difficult nonconvex optimization problems and then give an overview of real world applications whose solution methods are based on dca the third part is devoted to new trends and important open issues as well as suggestions for future developments

large scale molecular optimization from distance matrices by a d c optimization approach a so called dca method based on a d c difference of convex functions optimization approach algorithm for solving large scale distance geometry problems is developed different formulations of equivalent d c programs in the l 1 approach are stated via the lagrangian duality without gap relative to d c programming and new nonstandard nonsmooth reformulations in the l infty approach resp the l 1 l infty approach are introduced substantial subdifferential calculations permit us to compute sequences of iterations in the dca quite simply the computations actually require matrix vector products and only one cholesky factorization resp with an additional solution of a convex program in the l 1 approach resp the l 1 l infty approach and allow the exploitation of sparsity in the large scale setting two techniques respectively using shortest paths between all pairs of atoms to generate the complete dissimilarity matrix and the spanning trees procedure are investigated in order to compute a good starting point for the dca finally many numerical simulations of the molecular optimization problems with up to 12567 variables are reported which prove the practical usefulness of the nonstandard nonsmooth reformulations the globality of found solutions and the robustness and efficiency of our algorithms

the structured sensitivity of vandermonde like systems we consider a general class of structured matrices that includes possibly confluent vandermonde and vandermonde like matrices here the entries in the matrix depend nonlinearly upon a vector of parameters we define condition numbers that measure the componentwise sensitivity of the associated primal and dual solutions to small componentwise perturbations in the parameters and in the right hand side convenient expressions are derived for the infinity norm based condition numbers and order of magnitude estimates are given for condition numbers defined in terms of a general vector norm we then discuss the computation of the corresponding backward errors after linearising the constraints we derive an exact expression for the infinity norm dual backward error and show that the corresponding primal backward error is given by the minimum infinity norm solution of an underdetermined linear system exact componentwise condition numbers are also derived for matrix inversion and the least squares problem and the linearised least squares backward error is characterised

quadratically constrained least squares and quadratic problems we consider the following problem compute a vectorx such that ax b 2 min subject to the constraint x 2 a new approach to this problem based on gauss quadrature is given the method is especially well suited when the dimensions ofa are large and the matrix is sparse it is also possible to extend this technique to a constrained quadratic form for a symmetric matrixa we consider the minimization ofx t a x 2b t x subject to the constraint x 2 some numerical examples are given

a new matrix free algorithm for the large scale trust region subproblem we present a new method for the large scale trust region subproblem the method is matrix free in the sense that only matrix vector products are required we recast the trust region subproblem as a parameterized eigenvalue problem and compute an optimal value for the parameter we then find the solution of the trust region subproblem from the eigenvectors associated with two of the smallest eigenvalues of the parameterized eigenvalue problem corresponding to the optimal parameter the new algorithm uses a different interpolating scheme than existing methods and introduces a unified iteration that naturally includes the so called hard case we show that the new iteration is well defined and convergent at a superlinear rate we present computational results to illustrate convergence properties and robustness of the method

interplay between synchronization observability and dynamics synchronizing nonidentical chaotic oscillators is very often achieved by using various types of couplings in the practice of synchronization the right choice of the coupling variable char22 y for the r ossler system x for the lorenz equations and so on char22 is usually stated rather than explained or justified using the r ossler and rucklidge system in this paper it is shown that such optimal choices are strongly related to observability properties which in turn can be quantified in this paper it will also be shown that synchronizability does not only depend on the observability of the system but it is also a consequence of the dynamical regimes under study the paper aims at providing important insight into the critical problem of making the right choice when it comes to choosing the coupling variable in synchronization schemes

estimating system parameters from chaotic time series with synchronization optimized by a genetic algorithm a method is proposed to estimate system parameters by optimizing synchronization with a genetic algorithm this method can effectively find the parameter values of a chaotic system with a rugged parameter landscape furthermore even the parameters of a 200 dimensional coupled map lattice spatiotemporal chaotic system can be extracted from a scalar time series finally a chua s circuit experiment shows the capacity of this method to estimate multiple parameters of real systems

estimating the matrixp norm the holderp norm of anm n matrix has no explicit representation unlessp 1 2 or it is shown here that thep norm can be estimated reliably ino mn operations a generalization of the power method is used with a starting vector determined by a technique with a condition estimation flavour the algorithm nearly always computes ap norm estimate correct to the specified accuracy and the estimate is always within a factorn 1 1 p of a p as a by product a new way is obtained to estimate the 2 norm of a rectangular matrix this method is more general and produces better estimates in practice than a similar technique of cline conn and van loan

solving the trust region subproblem by a generalized eigenvalue problem the state of the art algorithms for solving the trust region subproblem trs are based on an iterative process involving solutions of many linear systems eigenvalue problems subspace optimization or line search steps a relatively underappreciated fact due to gander golub and von matt linear algebra appl 114 1989 pp 815 839 is that trss can be solved by one generalized eigenvalue problem with no outer iterations in this paper we rediscover this fact and discover its great practicality which exhibits good performance both in accuracy and efficiency moreover we generalize the approach in various directions namely by allowing for an ellipsoidal constraint dealing with the so called hard case and obtaining approximate solutions efficiently when high accuracy is unnecessary we demonstrate that the resulting algorithm is a general purpose trs solver effective both for dense and large sparse problems including the so called hard case our algorithm is easy to implement its essence i

on the extremal structure of least upper bound norms and their dual abstract given finite dimensional real or complex banach spaces e and f with norms e r and f r we denote by n the least upper bound norm induced on l e f some results are given on the extremal structures of b the unit ball of n of its polar b and of b which is the polar of the unit ball of the least upper bound norm n the exposed faces the extreme points and a large family of other faces of b and b are presented it turns out that b is a subset of b the set of tangency points of the surfaces of b and b is completely determined and represented as the union of the exposed faces of b which are normal to rank one mappings we determine sharp bounds on the ranks of mappings in these exposed faces

new and efficient dca based algorithms for minimum sum of squares clustering the purpose of this paper is to develop new efficient approaches based on dc difference of convex functions programming and dca dc algorithm to perform clustering via minimum sum of squares euclidean distance we consider the two most widely used models for the so called minimum sum of squares clustering mssc in short that are a bilevel programming problem and a mixed integer program firstly the mixed integer formulation of mssc is carefully studied and is reformulated as a continuous optimization problem via a new result on exact penalty technique in dc programming dca is then investigated to the resulting problem secondly we introduce a gaussian kernel version of the bilevel programming formulation of mssc named gkmssc the gkmssc problem is formulated as a dc program for which a simple and efficient dca scheme is developed a regularization technique is investigated for exploiting the nice effect of dc decomposition and a simple procedure for finding good starting points of dca is developed the proposed dca schemes are original and very inexpensive because they amount to computing at each iteration the projection of points onto a simplex and or onto a ball and or onto a box which are all determined in the explicit form numerical results on real word datasets show the efficiency the scalability of dca and its great superiority with respect to k means and kernel k means standard methods for clustering highlightsthe purpose of this paper is to develop new efficient approaches based on dc programming and dca for clustering we consider the two models for the minimum sum of squares clustering mssc a mixed integer program and a bilevel programming formulation the mixed integer program is reformulated as a dc program via an exact penalty technique for which dca is investigated a gaussian kernel version of the bilevel programming formulation is introduced and a simple and efficient dca scheme is developed the two main features of dca say the effect of dc decomposition and of starting points are carefully studied

double regularization methods for robust feature selection and svm classification via dc programming abstract in this work two novel formulations for embedded feature selection are presented a second order cone programming approach for support vector machines is extended by adding a second regularizer to encourage feature elimination the one and the zero norm penalties are used in combination with the tikhonov regularization under a robust setting designed to correctly classify instances up to a predefined error rate even for the worst data distribution the use of the zero norm leads to a nonconvex formulation which is solved by using difference of convex dc functions extending dc programming to second order cones experiments on high dimensional microarray datasets were performed and the best performance was obtained with our approaches compared with well known feature selection methods for support vector machines

estimation of parameters in nonlinear systems using balanced synchronization using synchronization between observations and a model with undetermined parameters is a natural way to complete the specification of the model the quality of the synchronization a cost function to be minimized typically is evaluated by a least squares difference between the data time series and the model time series if the coupling between the data and the model is too strong this cost function is small for any data and any model and the variation of the cost function with respect to the parameters of interest is too small to permit selection of an optimal value of the parameters we introduce two methods for balancing the competing desires of a small cost function for the quality of the synchronization and the numerical ability to determine parameters accurately one method of balanced synchronization adds to the synchronization cost function a requirement that the conditional lyapunov exponent of the model system conditioned on being driven by the data remain negative but small in magnitude the other method allows the coupling between the data and the model to vary in time according to the error in synchronization this method succeeds because the data and the model exhibit generalized synchronization in the region where the parameters of the model are well determined examples are explored which have deterministic chaos with and without noise in the data signal

an efficient algorithm for globally minimizing a quadratic function under convex quadratic constraints in this paper we investigate two approaches to minimizing a quadratic form subject to the intersection of finitely many ellipsoids the first approach is the d c difference of convex functions optimization algorithm abbr dca whose main tools are the proximal point algorithm and or the projection subgradient method in convex minimization the second is a branch and bound scheme using lagrangian duality for bounding and ellipsoidal bisection in branching the dca was first introduced by pham dinh in 1986 for a general d c program and later developed by our various work is a local method but from a good starting point it provides often a global solution this motivates us to combine the dca and our branch and bound algorithm in order to obtain a good initial point for the dca and to prove the globality of the dca in both approaches we attempt to use the ellipsoidal constrained quadratic programs as the main subproblems the idea is based upon the fact that these programs can be efficiently solved by some available polynomial and nonpolynomial time algorithms among them the dca with restarting procedure recently proposed by pham dinh and le thi has been shown to be the most robust and fast for large scale problems several numerical experiments with dimension up to 200 are given which show the effectiveness and the robustness of the dca and the combined dca branch and bound algorithm

a d c optimization algorithm for solving the trust region subproblem this paper is devoted to difference of convex functions d c optimization d c duality local and global optimality conditions in d c programming the d c algorithm dca and its application to solving the trust region problem the dca is an iterative method that is quite different from well known related algorithms thanks to the particular structure of the trust region problem the dca is very simple requiring only matrix vector products and in practice converges to the global solution the inexpensive implicitly restarted lanczos method of sorensen is used to check the optimality of solutions provided by the dca when a nonglobal solution is found a simple numerical procedure is introduced both to find a feasible point having a smaller objective value and to restart the dca at this point it is shown that in the nonconvex case the dca converges to the global solution of the trust region problem using only matrix vector products and requiring at most 2m 2 restarts where m is the number of distinct negative eigenvalues of the coefficient matrix that defines the problem numerical simulations establish the robustness and efficiency of the dca compared to standard related methods especially for large scale problems

improving m sbl for joint sparse recovery using a subspace penalty a multiple measurement vector problem mmv is a generalization of the compressed sensing problem that addresses the recovery of a set of jointly sparse signal vectors one of the important contributions of this paper is to show that the seemingly least related state of the art mmv joint sparse recovery algorithms the m sbl multiple sparse bayesian learning and subspace based hybrid greedy algorithms have a very important link more specifically we show that replacing the log det cdot term in the m sbl by a rank surrogate that exploits the spark reduction property discovered in the subspace based joint sparse recovery algorithms provides significant improvements in particular if we use the schatten p quasi norm as the corresponding rank surrogate the global minimizer of the cost function in the proposed algorithm becomes identical to the true solution as p rightarrow 0 furthermore under regularity conditions we show that convergence to a local minimizer is guaranteed using an alternating minimization algorithm that has closed form expressions for each of the minimization steps which are convex numerical simulations under a variety of scenarios in terms of snr and the condition number of the signal amplitude matrix show that the proposed algorithm consistently outperformed the m sbl and other state of the art algorithms

twenty years of linear programming based portfolio optimization markowitz formulated the portfolio optimization problem through two criteria the expected return and the risk as a measure of the variability of the return the classical markowitz model uses the variance as the risk measure and is a quadratic programming problem many attempts have been made to linearize the portfolio optimization problem several different risk measures have been proposed which are computationally attractive as for discrete random variables they give rise to linear programming lp problems about twenty years ago the mean absolute deviation mad model drew a lot of attention resulting in much research and speeding up development of other lp models further the lp models based on the conditional value at risk cvar have a great impact on new developments in portfolio optimization during the first decade of the 21st century the lp solvability may become relevant for real life decisions when portfolios have to meet side constraints and take into account transaction costs or when large size instances have to be solved in this paper we review the variety of lp solvable portfolio optimization models presented in the literature the real features that have been modeled and the solution approaches to the resulting models in most of the cases mixed integer linear programming milp models we also discuss the impact of the inclusion of the real features

convergence of a subgradient method for computing the bound norm of matrices abstract let and be any norms on r m and r n respectively we study a subgradient method for computing the associated bound norm s a sup ax x 1 a nonconvex optimization problem it is proved that homodual method converges when one of the norms and is polyhedral

an exact algorithm for nonconvex quadratic integer minimization using ellipsoidal relaxations we propose a branch and bound algorithm for minimizing a not necessarily convex quadratic function over integer variables the algorithm is based on lower bounds computed as continuous minima of the objective function over appropriate ellipsoids in the nonconvex case we use ellipsoids enclosing the feasible region of the problem in spite of the nonconvexity these minima can be computed quickly the corresponding optimization problems are equivalent to trust region subproblems we present several ideas that allow us to accelerate the solution of the continuous relaxation within a branch and bound scheme and examine the performance of the overall algorithm by computational experiments good computational performance is shown especially for ternary instances

multicategory ψ learning in binary classification margin based techniques usually deliver high performance as a result a multicategory problem is often treated as a sequence of binary classifications in the absence of a dominating class this treatment may be suboptimal and may yield poor performance such as for support vector machines svms we propose a novel multicategory generalization of learning that treats all classes simultaneously the new generalization eliminates this potential problem while at the same time retaining the desirable properties of its binary counterpart we develop a statistical learning theory for the proposed methodology and obtain fast convergence rates for both linear and nonlinear learning examples we demonstrate the operational characteristics of this method through a simulation our results indicate that the proposed methodology can deliver accurate class prediction and is more robust against extreme observations than its svm counterpart

linear convergence of a type of iterative sequences in nonconvex quadratic programming abstract by using error bounds for affine variational inequalities we prove that any iterative sequence generated by the projection dc difference of convex functions decomposition algorithm in quadratic programming is r linearly convergent provided that the original problem has solutions our result solves in the affirmative the first part of the conjecture stated by le thi pham dinh and yen in their recent paper 8 p 489

sparse estimation of a covariance matrix we suggest a method for estimating a covariance matrix on the basis of a sample of vectors drawn from a multivariate normal distribution in particular we penalize the likelihood with a lasso penalty on the entries of the covariance matrix this penalty plays two important roles it reduces the effective number of parameters which is important even when the dimension of the vectors is smaller than the sample size since the number of parameters grows quadratically in the number of variables and it produces an estimate which is sparse in contrast to sparse inverse covariance estimation our method s close relative the sparsity attained here is in the covariance matrix itself rather than in the inverse matrix zeros in the covariance matrix correspond to marginal independencies thus our method performs model selection while providing a positive definite estimate of the covariance the proposed penalized maximum likelihood problem is not convex so we use a majorize minimize approach in which we iteratively solve convex approximations to the original nonconvex problem we discuss tuning parameter selection and demonstrate on a flow cytometry dataset how our method produces an interpretable graphical display of the relationship between variables we perform simulations that suggest that simple elementwise thresholding of the empirical covariance matrix is competitive with our method for identifying the sparsity structure additionally we show how our method can be used to solve a previously studied special case in which a desired sparsity pattern is prespecified

a proof of convergence of the concave convex procedure using zangwill s theory the concave convex procedure cccp is an iterative algorithm that solves d c difference of convex functions programs as a sequence of convex programs in machine learning cccp is extensively used in many learning algorithms including sparse support vector machines svms transductive svms and sparse principal component analysis though cccp is widely used in many applications its convergence behavior has not gotten a lot of specific attention yuille and rangarajan analyzed its convergence in their original paper however we believe the analysis is not complete the convergence of cccp can be derived from the convergence of the d c algorithm dca proposed in the global optimization literature to solve general d c programs whose proof relies on d c duality in this note we follow a different reasoning and show how zangwill s global convergence theory of iterative algorithms provides a natural framework to prove the convergence of cccp this underlines zangwill s theory as a powerful and general framework to deal with the convergence issues of iterative algorithms after also being used to prove the convergence of algorithms like expectation maximization and generalized alternating minimization in this note we provide a rigorous analysis of the convergence of cccp by addressing two questions when does cccp find a local minimum or a stationary point of the d c program under consideration and when does the sequence generated by cccp converge we also present an open problem on the issue of local convergence of cccp

single shooting homotopy method for parameter identification in dynamical systems an algorithm for identifying parameters in dynamical systems is developed in this work using homotopy transformations and the single shooting method the equations governing the dynamics of the mathematical model are augmented with observer like homotopy terms that smooth the objective function as a result premature convergence to a local minimum is avoided and the obtained parameter estimates are globally optimal numerical examples are presented to demonstrate the application of the proposed approach to chaotic systems

sparse view spectral ct reconstruction using spectral patch based low rank penalty spectral computed tomography ct is a promising technique with the potential for improving lesion detection tissue characterization and material decomposition in this paper we are interested in kvp switching based spectral ct that alternates distinct kvp x ray transmissions during gantry rotation this system can acquire multiple x ray energy transmissions without additional radiation dose however only sparse views are generated for each spectral measurement and the spectra themselves are limited in number to address these limitations we propose a penalized maximum likelihood method using spectral patch based low rank penalty which exploits the self similarity of patches that are collected at the same position in spectral images the main advantage is that the relatively small number of materials within each patch allows us to employ the low rank penalty that is less sensitive to intensity changes while preserving edge directions in our optimization formulation the cost function consists of the poisson log likelihood for x ray transmission and the nonconvex patch based low rank penalty since the original cost function is difficult to minimize directly we propose an optimization method using separable quadratic surrogate and concave convex procedure algorithms for the log likelihood and penalty terms which results in an alternating minimization that provides a computational advantage because each subproblem can be solved independently we performed computer simulations and a real experiment using a kvp switching based spectral ct with sparse view measurements and compared the proposed method with conventional algorithms we confirmed that the proposed method improves spectral images both qualitatively and quantitatively furthermore our gpu implementation significantly reduces the computational cost

globally solving the trust region subproblem using simple first order methods we consider the trust region subproblem which is given by a minimization of a quadratic not necessarily convex function over the euclidean ball based on the well known second order necessary and

perturbations of smooth actions with nontrivial cohomology we prove a general perturbation result for smooth lie group actions with nontrivial finite dimensional cohomology it describes sufficient conditions on cohomology over an action which imply that the action lies in a finite dimensional family of actions such that any small perturbation of the family intersects the smooth conjugacy class of the given action we cast the classical kam result on perturbations of diophantine vector fields on tori into this general setup and we address a few applications and potential applications of this result to homogeneous lie group actions with finite dimensional first cohomology 2014 wiley periodicals inc

estimating hadamard operator norms with application to triangular truncation abstract it is shown how the power method can be used to estimate hadamard operator norms its application to the problem of finding the norm of the triangular truncation operator is considered

use of synchronization and adaptive control in parameter estimation from a time series a technique is introduced for estimating unknown parameters when a time series of only one variable from a multivariate nonlinear dynamical system is given the technique employs a combination of two different control methods linear feedback for synchronizing system variables and adaptive control and is based on dynamic minimization of synchronization error the technique is shown to work even when the unknown parameters appear in the evolution equations of the variables other than the one for which the time series is given the technique not only establishes that explicit detailed information about all system variables and parameters is contained in a scalar time series but also gives a way to extract it out under suitable conditions illustrations are presented for lorenz and rossler systems and a nonlinear dynamical system in plasma physics also it is found that the technique is reasonably stable against noise in the given time series and the estimated value of a parameter fluctuates around the correct value with the error of estimation growing linearly with the noise strength for small noise

kernelized sorting object matching is a fundamental operation in data analysis it typically requires the definition of a similarity measure between the classes of objects to be matched instead we develop an approach which is able to perform matching by requiring a similarity measure only within each of the classes this is achieved by maximizing the dependency between matched pairs of observations by means of the hilbert schmidt independence criterion this problem can be cast as one of maximizing a quadratic assignment problem with special structure and we present a simple algorithm for finding a locally optimal solution

newton s method with a model trust region modification a modified newton method for unconstrained minimization is presented and analyzed the modification is based upon the model trust region approach this report contains a thorough analysis of the locally constrained quadratic minimizations that arise as subproblems in the modified newton iteration several promising alternatives are presented for solving these subproblems in ways that overcome certain theoretical difficulties exposed by this analysis very strong convergence results are presented concerning the minimization algorithm in particular the explicit use of second order information is justified by demonstrating that the iterates converge to a point which satisfies the second order necessary conditions for minimization with the exception of very pathological cases this occurs whenever the algorithm is applied to problems with continuous second partial derivatives

estimating the largest elements of a matrix we derive an algorithm for estimating the largest p a a a 1 values a ij or a ij for an m a a n matrix a along with their locations in the matrix the matrix is accessed using only matrixa a a vector or matrixa a a matrix products for p 1 the algorithm estimates the norm a m max i j a ij or max i j a ij the algorithm is based on a power method for mixed subordinate matrix norms and iterates on n a a t matrices where t a a a p is a parameter for p t 1 we show that the algorithm is essentially equivalent to rook pivoting in gaussian elimination we also obtain a bound for the expected number of matrixa a a vector products for random matrices and give a class of counter examples our numerical experiments show that for p 1 the algorithm usually converges in just two iterations requiring the equivalent of 4t matrixa a a vector products and for t 2 the algorithm already provides excellent estimates that are usually within a factor 2 of the largest element and frequently exact for p 1 we incorporate deflation to improve the performance of the algorithm experiments on real life datasets show that the algorithm is highly effective in practice

minimization of a large scale quadratic function subject to a spherical constraint an important problem in linear algebra and optimization is the trust region subproblem minimize a quadratic function subject to an ellipsoidal or spherical constraint this basic problem has several important large scale applications including seismic inversion and forcing convergence in optimization methods existing methods to solve the trust region subproblem require matrix factorizations which are not feasible in the large scale setting this paper presents an algorithm for solving the large scale trust region subproblem that requires a fixed size limited storage proportional to the order of the quadratic and that relies only on matrix vector products the algorithm recasts the trust region subproblem in terms of a parameterized eigenvalue problem and adjusts the parameter with a superlinearly convergent iteration to find the optimal solution from the eigenvector of the parameterized problem only the smallest eigenvalue and corresponding eigenvector of the parameterized problem needs to be computed the implicitly restarted lanczos method is well suited to this subproblem

stability of linear quadratic minimization over euclidean balls stability properties of the problem of minimizing a nonconvex linear quadratic function over a euclidean ball known as the trust region subproblem are studied in this paper we investigate in detail the case where the linear part of the objective function is perturbed and obtain necessary and sufficient conditions for the upper lower semicontinuity of the karush kuhn tucker kkt point set map and the global solution map explicit formulas for computing the directional derivative and the frechet derivative of the optimal value function stability of the karush kuhn tucker point set under the perturbation of the quadratic form is also studied

a class of linear generalized equations solution stability of a class of linear generalized equations in finite dimensional euclidean spaces is investigated by means of generalized differentiation exact formulas for the frechet and the mordukhovich coderivatives of the normal cone mappings of perturbed euclidean balls are obtained necessary and sufficient conditions for the local lipschitz like property of the solution maps of such linear generalized equations are derived from these coderivative formulas since the trust region subproblems in nonlinear programming can be regarded as linear generalized equations these conditions lead to new results on stability of the parametric trust region subproblems

algorithmes de calcul du maximum des formes quadratiques sur la boule unite de la norme du maximum certaines methodes directes et indirectes pour le calcul de max x t ax x 1 sont etudiees les methodes directes sont basees sur les proprietes particulieres des normes 1 2 et ces methodes sont tres simples mais ne s appliquent qu a certaines familles de matrices la methode indirecte est la methode autoduale introduite dans 25 26 avec 1 dans ce cas le choix du vecteur initial pour qu il y ait convergence vers une solution optimale est largement discute some direct and indirect methods are studied for computing max x t ax x 1 wherea is symmetric definite positive direct methods are constructed using particular properties of 1 2 norms these methods are very simple but uniquely suitable to certains families of matrices the indirect method is the autodual method introduced in 25 26 29 with 1 in this case the problem of choosing an initial vector so that convergence of the iterative sequence occurs to an optimal solution is largely discussed

a difference of convex functions algorithm for switched linear regression this paper deals with switched linear system identification and more particularly aims at solving switched linear regression problems in a large scale setting with both numerous data and many parameters to learn we consider the recent minimum of error framework with a quadratic loss function in which an objective function based on a sum of minimum errors with respect to multiple submodels is to be minimized the paper proposes a new approach to the optimization of this nonsmooth and nonconvex objective function which relies on difference of convex dc functions programming in particular we formulate a proper dc decomposition of the objective function which allows us to derive a computationally efficient dc algorithm numerical experiments show that the method can efficiently and accurately learn switching models in large dimensions and from many data points

a stochastic successive minimization method for nonsmooth nonconvex optimization with applications to transceiver design in wireless communication networks consider the problem of minimizing the expected value of a cost function parameterized by a random variable the classical sample average approximation method for solving this problem requires minimization of an ensemble average of the objective at each step which can be expensive in this paper we propose a stochastic successive upper bound minimization method ssum which minimizes an approximate ensemble average at each iteration to ensure convergence and to facilitate computation we require the approximate ensemble average to be a locally tight upper bound of the expected cost function and be easily optimized the main contributions of this work include the development and analysis of the ssum method as well as its applications in linear transceiver design for wireless communication networks and online dictionary learning moreover using the ssum framework we extend the classical stochastic sub gradient method to the case of minimizing a nonsmooth nonconvex objective function and establish its convergence

maximum margin clustering made practical motivated by the success of large margin methods in supervised learning maximum margin clustering mmc is a recent approach that aims at extending large margin methods to unsupervised learning however its optimization problem is nonconvex and existing mmc methods all rely on reformulating and relaxing the nonconvex optimization problem as semidefinite programs sdp though sdp is convex and standard solvers are available they are computationally very expensive and only small data sets can be handled to make mmc more practical we avoid sdp relaxations and propose in this paper an efficient approach that performs alternating optimization directly on the original nonconvex problem a key step to avoid premature convergence in the resultant iterative procedure is to change the loss function from the hinge loss to the laplacian square loss so that overconfident predictions are penalized experiments on a number of synthetic and real world data sets demonstrate that the proposed approach is more accurate much faster hundreds to tens of thousands of times faster and can handle data sets that are hundreds of times larger than the largest data set reported in the mmc literature

a continuous dc programming approach to the strategic supply chain design problem from qualified partner set we present a new continuous approach based on the dc difference of convex functions programming and dc algorithms dca to the problem of supply chain design at the strategic level when production of a new market opportunity has to be launched among a set of qualified partners a well known formulation of this problem is the mixed integer linear program in this paper we reformulate this problem as a dc program by using an exact penalty technique the proposed algorithm is a combination of dca and branch and bound scheme it works in a continuous domain but provides mixed integer solutions numerical simulations on many empirical data sets show the efficiency of our approach with respect to the standard branch and bound algorithm

on image reconstruction algorithms for binary electromagnetic geotomography in this paper we discuss the selected image reconstruction methods of binary tomography in the context of their application to geophysical imaging we restrict our considerations to a discrete version of high frequency electromagnetic geotomography which we label as binary electromagnetic geotomography beg basically such an imaging technique may be applied to detect subsurface anomalies air filled voids whose attenuation coefficient is very low nearly zero value and considerably different from that for the background the assumption for a binary representation of the image to be reconstructed substantially relaxes image reconstruction problems related to ill posedness that comes from an intrinsic limitation of an angular range of projections we test two algorithms for binary tomography where the penalty term is based on the markov random field mrf model the mean field reference distribution and mean field annealing are applied to estimate the global maximizer of the gibbs boltzmann distribution associated with the objective function we also apply the projected gradient algorithm that uses a binary steering very efficient implementations of the algorithms are also given the numerical results are presented for noise free noisy and real data

fixed point continuation for ell_1 minimization methodology and convergence we present a framework for solving the large scale ell 1 regularized convex minimization problem min x 1 mu f x our approach is based on two powerful algorithmic ideas operator splitting and continuation operator splitting results in a fixed point algorithm for any given scalar mu continuation refers to approximately following the path traced by the optimal value of x as mu increases in this paper we study the structure of optimal solution sets prove finite convergence for important quantities and establish q linear convergence rates for the fixed point algorithm applied to problems with f x convex but not necessarily strictly convex the continuation framework motivated by our convergence results is demonstrated to facilitate the construction of practical algorithms

on some properties of quadratic programs with a convex quadratic constraint in this paper we consider the problem of minimizing a possibly nonconvex quadratic function with a quadratic constraint we point out some new properties of the problem in particular in the first part of the paper we show that i given a kkt point that is not a global minimizer it is easy to find a better feasible point ii strict complementarity holds at the local nonglobal minimizer in the second part of this paper we show that the original constrained problem is equivalent to the unconstrained minimization of a piecewise quartic merit function using the unconstrained formulation we give in the nonconvex case a new second order necessary condition for global minimizers in the third part of this paper algorithmic applications of the preceding results are briefly outlined and some preliminary numerical experiments are reported

local minimizers of quadratic functions on euclidean balls and spheres in this paper a characterization of the local nonglobal minimizes of a quadratic function defined on a euclidean ball or a sphere is given it is proven that there exists at most one local nonglobal minimizes and that the lagrange multiplier that corresponds to this minimizes is the largest solution of a nonlinear scalar equation an algorithm is proposed for computing the local nonglobal minimizes

a new nonconvex approach for image restoration with gamma noise abstract in this paper a new total generalized variational tgv model for restoring images with multiplicative noise is proposed which contains a nonconvex fidelity term and a tgv term we use a difference of convex functions algorithm dca to deal with the proposed model for multiplicative noise removal there exist many models and algorithms most of which focus on convex approximation so that numerical algorithms with guaranteed convergence can be designed unlike these algorithms we use the dca algorithm to remove multiplicative noise by numerical experiments it is shown that the proposed approach leads to a better solution compared with the gradient projection algorithm for solving the classic multiplicative noise removal models we prove that the sequence generated by the dca algorithm converges to a stationary point which satisfies the first order optimality condition finally we demonstrate the performance of our whole scheme by numerical examples a comparison with other methods is provided as well numerical results demonstrate that the proposed algorithm significantly outperforms some previous methods for multiplicative gamma noise removal

efficient nonnegative matrix factorization by dc programming and dca in this letter we consider the nonnegative matrix factorization nmf problem and several nmf variants two approaches based on dc difference of convex functions programming and dca dc algorithm are developed the first approach follows the alternating framework that requires solving at each iteration two nonnegativity constrained least squares subproblems for which dca based schemes are investigated the convergence property of the proposed algorithm is carefully studied we show that with suitable dc decompositions our algorithm generates most of the standard methods for the nmf problem the second approach directly applies dca on the whole nmf problem two algorithms one computing all variables and one deploying a variable selection strategy are proposed the proposed methods are then adapted to solve various nmf variants including the nonnegative factorization the smooth regularization nmf the sparse regularization nmf the multilayer nmf the convex convex hull nmf and the symmetric nmf we also show that our algorithms include several existing methods for these nmf variants as special versions the efficiency of the proposed approaches is empirically demonstrated on both real world and synthetic data sets it turns out that our algorithms compete favorably with five state of the art alternating nonnegative least squares algorithms

multiclass feature selection with kernel gram matrix based criteria feature selection has been an important issue in recent decades to determine the most relevant features according to a given classification problem numerous methods have emerged that take into account support vector machines svms in the selection process such approaches are powerful but often complex and costly in this paper we propose new feature selection methods based on two criteria designed for the optimization of svm kernel target alignment and kernel class separability we demonstrate how these two measures when fully expressed can build efficient and simple methods easily applicable to multiclass problems and iteratively computable with minimal memory requirements an extensive experimental study is conducted both on artificial and real world datasets to compare the proposed methods to state of the art feature selection algorithms the results demonstrate the relevance of the proposed methods both in terms of performance and computational cost

computing the local nonglobal minimizer of a large scale trust region subproblem we propose two algorithms for computing the local nonglobal minimizer of a quadratic function subject to a constraint set that is a euclidean sphere we also discuss the case where the constraint set is a euclidean ball at each iteration of the algorithms we compute the two smallest eigenvalues of a parametric matrix using an arpack subroutine only matrix vector multiplications are required hence we are able to exploit the possible sparsity of the hessian matrix of the quadratic objective making the algorithms suitable for large problems this improves previous approaches based on matrix factorizations we also give a geometric relationship based on extremal ellipsoids between the global and the local nonglobal minimizers of the quadratic function under the given sphere constraint

coderivatives of a karush kuhn tucker point set map and applications abstract the problem of minimizing a linear quadratic function over the euclidean ball is encountered frequently in the theory of trust region methods in nonlinear programming by some tools from variational analysis we investigate the stability of the karush kuhn tucker point set map of that problem with respect to total perturbations of its data verifiable sufficient conditions for the local lipschitz like property of the map are obtained and the connection of our results with the existing criteria for the lower semicontinuity of this karush kuhn tucker point set map is shown

simultaneous grouping pursuit and feature selection over an undirected graph in high dimensional regression grouping pursuit and feature selection have their own merits while complementing each other in battling the curse of dimensionality to seek a parsimonious model we perform simultaneous grouping pursuit and feature selection over an arbitrary undirected graph with each node corresponding to one predictor when the corresponding nodes are reachable from each other over the graph regression coefficients can be grouped whose absolute values are the same or close this is motivated from gene network analysis where genes tend to work in groups according to their biological functionalities through a nonconvex penalty we develop a computational strategy and analyze the proposed method theoretical analysis indicates that the proposed method reconstructs the oracle estimator that is the unbiased least square estimator given the true grouping leading to consistent reconstruction of grouping structures and informative features as well as to optimal parameter estimation sim

combining convex concave decompositions and linearization approaches for solving bmis with application to static output feedback a novel optimization method is proposed to minimize a convex function subject to bilinear matrix inequality bmi constraints the key idea is to decompose the bilinear mapping as a difference between two positive semidefinite convex mappings at each iteration of the algorithm the concave part is linearized leading to a convex subproblem applications to various output feedback controller synthesis problems are presented in these applications the subproblem in each iteration step can be turned into a convex optimization problem with linear matrix inequality lmi constraints the performance of the algorithm has been benchmarked on the data from the compleib library

combination between global and local methods for solving an optimization problem over the efficient set abstract in this paper we propose an algorithm for globally solving optimization problems over efficient sets the algorithm is established based on a branch and bound scheme in which the bounding procedure is performed by using the well known weak duality theorem in lagrange duality a suitable combination of this algorithm with a local search procedure in d c optimization named dca leads to a promising global algorithm whose efficiency is more or less confirmed by computational experiments on a large set of test problems

block clustering based on difference of convex functions dc programming and dc algorithms we investigate difference of convex functions dc programming and the dc algorithm dca to solve the block clustering problem in the continuous framework which traditionally requires solving a hard combinatorial optimization problem dc reformulation techniques and exact penalty in dc programming are developed to build an appropriate equivalent dc program of the block clustering problem they lead to an elegant and explicit dca scheme for the resulting dc program computational experiments show the robustness and efficiency of the proposed algorithm and its superiority over standard algorithms such as two mode k means two mode fuzzy clustering and block classification em

collusive game solutions via optimization a nash based collusive game among a finite set of players is one in which the players coordinate in order for each to gain higher payoffs than those prescribed by the nash equilibrium solution in this paper we study the optimization problem of such a collusive game in which the players collectively maximize the nash bargaining objective subject to a set of incentive compatibility constraints we present a smooth reformulation of this optimization problem in terms of a nonlinear complementarity problem we establish the convexity of the optimization problem in the case where each player s strategy set is unidimensional in the multivariate case we propose upper and lower bounding procedures for the collusive optimization problem and establish convergence properties of these procedures computational results with these procedures for solving some test problems are reported

on the solution of the tikhonov regularization of the total least squares problem total least squares tls is a method for treating an overdetermined system of linear equations bf a bf x approx bf b where both the matrix bf a and the vector bf b are contaminated by noise tikhonov regularization of the tls trtls leads to an optimization problem of minimizing the sum of fractional quadratic and quadratic functions as such the problem is nonconvex we show how to reduce the problem to a single variable minimization of a function mathcal g over a closed interval computing a value and a derivative of mathcal g consists of solving a single trust region subproblem for the special case of regularization with a squared euclidean norm we show that mathcal g is unimodal and provide an alternative algorithm which requires only one spectral decomposition a numerical example is given to illustrate the effectiveness of our method

a constrained eigenvalue problem in this paper we consider the following mathematical and computational problem given the quantities a n m by n m matrix symmetric n 0 n n m by m matrix with full rank t vector of dimension m with nt t 1 determine an x such that operatorname x t ax min subject to the constraints n t x t i x t x 1 ii variants of this problem occur in many applications 1 5 7 8 11 the problem has been studied previously when t 0 the null vector cf 4 6 when t 0 then the problem becomes more complicated

optimization based dc programming and dca for hierarchical clustering one of the most promising approaches for clustering is based on methods of mathematical programming in this paper we propose new optimization methods based on dc difference of convex functions programming for hierarchical clustering a bilevel hierarchical clustering model is considered with different optimization formulations they are all nonconvex nonsmooth optimization problems for which we investigate attractive dc optimization algorithms called dca numerical results on some artificial and real world databases are reported the results demonstrate that the proposed algorithms are more efficient than related existing methods

accelerating the dc algorithm for smooth functions we introduce two new algorithms to minimise smooth difference of convex dc functions that accelerate the convergence of the classical dc algorithm dca we prove that the point computed by dca can be used to define a descent direction for the objective function evaluated at this point our algorithms are based on a combination of dca together with a line search step that uses this descent direction convergence of the algorithms is proved and the rate of convergence is analysed under the ojasiewicz property of the objective function we apply our algorithms to a class of smooth dc programs arising in the study of biochemical reaction networks where the objective function is real analytic and thus satisfies the ojasiewicz property numerical tests on various biochemical models clearly show that our algorithms outperform dca being on average more than four times faster in both computational time and the number of iterations numerical experiments show that the algorithms are globally convergent to a non equilibrium steady state of various biochemical networks with only chemically consistent restrictions on the network topology

sparse covariance matrix estimation by dca based algorithms this letter proposes a novel approach using the formula see text norm regularization for the sparse covariance matrix estimation scme problem the objective function of scme problem is composed of a nonconvex part and the formula see text term which is discontinuous and difficult to tackle appropriate dc difference of convex functions approximations of formula see text norm are used that result in approximation scme problems that are still nonconvex dc programming and dca dc algorithm powerful tools in nonconvex programming framework are investigated two dc formulations are proposed and corresponding dca schemes developed two applications of the scme problem that are considered are classification via sparse quadratic discriminant analysis and portfolio optimization a careful empirical experiment is performed through simulated and real data sets to study the performance of the proposed algorithms numerical results showed their efficiency and their superiority compared with seven state of the art methods

a new decomposition method for multiuser dc programming and its applications we propose a novel decomposition framework for the distributed optimization of difference convex dc type nonseparable sum utility functions subject to coupling convex constraints a major contribution of the paper is to develop for the first time a class of inexact best response like algorithms with provable convergence where a suitably convexified version of the original dc program is iteratively solved the main feature of the proposed successive convex approximation method is its decomposability structure across the users which leads naturally to distributed algorithms in the primal and or dual domain the proposed framework is applicable to a variety of multiuser dc problems in different areas ranging from signal processing to communications and networking as a case study in the second part of the paper we focus on two examples namely i a novel resource allocation problem in the emerging area of cooperative physical layer security and ii and the renowned sum rate maximization of mimo cognitive radio networks our contribution in this context is to devise a class of easy to implement distributed algorithms with provable convergence to stationary solution of such problems numerical results show that the proposed distributed schemes reach performance close to and sometimes better than that of centralized methods

decomposition by partial linearization parallel optimization of multi agent systems we propose a novel decomposition framework for the distributed optimization of general nonconvex sum utility functions arising naturally in the system design of wireless multi user interfering systems our main contributions are i the development of the first class of inexact jacobi best response algorithms with provable convergence where all the users simultaneously and iteratively solve a suitably convexified version of the original sum utility optimization problem ii the derivation of a general dynamic pricing mechanism that provides a unified view of existing pricing schemes that are based instead on heuristics and iii a framework that can be easily particularized to well known applications giving rise to very efficient practical jacobi or gauss seidel algorithms that outperform existing ad hoc methods proposed for very specific problems interestingly our framework contains as special cases well known gradient algorithms for nonconvex sum utility problems and many block coordinate descent schemes for convex functions

nonconvex regularizations for feature selection in ranking with sparse svm feature selection in learning to rank has recently emerged as a crucial issue whereas several preprocessing approaches have been proposed only a few have focused on integrating feature selection into the learning process in this paper we propose a general framework for feature selection in learning to rank using support vector machines with a sparse regularization term we investigate both classical convex regularizations such as ell 1 or weighted ell 1 and nonconvex regularization terms such as log penalty minimax concave penalty or ell p pseudo norm with p two algorithms are proposed the first an accelerated proximal approach for solving the convex problems and the second a reweighted ell 1 scheme to address nonconvex regularizations we conduct intensive experiments on nine datasets from letor 3 0 and letor 4 0 corpora numerical results show that the use of nonconvex regularizations we propose leads to more sparsity in the resulting models while preserving the prediction performance the number of features is decreased by up to a factor of 6 compared to the ell 1 regularization in addition the software is publicly available on the web 1 available at http remi flamary com soft soft ranksvm nc html

combining dc algorithms dcas and decomposition techniques for the training of nonpositive semidefinite kernels today decomposition methods are one of the most popular methods for training support vector machines svms with the use of kernels that do not satisfy mercer s condition new techniques must be designed to handle nonpositive semidefinite kernels resulting to this choice in this work we incorporate difference of convex dc functions optimization techniques into decomposition methods to tackle this difficulty the new approach needs no problem modification and we show that the only use of a truncated dc algorithms dcas in the decomposition scheme produces a sufficient decrease of the objective function at each iteration thanks to this property an asymptotic convergence proof of the new algorithm is produced without any blockwise convexity assumption on the objective function we also investigate a working set selection rule using second order information for sequential minimal optimization smo type decomposition in the spirit of dc optimization numerical results show the robustness and the efficiency of the new methods compared with state of the art software

a homotopy method applied to elastica problems abstract the inverse problem in nonlinear incompressible elastica theory where the end positions and inclinations rather than the forces and moment are specified is considered based on the globally convergent chow yorke algorithm a new homotopy method which is simple accurate stable and efficient is developed for comparison numerical results of some other simple approaches e g newton s method based on shooting or finite differences standard embedding are presented the new homotopy method does not require a good initial estimate and is guaranteed to have no singular points the homotopy method is applied to the problem of a circular elastica ring subjected to n symmetrical point loads and numerical results are given for n 2 3 4