adjoint based predictor corrector sequential convex programming for parametric nonlinear optimization this paper proposes an algorithmic framework for solving parametric optimization problems which we call adjoint based predictor corrector sequential convex programming after presenting the algorit

aspects of total variation regularized l1 function approximation the total variation based image denoising model of rudin osher and fatemi phys d 60 1992 pp 259 268 has been generalized and modified in many ways in the literature one of these modifications is to use the l1 norm as the fidelity term we study the interesting consequences of this modification especially from the point of view of geometric properties of its solutions it turns out to have interesting new implications for data driven scale selection and multiscale image decomposition

on identification of active constraints ii the nonconvex case in this paper the results of burke and more 5 on the identification of active constraints are extended to the nonconvex constrained nonlinear programming problem the approach is motivated by the geometric structure of a certain polyhedral convex linearization of the constraint region at each iteration as in burke and more 5 questions of constraint identification are couched in terms of the faces of these polyhedra the main result employs a nondegeneracy condition due to dunn 7 and the linear independence condition to obtain a characterization of those algorithms that identify the optimal active constraints in a finite number of iterations the role of the linear independence condition is carefully examined and it is argued that it is required within the context of a first order theory of constraint identification in conclusion the characterization theorem is applied to the wilson han powell sequential quadratic programming algorithm j optim theory appl 22 1977 pp 297 309 proceedin

hierarchical multi label classification using fully associative ensemble learning developing a local hierarchical ensemble framework for hierarchical multi label classification hmc in which all the structural relationships in the class hierarchy are used to obtain global prediction introducing empirical loss minimization into hmc so that the learned model can capture the most useful information from historical data proposing sparse kernel and binary constraint hmc models traditional flat classification methods e g binary or multi class classification neglect the structural information between different classes in contrast hierarchical multi label classification hmc considers the structural information embedded in the class hierarchy and uses it to improve classification performance in this paper we propose a local hierarchical ensemble framework for hmc fully associative ensemble learning fael we model the relationship between each class nodes global prediction and the local predictions of all the class nodes as a multi variable regression problem with frobenius norm or l1 norm regularization it can be extended using the kernel trick which explores the complex correlation between global and local prediction in addition we introduce a binary constraint model to restrict the optimal weight matrix learning the proposed models have been applied to image annotation and gene function prediction datasets with tree structured class hierarchy and large scale visual recognition dataset with direct acyclic graph dag structured class hierarchy the experimental results indicate that our models achieve better performance when compared with other baseline methods

connected component model for multi object tracking in multi object tracking it is critical to explore the data associations by exploiting the temporal information from a sequence of frames rather than the information from the adjacent two frames since straightforwardly obtaining data associations from multi frames is an np hard multi dimensional assignment mda problem most existing methods solve this mda problem by either developing complicated approximate algorithms or simplifying mda as a 2d assignment problem based upon the information extracted only from adjacent frames in this paper we show that the relation between associations of two observations is the equivalence relation in the data association problem based on the spatial temporal constraint that the trajectories of different objects must be disjoint therefore the mda problem can be equivalently divided into independent subproblems by equivalence partitioning in contrast to existing works for solving the mda problem we develop a connected component model ccm by exploiting the constraints of the data association and the equivalence relation on the constraints based upon ccm we can efficiently obtain the global solution of the mda problem for multi object tracking by optimizing a sequence of independent data association subproblems experiments on challenging public data sets demonstrate that our algorithm outperforms the state of the art approaches

bundle level type methods uniformly optimal for smooth and nonsmooth convex optimization the main goal of this paper is to develop uniformly optimal first order methods for convex programming cp by uniform optimality we mean that the first order methods themselves do not require the input of any problem parameters but can still achieve the best possible iteration complexity bounds by incorporating a multi step acceleration scheme into the well known bundle level method we develop an accelerated bundle level method and show that it can achieve the optimal complexity for solving a general class of black box cp problems without requiring the input of any smoothness information such as whether the problem is smooth nonsmooth or weakly smooth as well as the specific values of lipschitz constant and smoothness level we then develop a more practical restricted memory version of this method namely the accelerated prox level apl method we investigate the generalization of the apl method for solving certain composite cp problems and an important class of saddle point problems recently studied by nesterov math program 103 127 152 2005 we present promising numerical results for these new bundle level methods applied to solve certain classes of semidefinite programming and stochastic programming problems

matrix reordering using multilevel graph coarsening for ilu preconditioning incomplete lu factorization ilu techniques are a well known class of preconditioners often used in conjunction with krylov accelerators for the iterative solution of linear systems of equations however for certain problems ilu factorizations can yield factors that are unstable and in some cases quite dense reordering techniques based on permuting the matrix prior to performing the factorization have been shown to improve the quality of the factorization and the resulting preconditioner in this paper we examine the effect of reordering techniques based on multilevel graph coarsening ideas on one level ilu factorizations such as the level based ilu k or the dual threshold ilut algorithms we consider an aggregation based coarsening idea that implements two main coarsening frameworks a top down approach and a bottom up approach each utilizing one of two different strategies to select the next level coarse graph numerical results are presented to support our findings

tripled fuzzy metric spaces and fixed point theorem abstract one of the most important topics of research in fuzzy sets is to get an appropriate notion of fuzzy metric space fms in the paper we propose a new fms tripled fuzzy metric space tfms which is a new generalization of george and veeramani s fms then we present some related examples topological properties convergence of sequences cauchy sequence cs and completeness of the tfms moreover we introduce two kinds of notions of generalized fuzzy contractive f c mappings and derive a fixed point theorem fpt on the mappings in the space

an optimal method for stochastic composite optimization this paper considers an important class of convex programming cp problems namely the stochastic composite optimization sco whose objective function is given by the summation of general nonsmooth and smooth stochastic components since sco covers non smooth smooth and stochastic cp as certain special cases a valid lower bound on the rate of convergence for solving these problems is known from the classic complexity theory of convex programming note however that the optimization algorithms that can achieve this lower bound had never been developed in this paper we show that the simple mirror descent stochastic approximation method exhibits the best known rate of convergence for solving these problems our major contribution is to introduce the accelerated stochastic approximation ac sa algorithm based on nesterov s optimal method for smooth cp nesterov in doklady an sssr 269 543 547 1983 nesterov in math program 103 127 152 2005 and show that the ac sa algorithm can achieve the aforementioned lower bound on the rate of convergence for sco to the best of our knowledge it is also the first universally optimal algorithm in the literature for solving non smooth smooth and stochastic cp problems we illustrate the significant advantages of the ac sa algorithm over existing methods in the context of solving a special but broad class of stochastic programming problems

multilabel classification with principal label space transformation we consider a hypercube view to perceive the label space of multilabel classification problems geometrically the view allows us not only to unify many existing multilabel classification approaches but also design a novel algorithm principal label space transformation plst that captures key correlations between labels before learning the simple and efficient plst relies on only singular value decomposition as the key step we derive the theoretical guarantee of plst and evaluate its empirical performance using real world data sets experimental results demonstrate that plst is faster than the traditional binary relevance approach and is superior to the modern compressive sensing approach in terms of both accuracy and efficiency

local rademacher complexity for multi label learning we analyze the local rademacher complexity of empirical risk minimization based multi label learning algorithms and in doing so propose a new algorithm for multi label learning rather than using the trace norm to regularize the multi label predictor we instead minimize the tail sum of the singular values of the predictor in multi label learning benefiting from the use of the local rademacher complexity our algorithm therefore has a sharper generalization error bound compared with methods that minimize over all singular values concentrating on the tail singular values results in better recovery of the low rank structure of the multi label predictor which plays an important role in exploiting label correlations we propose a new conditional singular value thresholding algorithm to solve the resulting objective function moreover a variance control strategy is employed to reduce the variance of variables in optimization empirical studies on real world data sets validate our theoretical results and demonstrate the effectiveness of the proposed algorithm for multi label learning

lean algebraic multigrid lamg fast graph laplacian linear solver laplacian matrices of graphs arise in large scale computational applications such as semisupervised machine learning spectral clustering of images genetic data and web pages transportation network flows electrical resistor circuits and elliptic partial differential equations discretized on unstructured grids with finite elements a lean algebraic multigrid lamg solver of the symmetric linear system ax b is presented where a is a graph laplacian lamg s run time and storage are empirically demonstrated to scale linearly with the number of edges lamg consists of a setup phase during which a sequence of increasingly coarser laplacian systems is constructed and an iterative solve phase using multigrid cycles general graphs pose algorithmic challenges not encountered in traditional multigrid applications lamg combines a lean piecewise constant interpolation judicious node aggregation based on a new node proximity measure the affinity and an energy correction of coarse level systems this

accelerated gradient methods for nonconvex nonlinear and stochastic programming in this paper we generalize the well known nesterov s accelerated gradient ag method originally designed for convex smooth optimization to solve nonconvex and possibly stochastic optimization problems we demonstrate that by properly specifying the stepsize policy the ag method exhibits the best known rate of convergence for solving general nonconvex smooth optimization problems by using first order information similarly to the gradient descent method we then consider an important class of composite optimization problems and show that the ag method can solve them uniformly i e by using the same aggressive stepsize policy as in the convex case even if the problem turns out to be nonconvex we demonstrate that the ag method exhibits an optimal rate of convergence if the composite problem is convex and improves the best known rate of convergence if the problem is nonconvex based on the ag method we also present new nonconvex stochastic approximation methods and show that they can improve a few existing rates of convergence for nonconvex stochastic optimization to the best of our knowledge this is the first time that the convergence of the ag method has been established for solving nonconvex nonlinear programming in the literature

adaptive smoothed aggregation alpha sa substantial effort has been focused over the last two decades on developing multilevel iterative methods capable of solving the large linear systems encountered in engineering practice these systems often arise from discretizing partial differential equations over unstructured meshes and the particular parameters or geometry of the physical problem being discretized may be unavailable to the solver algebraic multigrid amg and multilevel domain decomposition methods of algebraic type have been of particular interest in this context because of their promises of optimal performance without the need for explicit knowledge of the problem geometry these methods construct a hierarchy of coarse problems based on the linear system itself and on certain assumptions about the smooth components of the error for smoothed aggregation sa methods applied to discretizations of elliptic problems these assumptions typically consist of knowledge of the near nullspace of the weak form this paper introduces an extension of the sa method in which good convergence properties are achieved in situations where explicit knowledge of the near nullspace components is unavailable this extension is accomplished by using the method itself to determine near nullspace components and adjusting the coarsening processes accordingly

multigrid methods for convection diffusion problems discretized by a monotone scheme abstract we study multigrid mg methods for the solution of systems of linear algebraic equations obtained from a stable discretization of convection diffusion problems by an exponential fitting scheme the latter ensures the stability of the simplest possible coarse grid operators obtained from galerkin projections based on graph matching linear and nonlinear mg preconditioners are defined in the framework of algebraic multilevel iteration the option of using polynomial smoothers is investigated in context of nonsymmetric problems and a systematic performance comparison is presented for various algorithms on a representative set of two and three dimensional test problems

a fast optimization method for general binary code learning hashing or binary code learning has been recognized to accomplish efficient near neighbor search and has thus attracted broad interests in recent retrieval vision and learning studies one main challenge of learning to hash arises from the involvement of discrete variables in binary code optimization while the widely used continuous relaxation may achieve high learning efficiency the pursued codes are typically less effective due to accumulated quantization error in this paper we propose a novel binary code optimization method dubbed discrete proximal linearized minimization dplm which directly handles the discrete constraints during the learning process specifically the discrete thus nonsmooth nonconvex problem is reformulated as minimizing the sum of a smooth loss term with a nonsmooth indicator function the obtained problem is then efficiently solved by an iterative procedure with each iteration admitting an analytical discrete solution which is thus shown to converge very fast in addition the proposed method supports a large family of empirical loss functions which is particularly instantiated in this paper by both a supervised and an unsupervised hashing losses together with the bits uncorrelation and balance constraints in particular the proposed dplm with a supervised ell 2 loss encodes the whole nus wide database into 64 b binary codes within 10 s on a standard desktop computer the proposed approach is extensively evaluated on several large scale data sets and the generated binary codes are shown to achieve very promising results on both retrieval and classification tasks

accelerated block coordinate relaxation for regularized optimization we discuss minimization of a smooth function to which is added a separable regularization function that induces structure in the solution a block coordinate relaxation approach with proximal linearized subproblems yields convergence to critical points while identification of the optimal manifold under a nondegeneracy condition allows acceleration techniques to be applied on a reduced space the work is motivated by experience with an algorithm for regularized logistic regression and computational results for the algorithm on problems of this type are presented

on the consistency of multi label learning multi label learning has attracted much attention during the past few years many multi label approaches have been developed mostly working with surrogate loss functions because multi label loss functions are usually difficult to optimize directly owing to their non convexity and discontinuity these approaches are effective empirically however little effort has been devoted to the understanding of their consistency i e the convergence of the risk of learned functions to the bayes risk in this paper we present a theoretical analysis on this important issue we first prove a necessary and sufficient condition for the consistency of multi label learning based on surrogate loss functions then we study the consistency of two well known multi label loss functions i e ranking loss and hamming loss for ranking loss our results disclose that surprisingly none of convex surrogate loss is consistent we present the partial ranking loss with which some surrogate losses are proven to be consistent we also discuss on the consistency of univariate surrogate losses for hamming loss we show that two multi label learning methods i e one vs all and pairwise comparison which can be regarded as direct extensions from multi class learning are inconsistent in general cases yet consistent under the dominating setting and similar results also hold for some recent multi label approaches that are variations of one vs all in addition we discuss on the consistency of learning approaches that address multi label learning by decomposing into a set of binary classification problems

spectral correlation functions of the sum of two independent complex wishart matrices with unequal covariances we compute the spectral statistics of the sum h of two independent complex wishart matrices each of which is correlated with a different covariance matrix random matrix theory enjoys many applications including sums and products of random matrices typically ensembles with correlations among the matrix elements are much more difficult to solve using a combination of supersymmetry superbosonisation and bi orthogonal functions we are able to determine all spectral k point density correlation functions of h for arbitrary matrix size n in the half degenerate case when one of the covariance matrices is proportional to the identity the recent results by kumar for the joint eigenvalue distribution of h serve as our starting point in this case the ensemble has a bi orthogonal structure and we explicitly determine its kernel providing its exact solution for finite n the kernel follows from computing the expectation value of a single characteristic polynomial in the general non degenerate case the generating function for the k point resolvent is determined from a supersymmetric evaluation of the expectation value of k ratios of characteristic polynomials numerical simulations illustrate our findings for the spectral density at finite n and we also give indications how to do the asymptotic large n analysis

composite proximal bundle method we consider minimization of nonsmooth functions which can be represented as the composition of a positively homogeneous convex function and a smooth mapping this is a sufficiently rich class that includes max functions largest eigenvalue functions and norm 1 regularized functions the bundle method uses an oracle that is able to compute separately the function and subgradient information for the convex function and the function and derivatives for the smooth mapping with this information it is possible to solve approximately certain proximal linearized subproblems in which the smooth mapping is replaced by its taylor series linearization around the current serious step our numerical results show the good performance of the composite bundle method for a large class of problems

unstructured geometric multigrid in two and three dimensions on complex and graded meshes the use of multigrid and related preconditioners with the finite element method is often limited by the difficulty of applying the algorithm effectively to a problem especially when the domain has a complex shape or the mesh has adaptive refinement we introduce a simplification of a general topologically motivated mesh coarsening algorithm for use in creating hierarchies of meshes for geometric unstructured multigrid methods the connections between the guarantees of this technique and the quality criteria necessary for multigrid methods for nonquasi uniform problems are noted the implementation details in particular those related to coarsening remeshing and interpolation are discussed computational tests on pathological test cases from adaptive finite element methods show the performance of the technique

a algorithm for convex minimization for convex minimization we introduce an algorithm based on space decomposition the method uses a bundle subroutine to generate a sequence of approximate proximal points when a primal dual track leading to a solution and zero subgradient pair exists these points approximate the primal track points and give the algorithm s or corrector steps the subroutine also approximates dual track points that are gradients needed for the method s newton predictor steps with the inclusion of a simple line search the resulting algorithm is proved to be globally convergent the convergence is superlinear if the primal dual track points and the objective s hessian are approximated well enough

active sets nonsmoothness and sensitivity nonsmoothness pervades optimization but the way it typically arises is highly structured nonsmooth behavior of an objective function is usually associated locally with an active manifold on this manifold the function is smooth whereas in normal directions it is vee shaped active set ideas in optimization depend heavily on this structure important examples of such functions include the pointwise maximum of some smooth functions and the maximum eigenvalue of a parametrized symmetric matrix among possible foundations for practical nonsmooth optimization this broad class of partly smooth functions seems a promising candidate enjoying a powerful calculus and sensitivity theory in particular we show under a natural regularity condition that critical points of partly smooth functions are stable small perturbations to the function cause small movements of the critical point on the active manifold

a proximal method for composite minimization we consider minimization of functions that are compositions of convex or prox regular functions possibly extended valued with smooth vector functions a wide variety of important optimization problems fall into this framework we describe an algorithmic framework based on a subproblem constructed from a linearized approximation to the objective and a regularization term properties of local solutions of this subproblem underlie both a global convergence result and an identification property of the active manifold containing the solution of the original problem preliminary computational results on both convex and nonconvex examples are promising

reduced rank regression via adaptive nuclear norm penalization we propose an adaptive nuclear norm penalization approach for low rank matrix approximation and use it to develop a new reduced rank estimation method for high dimensional multivariate regression the adaptive nuclear norm is defined as the weighted sum of the singular values of the matrix and it is generally non convex under the natural restriction that the weight decreases with the singular value however we show that the proposed non convex penalized regression method has a global optimal solution obtained from an adaptively soft thresholded singular value decomposition the method is computationally efficient and the resulting solution path is continuous the rank consistency of and prediction estimation performance bounds for the estimator are established for a high dimensional asymptotic regime simulation studies and an application in genetics demonstrate its efficacy

local strong homogeneity of a regularized estimator this paper deals with regularized pointwise estimation of discrete signals which contain large strongly homogeneous zones where typically they are constant or linear or more generally satisfy a linear equation the estimate is defined as the minimizer of an objective function combining a quadratic data fidelity term and a regularization prior term the latter term is the sum of the values obtained by applying a potential function pf to each component called a difference of a linear transform of the signal minimizers of functions of this form arise in various settings in statistics and optimization the features exhibited by such an estimate are closely related to the shape of the pf our goal is to determine estimators providing solutions which involve large strongly homogeneous zones where more precisely the differences are null in spite of the noise corrupting the data to this end we require that the strongly homogeneous zones recovered by the estimator be insensitive to any variation of th

bilinear optimized product quantization for scalable visual content analysis product quantization pq has been recognized as a useful technique to encode visual feature vectors into compact codes to reduce both the storage and computation cost recent advances in retrieval and vision tasks indicate that high dimensional descriptors are critical to ensuring high accuracy on large scale data sets however optimizing pq codes with high dimensional data is extremely time consuming and memory consuming to solve this problem in this paper we present a novel pq method based on bilinear projection which can well exploit the natural data structure and reduce the computational complexity specifically we learn a global bilinear projection for pq where we provide both non parametric and parametric solutions the non parametric solution does not need any data distribution assumption the parametric solution can avoid the problem of local optima caused by random initialization and enjoys a theoretical error bound besides we further extend this approach by learning locally bilinear projections to fit underlying data distributions we show by extensive experiments that our proposed method dubbed bilinear optimization product quantization achieves competitive retrieval and classification accuracies while having significant lower time and space complexities

identifiable surfaces in constrained optimization the concept of a class c p identifiable surface of a convex set in euclidean space is introduced the paper shows how the smoothness of these surfaces is related to the smoothness of the projection operator and presents finite identification results for certain algorithms for minimization of a function over this set the work uses a partially geometric view of constrained optimization to generalize previous finite identification results

parallel multilevel series k way partitioning scheme for irregular graphs in this paper we present a parallel formulation of a multilevel k way graph partitioning algorithm a key feature of this parallel formulation is that it is able to achieve a high degree of concurrency while maintaining the high quality of the partitions produced by the serial multilevel k way partitioning algorithm in particular the time taken by our parallel graph partitioning algorithm is only slightly longer than the time taken for re arrangement of the graph among processors according to the new partition experiments with a variety of finite element graphs show that our parallel formulation produces high quality partitionings in a short amount of time for example a 128 way partitioning of graphs with one million vertices can be computed in a little over two seconds on a 128 processor cray t3d furthermore the quality of the partitions produced is comparable edge cuts within 5 to those produced by the serial multilevel k way algorithm thus our parallel algorithm makes it feasible to perform frequent repartitioning of graphs in dynamic computations without compromising the partitioning quality

algebraic multilevel preconditioners for the graph laplacian based on matching in graphs this paper presents estimates of the convergence rate and complexity of an algebraic multilevel preconditioner based on piecewise constant coarse vector spaces applied to the graph laplacian a bou

active set newton methods and partial smoothness diverse optimization algorithms correctly identify in finite time intrinsic constraints that must be active at optimality analogous behavior extends beyond optimization to systems involving part

a simplified view of first order methods for optimization we discuss the foundational role of the proximal framework in the development and analysis of some iconic first order optimization algorithms with a focus on non euclidean proximal distances of bregman type which are central to the analysis of many other fundamental first order minimization relatives we stress simplification and unification by highlighting self contained elementary proof patterns to obtain convergence rate and global convergence both in the convex and the nonconvex settings which in turn also allows to present some novel results

algebraic multigrid for moderate order finite elements we investigate the use of algebraic multigrid amg methods for the solution of large sparse linear systems arising from the discretization of scalar elliptic partial differential equations with lagrangian finite elements of order at most 4 the resulting system matrices do not have the m matrix property that is required by standard analyses of classical amg and aggregation based amg methods a unified approach is presented that allows us to extend these analyses it uses an intermediate m matrix and highlights the role of the spectral equivalence constant that relates this matrix to the original system matrix this constant is shown to be bounded independently of the problem size and jumps in the coefficients of the partial differential equations provided that jumps are located at elements boundaries for two dimensional problems it is further shown to be uniformly bounded if the angles in the triangulation also satisfy a uniform bound this analysis validates the application of the amg methods to the

fixed point continuation for ell_1 minimization methodology and convergence we present a framework for solving the large scale ell 1 regularized convex minimization problem min x 1 mu f x our approach is based on two powerful algorithmic ideas operator splitting and continuation operator splitting results in a fixed point algorithm for any given scalar mu continuation refers to approximately following the path traced by the optimal value of x as mu increases in this paper we study the structure of optimal solution sets prove finite convergence for important quantities and establish q linear convergence rates for the fixed point algorithm applied to problems with f x convex but not necessarily strictly convex the continuation framework motivated by our convergence results is demonstrated to facilitate the construction of practical algorithms

an adaptive fixed point proximity algorithm for solving total variation denoising models abstract we study an adaptive fixed point proximity algorithm to solve the total variation denoising model the objective function is a sum of two convex functions and one of them is composed by an affine transformation which is usually a regularization term by decoupling and splitting the problem is changed into two subproblems due to the nonsmooth and nondifferentiability of the subproblem we solve its proximity minimization problem instead of the original one to overcome the staircase effect during the process of denoising an adaptive criterion on proximity parameters is put forward at last we apply the improved algorithm to solve the isotropic total variation denoising model the numerical results are given to illustrate the efficiency of the algorithm

multilevel adaptive aggregation for markov chains with application to web ranking a multilevel adaptive aggregation method for calculating the stationary probability vector of an irreducible stochastic matrix is described the method is a special case of the adaptive smoothed aggregation and adaptive algebraic multigrid methods for sparse linear systems and is also closely related to certain extensively studied iterative aggregation disaggregation methods for markov chains in contrast to most existing approaches our aggregation process does not employ any explicit advance knowledge of the topology of the markov chain instead adaptive agglomeration is proposed that is based on the strength of connection in a scaled problem matrix in which the columns of the original problem matrix at each recursive fine level are scaled with the current probability vector iterate at that level the strength of connection is determined as in the algebraic multigrid method and the aggregation process is fully adaptive with optimized aggregates chosen in each step of the iteration and at all recursive levels the multilevel method is applied to a set of stochastic matrices that provide models for web page ranking numerical tests serve to illustrate for which types of stochastic matrices the multilevel adaptive method may provide significant speedup compared to standard iterative methods the tests also provide more insight into why google s pagerank model is a successful model for determining a ranking of web pages

ica based on asymmetry we build a new approach to ica which is based on the data asymmetry instead of densities with heavy tails we use asymmetric ones split gaussians we verified our approach on images sound and eeg data in the case of source signal reconstructing our approach gives better results independent component analysis ica one of the basic tools in data analysis aims to find a coordinate system in which the components of the data are independent most of existing methods are based on the minimization of the function of fourth order moment kurtosis skewness third order moment has received much less attention in this paper we present a competitive approach to ica based on the split gaussian distribution which is well adapted to asymmetric data consequently we obtain a method which works better than the classical approaches especially in the case when the underlying density is not symmetric which is a typical situation in the color distribution in images

a fast algorithm for sparse reconstruction based on shrinkage subspace optimization and continuation we propose a fast algorithm for solving the ell 1 regularized minimization problem min x in mathbb r n mu x 1 ax b 2 2 for recovering sparse solutions to an undetermined system of linear equations ax b the algorithm is divided into two stages that are performed repeatedly in the first stage a first order iterative shrinkage method yields an estimate of the subset of components of x likely to be nonzero in an optimal solution restricting the decision variables x to this subset and fixing their signs at their current values reduces the ell 1 norm x 1 to a linear function of x the resulting subspace problem which involves the minimization of a smaller and smooth quadratic function is solved in the second phase our code fpc as embeds this basic two stage algorithm in a continuation homotopy approach by assigning a decreasing sequence of values to mu this code exhibits state of the art performance in terms of both its speed and its ability to recover sparse signals

newton methods for nonsmooth convex minimization connections among lagrangian riemannian newton and sqp methods this paper studies newton type methods for minimization of partly smooth convex functions sequential newton methods are provided using local parameterizations obtained from lagrangian theory and from riemannian geometry the hessian based on the lagrangian depends on the selection of a dual parameter g by revealing the connection to riemannian geometry a natural choice of g emerges for which the two newton directions coincide this choice of g is also shown to be related to the least squares multiplier estimate from a sequential quadratic programming sqp approach and with this multiplier sqp gives the same search direction as the newton methods

on a class of nonsmooth composite functions we discuss in this paper a class of nonsmooth functions which can be represented in a neighborhood of a considered point as a composition of a positively homogeneous convex function and a smooth mapping which maps the considered point into the null vector we argue that this is a sufficiently rich class of functions and that such functions have various properties useful for purposes of optimization

nonlinear programming and nonsmooth optimization by successive linear programming methods are considered for solving nonlinear programming problems using an exactl1 penalty function lp like subproblems incorporating a trust region constraint are solved successively both to estimate the active set and to provide a foundation for proving global convergence in one particular method second order information is represented by approximating the reduced hessian matrix and coleman conn steps are taken a criterion for accepting these steps is given which enables the superlinear convergence properties of the coleman conn method to be retained whilst preserving global convergence and avoiding the maratos effect the methods generalize to solve a wide range of composite nonsmooth optimization problems and the theory is presented in this general setting a range of numerical experiments on small test problems is described

supervised matrix factorization hashing for cross modal retrieval the target of cross modal hashing is to embed heterogeneous multimedia data into a common low dimensional hamming space which plays a pivotal part in multimedia retrieval due to the emergence of big multimodal data recently matrix factorization has achieved great success in cross modal hashing however how to effectively use label information and local geometric structure is still a challenging problem for these approaches to address this issue we propose a cross modal hashing method based on collective matrix factorization which considers both the label consistency across different modalities and the local geometric consistency in each modality these two elements are formulated as a graph laplacian term in the objective function leading to a substantial improvement on the discriminative power of latent semantic features obtained by collective matrix factorization moreover the proposed method learns unified hash codes for different modalities of an instance to facilitate cross modal search and the objective function is solved using an iterative strategy the experimental results on two benchmark data sets show the effectiveness of the proposed method and its superiority over state of the art cross modal hashing methods

generic optimality conditions for semialgebraic convex programs we consider linear optimization over a nonempty convex semialgebraic feasible region f semidefinite programming is an example if f is compact then for almost every linear objective there is a unique optimal solution lying on a unique active manifold around which f is partly smooth and the second order sufficient conditions hold perturbing the objective results in smooth variation of the optimal solution the active manifold consists locally of these perturbed optimal solutions it is independent of the representation of f and is eventually identified by a variety of iterative algorithms such as proximal and projected gradient schemes these results extend to unbounded sets f

multi label image categorization with sparse factor representation the goal of multilabel classification is to reveal the underlying label correlations to boost the accuracy of classification tasks most of the existing multilabel classifiers attempt to exhaustively explore dependency between correlated labels it increases the risk of involving unnecessary label dependencies which are detrimental to classification performance actually not all the label correlations are indispensable to multilabel model negligible or fragile label correlations cannot be generalized well to the testing data especially if there exists label correlation discrepancy between training and testing sets to minimize such negative effect in the multilabel model we propose to learn a sparse structure of label dependency the underlying philosophy is that as long as the multilabel dependency cannot be well explained the principle of parsimony should be applied to the modeling process of the label correlations the obtained sparse label dependency structure discards the outlying correlations between labels which makes the learned model more generalizable to future samples experiments on real world data sets show the competitive results compared with existing algorithms

inexact variants of the proximal point algorithm without monotonicity this paper studies convergence properties of inexact variants of the proximal point algorithm when applied to a certain class of nonmonotone mappings the presented algorithms allow for constant relative errors in the line of the recently proposed hybrid proximal extragradient algorithm the main convergence result extends a recent work of the second author where exact solutions for the proximal subproblems were required we also show that the linear convergence property is preserved in the case when the inverse of the operator is locally lipschitz continuous near the origin as an application we give a convergence analysis for an inexact version of the proximal method of multipliers for a rather general family of problems which includes variational inequalities and constrained optimization problems

spectral domain of large nonsymmetric correlated wishart matrices we study complex eigenvalues of the wishart model for nonsymmetric correlation matrices the model is defined for two statistically equivalent but different gaussian real matrices as c ab t t where b t is the transpose of b and both matrices a and b are of dimensions n t if a and b are uncorrelated or equivalently if c vanishes on average it is known that at large matrix dimension the domain of the eigenvalues of c is a circle centered at origin and the eigenvalue density depends only on the radial distances we consider actual correlation in a and b and derive a result for the contour describing the domain of the bulk of the eigenvalues of c in the limit of large n and t where the ratio n t is finite in particular we show that the eigenvalue domain is sensitive to the correlations for example when c is diagonal on average with the same element c 0 the contour is no longer a circle centered at origin but a shifted ellipse in this case we explicitly derive a result for the spectral density which again depends only on the radial distances for more general cases we show that the contour depends on the symmetric and antisymmetric parts of the correlation matrix resulting from the ensemble averaged c if the correlation matrix is normal then the contour depends only on its spectrum we also provide numerics to justify our analytics

conditions for convergence of trust region algorithms for nonsmooth optimization this paper discusses some properties of trust region algorithms for nonsmooth optimization the problem is expressed as the minimization of a functionh f x whereh is convex andf is a continuously differentiable mapping from to bounds for the second order derivative approximation matrices are discussed it is shown that powel s 7 8 results hold for nonsmooth optimization

an infeasible primal dual algorithm for total bounded variation based inf convolution type image restoration in this paper a primal dual algorithm for total bounded variation tv type image restoration is analyzed and tested analytically it turns out that employing a global boldsymbol l s regularization with 1 s leq 2 in the dual problem results in a local smoothing of the tv regularization term in the primal problem the local smoothing can alternatively be obtained as the infimal convolution of the ell r norm with r 1 s 1 1 and a smooth function in the case r s 2 this results in gauss tv type image restoration the globalized primal dual algorithm introduced in this paper works with generalized derivatives converges locally at a superlinear rate and is stable with respect to noise in the data in addition it utilizes a projection technique which reduces the size of the linear system that has to be solved per iteration a comprehensive numerical study ends the paper

rate of convergence analysis of decomposition methods based on the proximal method of multipliers for convex minimization this paper presents two classes of decomposition algorithms based on the proximal method of multipliers pmm introduced in the mid 1970s by rockafellar for convex minimization we first show that

convergence of algebraic multigrid based on smoothed aggregation we prove aconvergence estimate for the algebraic multigrid method with prolongations defined by aggregation using zero energy modes followed by a smoothing the method input is the problem matrix and a matrix of the zero energy modes the estimate depends only polylogarithmically on the mesh size and requires only a weak approximation property for the aggregates which can be a priori verified computationally construction of the prolongator in the case of a general second order system is described and the assumptions of the theorem are verified for a scalar problem discretized by linear conforming finite elements

on convergence rates of linearized proximal algorithms for convex composite optimization with applications in the present paper we investigate a linearized proximal algorithm lpa for solving a convex composite optimization problem each iteration of the lpa is a proximal minimization of the convex composite function with the inner function being linearized at the current iterate the lpa has the attractive computational advantage that the solution of each subproblem is a singleton which avoids the difficulty as in the gauss newton method gnm of finding a solution with minimum norm among the set of minima of its subproblem while still maintaining the same local convergence rate as that of the gnm under the assumptions of local weak sharp minima of order p p in 1 2 and a quasi regularity condition we establish a local superlinear convergence rate for the lpa we also propose a globalization strategy for the lpa based on a backtracking line search and an inexact version of the lpa we further apply the lpa to solve a possibly nonconvex feasibility problem as well as a sensor network localiza

dual diversified dynamical gaussian process latent variable model for video repairing in this paper we propose a dual diversified dynamical gaussian process latent variable model text d 3 gplvm to tackle the video repairing issue for preservation purposes videos have to be conserved on media however storing on media such as films and hard disks can suffer from unexpected data loss for instance physical damage so repairing of missing or damaged pixels is essential for better video maintenance most methods seek to fill in missing holes by synthesizing similar textures from local patches the neighboring pixels consecutive frames or the whole video however these can introduce incorrect contexts especially when the missing hole or number of damaged frames is large furthermore simple texture synthesis can introduce artifacts in undamaged and recovered areas to address aforementioned problems we introduce two diversity encouraging priors to both of inducing points and latent variables for considering the variety in existing videos in text d 3 gplvm the inducing points constitute a smaller subset of observed data while latent variables are a low dimensional representation of observed data since they have a strong correlation with the observed data it is essential that both of them can capture distinct aspects of and fully represent the observed data the dual diversity encouraging priors ensure that the trained inducing points and latent variables are more diverse and resistant for context aware and artifacts free based video repairing the defined objective function in our proposed model is initially not analytically tractable and must be solved by variational inference finally experimental testing results illustrate the robustness and effectiveness of our method for damaged video repairing

proximal methods for cohypomonotone operators conditions are given for the viability and the weak convergence of an inexact relaxed proximal point algorithm for finding a common zero of countably many cohypomonotone operators in a hilbert space in turn new convergence results are obtained for an extended version of the proximal method of multipliers in nonlinear programming

on the convergence of successive linear quadratic programming algorithms the global convergence properties of a class of penalty methods for nonlinear programming are analyzed these methods include successive linear programming approaches and more specifically the successive linear quadratic programming approach presented by byrd et al math program 100 2004 pp 27 48 every iteration requires the solution of two trust region subproblems involving piecewise linear and quadratic models respectively it is shown that for a fixed penalty parameter the sequence of iterates approaches stationarity of the penalty function a procedure for dynamically adjusting the penalty parameter is described and global convergence results for it are established

wavelet algorithms for high resolution image reconstruction high resolution image reconstruction refers to the reconstruction of high resolution images from multiple low resolution shifted degraded samples of a true image in this paper we analyze this problem from the wavelet point of view by expressing the true image as a function in cal l bbb r 2 we derive iterative algorithms which recover the function completely in the cal l sense from the given low resolution functions these algorithms decompose the function obtained from the previous iteration into different frequency components in the wavelet transform domain and add them into the new iterate to improve the approximation we apply wavelet packet thresholding methods to denoise the function obtained in the previous step before adding it into the new iterate our numerical results show that the reconstructed images from our wavelet algorithms are better than that from the tikhonov least squares approach extension to super resolution image reconstruction where some of the low resolution images are missing is also considered

on the superlinear convergence of a trust region algorithm for nonsmooth optimization it is proved that the second order correction trust region algorithm of fletcher 5 ensures superlinear convergence if some mild conditions are satisfied

descent methods for composite nondifferentiable optimization problems we present a framework for the development of globally defined descent algorithms for the minimization of non differentiable objective functionsf h o f withh convex within our structure the global convergence properties of the cauchy modified newton gauss newton and variable metric methods are easily established along with that of several new approaches examples illustrating the calculational techniques are provided

on the identification of active constraints nondegeneracy conditions that guarantee that the optimal active constraints are identified in a finite number of iterations are studied results of this type have only been established for a few algorithms and then under restrictive hypothesis the main result is a characterization of those algorithms that identify the optimal constraints in a finite number of iterations this result is obtained with a nondegeneracy assumption which is equivalent in the standard nonlinear programming problem to the assumption that there is a set of strictly complementary lagrange multipliers as an important consequence of the authors results the way that this characterization applies to gradient projection and sequential quadratic programming algorithms is shown

neighborhood discriminant hashing for large scale image retrieval with the proliferation of large scale community contributed images hashing based approximate nearest neighbor search in huge databases has aroused considerable interest from the fields of computer vision and multimedia in recent years because of its computational and memory efficiency in this paper we propose a novel hashing method named neighborhood discriminant hashing ndh for short to implement approximate similarity search different from the previous work we propose to learn a discriminant hashing function by exploiting local discriminative information i e the labels of a sample can be inherited from the neighbor samples it selects the hashing function is expected to be orthogonal to avoid redundancy in the learned hashing bits as much as possible while an information theoretic regularization is jointly exploited using maximum entropy principle as a consequence the learned hashing function is compact and nonredundant among bits while each bit is highly informative extensive experiments are carried out on four publicly available data sets and the comparison results demonstrate the outperforming performance of the proposed ndh method over state of the art hashing techniques

the smallest eigenvalue distribution in the real wishart laguerre ensemble with even topology we consider rectangular random matrices of size p times n belonging to thereal wishart laguerre ensemble also known as the chiral gaussian orthogonalensemble this ensemble appears in many applications like qcd mesoscopicphysics and time series analysis we are particularly interested in thedistribution of the smallest non zero eigenvalue and the gap probability tofind no eigenvalue in an interval 0 t while for odd topology nu n p explicit closed results are known for finite and infinite matrix size for even nu 2 only recursive expressions in p are available the smallest eigenvaluedistribution as well as the gap probability for general even nu isequivalent to expectation values of characteristic polynomials raised to ahalf integer the computation of such averages is done via a combination ofskew orthogonal polynomials and bosonisation methods the results are given interms of pfaffian determinants both at finite p and in the hard edge scalinglimit p to infty and nu fixed for an arbitrary even topology nu numerical simulations for the correlated wishart ensemble illustrate theuniversality of our results in this particular limit these simulations pointto a validity of the hard edge scaling limit beyond the invariant case

probability estimates for multi class classification by pairwise coupling pairwise coupling is a popular multi class classification method that combines all comparisons for each pair of classes this paper presents two approaches for obtaining class probabilities both methods can be reduced to linear systems and are easy to implement we show conceptually and experimentally that the proposed approaches are more stable than the two existing popular methods voting and the method by hastie and tibshirani 1998

correlated logistic model with elastic net regularization for multilabel image classification in this paper we present correlated logistic corrlog model for multilabel image classification corrlog extends conventional logistic regression model into multilabel cases via explicitly modeling the pairwise correlation between labels in addition we propose to learn the model parameters of corrlog with elastic net regularization which helps exploit the sparsity in feature selection and label correlations and thus further boost the performance of multilabel classification corrlog can be efficiently learned though approximately by regularized maximum pseudo likelihood estimation and it enjoys a satisfying generalization bound that is independent of the number of labels corrlog performs competitively for multilabel image classification on benchmark data sets mulan scene mit outdoor scene pascal voc 2007 and pascal voc 2012 compared with the state of the art multilabel classification algorithms

sce a manifold regularized set covering method for data partitioning cluster analysis plays a very important role in data analysis in these years cluster ensemble as a cluster analysis tool has drawn much attention for its robustness stability and accuracy many efforts have been done to combine different initial clustering results into a single clustering solution with better performance however they neglect the structure information of the raw data in performing the cluster ensemble in this paper we propose a structural cluster ensemble sce algorithm for data partitioning formulated as a set covering problem in particular we construct a laplacian regularized objective function to capture the structure information among clusters moreover considering the importance of the discriminative information underlying in the initial clustering results we add a discriminative constraint into our proposed objective function finally we verify the performance of the sce algorithm on both synthetic and real data sets the experimental results show the effectiveness of our proposed method sce algorithm

a bootstrap algebraic multilevel method for markov chains this work concerns the development of an algebraic multilevel method for computing state vectors of markov chains we present an efficient bootstrap algebraic multigrid amg method for this task in our proposed approach we employ a multilevel eigensolver with interpolation built using ideas based on compatible relaxation algebraic distances and least squares fitting of test vectors our adaptive variational strategy for computation of the state vector of a given markov chain is then a combination of this multilevel eigensolver and an associated additive multilevel preconditioned correction process we show that the bootstrap amg eigensolver by itself can efficiently compute accurate approximations to the steady state vector an additional benefit of the bootstrap approach is that it yields an accurate interpolation operator for many other eigenmodes this in turn allows for the use of the resulting multigrid hierarchy as a preconditioner to accelerate the generalized minimal residual gmres iteration for computing an additive correction equation for the approximation to the steady state vector unlike other existing multilevel methods for markov chains our method does not employ any special processing of the coarse level systems to ensure that stochastic properties of the fine level system are maintained there the proposed approach is applied to a range of test problems involving nonsymmetric m matrices arising from stochastic matrices and showing promising results

an algorithm for the minimization of nonsmooth nonconvex functions using inexact evaluations and its worst case complexity an adaptive regularization algorithm using inexact function and derivatives evaluations is proposed for the solution of composite nonsmooth nonconvex optimization it is shown that this algorithm needs at most o log epsilon epsilon 2 evaluations of the problem s functions and their derivatives for finding an epsilon approximate first order stationary point this complexity bound therefore generalizes that provided by bellavia et al theoretical study of an adaptive cubic regularization method with dynamic inexact hessian information arxiv 1808 06239 2018 for inexact methods for smooth nonconvex problems and is within a factor log epsilon of the optimal bound known for smooth and nonsmooth nonconvex minimization with exact evaluations a practically more restrictive variant of the algorithm with worst case complexity o log epsilon epsilon 2 is also presented

a constrained least squares regression model abstract least squares regression lsr is a widely used regression technique for multicategory classification conventional lsr model assumes that during the learning phase the labeled samples can be exactly transformed into a discrete label matrix which is too strict to learn a regression matrix for fitting the labels to overcome this drawback lots of lsr s variants utilize the soft target label which contains the continuous values to replace this discrete label to improve the learning performance since the regression matrix can be learnt from these soft target labels it is reasonable to assume that the samples in the same class have similar soft target labels nevertheless most of existing lsr based models don t adequately consider this similarity assumption in this paper we propose a constrained least squares regression clsr model for multicategory classification the main motivation of clsr is to force the samples in the same class to obtain the similar soft target labels to effectively optimize clsr we propose a novel alternating algorithm which can converge to the globally optimal solution extensive experiments results on face and digit data sets confirm the effectiveness of our proposed model

a multilevel proximal gradient algorithm for a class of composite optimization problems composite optimization models consist of the minimization of the sum of a smooth not necessarily convex function and a nonsmooth convex function such models arise in many applications where in addition to the composite nature of the objective function a hierarchy of models is readily available it is common to take advantage of this hierarchy of models by first solving a low fidelity model and then using the solution as a starting point to a high fidelity model we adopt an optimization point of view and show how to take advantage of the availability of a hierarchy of models in a consistent manner we do not use the low fidelity model just for the computation of promising starting points but also for the computation of search directions we establish the convergence and convergence rate of the proposed algorithm our numerical experiments on large scale image restoration problems and the transition path problem suggest that for certain classes of problems the proposed algorithm is significantly fas

local rademacher complexities we propose new bounds on the error of learning algorithms in terms of a data dependent notion of complexity the estimates we establish give optimal rates and are based on a local and empirical version of rademacher averages in the sense that the rademacher averages are computed from the data on a subset of functions with small empirical error we present some applications to classification and prediction with convex function classes and with kernel classes in particular

local differentiability of distance functions recently clarke stern and wolenski characterized in a hilbert space the closed subsets c for which the distance function dc is continuously differentiable everywhere on an open tube of uniform thickness around c here a corresponding local theory is developed for the property of dc being continuously differentiable outside of c on some neighborhood of a point x c this is shown to be equivalent to the prox regularity of c at x which is a condition on normal vectors that is commonly fulfilled in variational analysis and has the advantage of being verifiable by calculation additional characterizations are provided in terms of dc being locally of class c 1 or such that dc 2 is convex around x for some 0 prox regularity of c at x corresponds further to the normal cone mapping nc having a hypomonotone truncation around x and leads to a formula for pc by way of nc the local theory also yields new insights on the global level of the clarke stern wolenski results and on a property of sets introduced by shapiro as well as on the concept of sets with positive reach considered by federer in the finite dimensional setting

algebraic multilevel preconditioning methods ii a recursive way of constructing preconditioning matrices for the stiffness matrix in the discretization of selfadjoint second order elliptic boundary value problems is proposed it is based on a sequence of nested finite element spaces with the usual nodal basis functions using a nodeordering corresponding to the nested meshes the finite element stiffness matrix is recursively split up into two level block structures and is factored approximately in such a way that any successive schur complement is replaced approximated by a matrix defined recursively and thereform only implicitely given to solve a system with this matrix we need to perform a fixed number v of iterations on the preceding level using as an iteration matrix the preconditioning matrix already defined on that level it is shown that by a proper choice of iteration parameters it suffices to use v left 1 gamma 2 right tfrac 1 2 iterations for the so constructedv foldv cycle wherev 2 corresponds to aw cycle preconditioning matrices to be spectrally equivalent to the stiffness matrix the conditions involve only the constant in the strengthened c b s inequality for the corresponding two level hierarchical basis function spaces and are therefore independent of the regularity of the solution for instance if we use successive uniform refinements of the meshes the method is of optimal order of computational complexity if gamma 2 tfrac 8 9 under reasonable assumptions of the finite element mesh the condition numbers turn out to be so small that there are in practice few reasons to use an accelerated iterative method like the conjugate gradient method for instance