analysis of singular systems using orthogonal functions the use of orthogonal functions to analyze singular systems is investigated it is shown that the differential algebraic system equation may be converted to an algebraic generalized lyapunov equation that can be solved for the coefficients of x t in terms of the orthogonal basis functions this generalized lyapunov equation may be considered as a discrete equation on the slow subspace of the system and as a continuous equation on its fast subspace necessary and sufficient conditions for the existence of a unique solution are given in terms of the relative spectrum of the system a generalized bartels stewart algorithm based on the qz algorithm is presented for its efficient solution relations are drawn with the invariant subspaces of the system

dc programming and dca thirty years of developments the year 2015 marks the 30th birthday of dc difference of convex functions programming and dca dc algorithms which constitute the backbone of nonconvex programming and global optimization in this article we offer a short survey on thirty years of developments of these theoretical and algorithmic tools the survey is comprised of three parts in the first part we present a brief history of the field while in the second we summarize the state of the art results and recent advances we focus on main theoretical results and dca solvers for important classes of difficult nonconvex optimization problems and then give an overview of real world applications whose solution methods are based on dca the third part is devoted to new trends and important open issues as well as suggestions for future developments

a group based image inpainting using patch refinement in mrf framework this paper presents a markov random field mrf based image inpainting algorithm using patch selection from groups of similar patches and optimal patch assignment through joint patch refinement in patch selection a novel group formation strategy based on subspace clustering is introduced to search the candidate patches in relevant source region only this improves patch searching in terms of both quality and time we also propose an efficient patch refinement scheme using higher order singular value decomposition to capture underlying pattern among the candidate patches this eliminates random variation and unwanted artifacts as well finally a weight term is computed based on the refined patches and is incorporated in the objective function of the mrf model to improve the optimal patch assignment experimental results on a large number of natural images and comparison with well known existing methods demonstrate the efficacy and superiority of the proposed method

euler s elastica and curvature based inpainting image inpainting is a special image restoration problem for which image prior models play a crucial role euler s elastica was first introduced to computer vision by mumford algebraic geometry and its applications springer verlag new york 1994 pp 491 506 as a curve prior model by functionalizing the elastica energy masnou and morel proceedings of the 5th ieee international conference image processing 3 1998 pp 259 263 proposed an elastica based variational inpainting model the current paper is intended to contribute to the development of its mathematical foundation and the study of its properties and connections to the earlier works of bertalmio sapiro caselles and ballester siggraph 2000 acm press new york 2000 and chan and shen j visual comm image rep 12 2001 pp 436 449 a computational scheme based on numerical pdes is presented which allows the automatic handling of topologically complex inpainting domains

walsh function analysis of 2 d generalized continuous systems the importance of the generalized or singular 2d continuous systems are demonstrated by showing their use in the solution of partial differential equations in two variables a technique is presented for solving these systems in terms of walsh functions the method replaces the solution of a two variable partial differential equation with the solution of a linear algebraic generalized 2d sylvester equation an efficient technique for the recursive solution of the latter equation is offered all the results apply also in the usual roesser 2d state space case

observability of structured linear systems in descriptor form a graph theoretic approach this paper deals with the right hand side observability of structured linear systems in descriptor form the method we propose is based on a graph theoretic approach it is applicable to a large class of descriptor systems including regular and nonregular systems even if they are square underdetermined or overdetermined after defining digraphs which represent such systems we express in graphic terms necessary and sufficient conditions to achieve the observability of the whole or of a given part of the state these conditions are easy to check because they are based on comparison of integers and on finding edges and subgraphs in a digraph therefore our approach is suited to study large scale and or uncertain systems

nonparametric modeling of longitudinal covariance structure in functional mapping of quantitative trait loci estimation of the covariance structure of longitudinal processes is a fundamental prerequisite for the practical deployment of functional mapping designed to study the genetic regulation and network of quantitative variation in dynamic complex traits we present a nonparametric approach for estimating the covariance structure of a quantitative trait measured repeatedly at a series of time points specifically we adopt huang et al s 2006 biometrika 93 85 98 approach of invoking the modified cholesky decomposition and converting the problem into modeling a sequence of regressions of responses a regularized covariance estimator is obtained using a normal penalized likelihood with an l 2 penalty this approach embedded within a mixture likelihood framework leads to enhanced accuracy precision and flexibility of functional mapping while preserving its biological relevance simulation studies are performed to reveal the statistical properties and advantages of the proposed method a real example from a mouse genome project is analyzed to illustrate the utilization of the methodology the new method will provide a useful tool for genome wide scanning for the existence and distribution of quantitative trait loci underlying a dynamic trait important to agriculture biology and health sciences

feedback and pole placement in descriptor variable systems the effects and uses of applying linear feedback to continuous time descriptor systems are studied structural changes resulting from feeding back the slow and fast parts of the trajectory separately are characterized it is shown that under certain conditions related to controllability the poles of the slow subsystem may be shifted arbitrarily and the impulsive behaviour of the fast subsystem may be eliminated

the joint graphical lasso for inverse covariance estimation across multiple classes we consider the problem of estimating multiple related gaussian graphical models from a high dimensional data set with observations belonging to distinct classes we propose the joint graphical lasso which borrows strength across the classes in order to estimate multiple graphical models that share certain characteristics such as the locations or weights of nonzero edges our approach is based upon maximizing a penalized log likelihood we employ generalized fused lasso or group lasso penalties and implement a fast admm algorithm to solve the corresponding convex optimization problems the performance of the proposed method is illustrated through simulated and real data examples

new and efficient dca based algorithms for minimum sum of squares clustering the purpose of this paper is to develop new efficient approaches based on dc difference of convex functions programming and dca dc algorithm to perform clustering via minimum sum of squares euclidean distance we consider the two most widely used models for the so called minimum sum of squares clustering mssc in short that are a bilevel programming problem and a mixed integer program firstly the mixed integer formulation of mssc is carefully studied and is reformulated as a continuous optimization problem via a new result on exact penalty technique in dc programming dca is then investigated to the resulting problem secondly we introduce a gaussian kernel version of the bilevel programming formulation of mssc named gkmssc the gkmssc problem is formulated as a dc program for which a simple and efficient dca scheme is developed a regularization technique is investigated for exploiting the nice effect of dc decomposition and a simple procedure for finding good starting points of dca is developed the proposed dca schemes are original and very inexpensive because they amount to computing at each iteration the projection of points onto a simplex and or onto a ball and or onto a box which are all determined in the explicit form numerical results on real word datasets show the efficiency the scalability of dca and its great superiority with respect to k means and kernel k means standard methods for clustering highlightsthe purpose of this paper is to develop new efficient approaches based on dc programming and dca for clustering we consider the two models for the minimum sum of squares clustering mssc a mixed integer program and a bilevel programming formulation the mixed integer program is reformulated as a dc program via an exact penalty technique for which dca is investigated a gaussian kernel version of the bilevel programming formulation is introduced and a simple and efficient dca scheme is developed the two main features of dca say the effect of dc decomposition and of starting points are carefully studied

a d c optimization algorithm for solving the trust region subproblem this paper is devoted to difference of convex functions d c optimization d c duality local and global optimality conditions in d c programming the d c algorithm dca and its application to solving the trust region problem the dca is an iterative method that is quite different from well known related algorithms thanks to the particular structure of the trust region problem the dca is very simple requiring only matrix vector products and in practice converges to the global solution the inexpensive implicitly restarted lanczos method of sorensen is used to check the optimality of solutions provided by the dca when a nonglobal solution is found a simple numerical procedure is introduced both to find a feasible point having a smaller objective value and to restart the dca at this point it is shown that in the nonconvex case the dca converges to the global solution of the trust region problem using only matrix vector products and requiring at most 2m 2 restarts where m is the number of distinct negative eigenvalues of the coefficient matrix that defines the problem numerical simulations establish the robustness and efficiency of the dca compared to standard related methods especially for large scale problems

geometric structure and feedback in singular systems the output nulling a e r b invariant subspaces are defined for singular systems rigorously justifying the name and demonstrating that special cases of these geometric objects are the familiar subspace of admissible conditions and the supremal a e r b invariant subspace a novel singular system structure algorithm is used to compute them by numerically efficient means their importance for describing the possible closed loop geometric structure in terms of the open loop geometric structure is shown an approach to spectrum assignment in singular systems that is based on a generalized lyapunov equation is introduced the equation is used to compute feedback gains to place poles and assign various closed loop invariant subspaces while guaranteeing closed loop regularity

boundary value descriptor systems well posedness reachability and observability in this paper we introduce the class of two point boundary value descriptor systems tpbvds discrete time systems described by possibly linear dynamics and a set of boundary conditions constraining the values of the system state at the two endpoints of the system s interval of definition by introducing a standard form for regular pencils we obtain a new and simple generalized cayley hamilton theorem that simplifies our investigation of well posedness green s function solutions and reachability and observability for tpbvds there are two distinct notions of reachability and observability that one can define for tpbvds associated with processes that propagate inward from and outward toward the boundaries we investigate each of these in detail obtaining among other things far simpler forms for the reachability and observability results than found previously in the literature in addition we describe several methods for the efficient solution of tpbvds one involving recursions from each end of t

estimation of a covariance matrix with zeros summary we consider estimation of the covariance matrix of a multivariate random vector under the constraint that certain covariances are zero we first present an algorithm which we call iterative conditional fitting for computing the maximum likelihood estimate of the constrained covariance matrix under the assumption of multivariate normality in contrast to previous approaches this algorithm has guaranteed convergence properties dropping the assumption of multivariate normality we show how to estimate the covariance matrix in an empirical likelihood approach these approaches are then compared via simulation and on an example of gene expression

analysis of bilinear systems using walsh functions by using walsh functions to analyze bilinear systems it is shown that the nonlinear differential system equation can be converted to a linear algebraic generalized lyapunov equation that can be solved for the coefficients of the state x t in terms of the walsh basis functions this lyapunov equation provides an approximate closed form solution for a bilinear system some guidelines are given for selecting the number of terms in the walsh approximating series

solvability controllability and observability of continuous descriptor systems in this paper we investigate the properties of the continuous descriptor system e dot x t ax t bu t 0 leqt leqb where e a and b are complex and possibly singular matrices and u t is a complex function differentiable sufficiently many times the traditional approach to such systems is to separate the state equations from the algebraic equations however such algorithms usually destroy the natural physically based sparsity and structure of the original system therefore we consider descriptor systems in their original form such systems possess numerous properties not shared by the well known state variable systems first we relate classical theories of matrix pencils to the solvability of descriptor systems then we extend the concepts of reachability controllability and observability of state variable systems to descriptor systems and describe the set of reachable states for descriptor systems

image inpainting by patch propagation using patch sparsity this paper introduces a novel examplar based inpainting algorithm through investigating the sparsity of natural image patches two novel concepts of sparsity at the patch level are proposed for modeling the patch priority and patch representation which are two crucial steps for patch propagation in the examplar based inpainting approach first patch structure sparsity is designed to measure the confidence of a patch located at the image structure e g the edge or corner by the sparseness of its nonzero similarities to the neighboring patches the patch with larger structure sparsity will be assigned higher priority for further inpainting second it is assumed that the patch to be filled can be represented by the sparse linear combination of candidate patches under the local patch consistency constraint in a framework of sparse representation compared with the traditional examplar based inpainting approach structure sparsity enables better discrimination of structure and texture and the patch sparse representation forces the newly inpainted regions to be sharp and consistent with the surrounding textures experiments on synthetic and natural images show the advantages of the proposed approach

a nonlinear luenberger like observer for nonlinear singular systems this paper investigates observer design problem for a large class of nonlinear singular systems with multiple outputs we firstly regularize the singular system by injecting the derivative of outputs into the system then differential geometric method is applied to transform the regularized system into a simple normal form for which a luenberger like observer is proposed

sparse estimation of a covariance matrix we suggest a method for estimating a covariance matrix on the basis of a sample of vectors drawn from a multivariate normal distribution in particular we penalize the likelihood with a lasso penalty on the entries of the covariance matrix this penalty plays two important roles it reduces the effective number of parameters which is important even when the dimension of the vectors is smaller than the sample size since the number of parameters grows quadratically in the number of variables and it produces an estimate which is sparse in contrast to sparse inverse covariance estimation our method s close relative the sparsity attained here is in the covariance matrix itself rather than in the inverse matrix zeros in the covariance matrix correspond to marginal independencies thus our method performs model selection while providing a positive definite estimate of the covariance the proposed penalized maximum likelihood problem is not convex so we use a majorize minimize approach in which we iteratively solve convex approximations to the original nonconvex problem we discuss tuning parameter selection and demonstrate on a flow cytometry dataset how our method produces an interpretable graphical display of the relationship between variables we perform simulations that suggest that simple elementwise thresholding of the empirical covariance matrix is competitive with our method for identifying the sparsity structure additionally we show how our method can be used to solve a previously studied special case in which a desired sparsity pattern is prespecified

correlated z values and the accuracy of large scale statistical estimates we consider large scale studies in which there are hundreds or thousands of correlated cases to investigate each represented by its own normal variate typically a z value a familiar example is provided by a microarray experiment comparing healthy with sick subjects expression levels for thousands of genes this paper concerns the accuracy of summary statistics for the collection of normal variates such as their empirical cdf or a false discovery rate statistic it seems like we must estimate an n by n correlation matrix n the number of cases but our main result shows that this is not necessary good accuracy approximations can be based on the root mean square correlation over all n n 1 2 pairs a quantity often easily estimated a second result shows that z values closely follow normal distributions even under nonnull conditions supporting application of the main theorem practical application of the theory is illustrated for a large leukemia microarray study

analysis of descriptor systems using numerical algorithms in this paper we analyze numerical methods for the solution of the large scale dynamical system e dot y t ay t g t y t 0 y 0 where e and a are matrices possibly singular systems of this type have been referred to as implicit systems and more recently as descriptor systems since they arise from formulating system equations in physical variables special cases of such systems are algebraic differential systems we discuss the numerical advantages of this formulation and identify a class of numerical integration algorithms which have accuracy and stability properties appropriate to descriptor systems and which preserve structure detect nonsolvable systems resolve initial value consistency problems and are applicable to stiff descriptor systems we also present an algorithm for the control of the local truncation error on only the state variables

nonlinear wavelet image processing variational problems compression and noise removal through wavelet shrinkage this paper examines the relationship between wavelet based image processing algorithms and variational problems algorithms are derived as exact or approximate minimizers of variational problems in particular we show that wavelet shrinkage can be considered the exact minimizer of the following problem given an image f defined on a square i minimize over all g in the besov space b11 l1 i the functional f g l2 i 2 g b11 l1 i we use the theory of nonlinear wavelet image compression in l2 i to derive accurate error bounds for noise removal through wavelet shrinkage applied to images corrupted with i i d mean zero gaussian noise a new signal to noise ratio snr which we claim more accurately reflects the visual perception of noise in images arises in this derivation we present extensive computations that support the hypothesis that near optimal shrinkage parameters can be derived if one knows or can estimate only two parameters about an image f the largest for which f bq lq i 1 q 2 1 2 and the norm f bq lq i both theoretical and experimental results indicate that our choice of shrinkage parameters yields uniformly better results than donoho and johnstone s visushrink procedure an example suggests however that donoho and johnstone s 1994 1995 1996 sureshrink method which uses a different shrinkage parameter for each dyadic level achieves a lower error than our procedure

exemplar based inpainting technical review and new heuristics for better geometric reconstructions this paper proposes a technical review of exemplar based inpainting approaches with a particular focus on greedy methods several comparative and illustrative experiments are provided to deeply explore and enlighten these methods and to have a better understanding on the state of the art improvements of these approaches from this analysis three improvements over criminisi et al algorithm are then presented and detailed 1 a tensor based data term for a better selection of pixel candidates to fill in 2 a fast patch lookup strategy to ensure a better global coherence of the reconstruction and 3 a novel fast anisotropic spatial blending algorithm that reduces typical block artifacts using tensor models relevant comparisons with the state of the art inpainting methods are provided that exhibit the effectiveness of our contributions

controllability and observability of second order descriptor systems we analyze controllability and observability conditions for second order descriptor systems and show how the classical conditions for first order systems can be generalized to this case we show that performing a classical transformation to first order form may destroy some controllability and observability properties as an example we demonstrate that the loss of impulse controllability in constrained multibody systems is due to the representation as a first order system to avoid this problem we will derive a canonical form and new first order formulations that do not destroy the controllability and observability properties

hierarchical super resolution based inpainting this paper introduces a novel framework for examplar based inpainting it consists in performing first the inpainting on a coarse version of the input image a hierarchical super resolution algorithm is then used to recover details on the missing areas the advantage of this approach is that it is easier to inpaint low resolution pictures than high resolution ones the gain is both in terms of computational complexity and visual quality however to be less sensitive to the parameter setting of the inpainting method the low resolution input picture is inpainted several times with different configurations results are efficiently combined with a loopy belief propagation and details are recovered by a single image super resolution algorithm experimental results in a context of image editing and texture synthesis demonstrate the effectiveness of the proposed method results are compared to five state of the art inpainting methods

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

image inpainting through metric labeling via guided patch mixing in this paper we present a novel formulation of exemplar based image inpainting as a metric labeling problem and solve it through the simulated annealing algorithm due to their greedy nature exemplar based methods sometimes produce inpainted images which are visually inconsistent these methods are highly dependent upon the initialization to solve these problems we generate five images with a different initialization a suitable mixture of these five images produces a good inpainted image the cost function of the proposed metric labeling problem consists of three components namely neighbor cost total variation cost and structure cost a linear combination among these components is used to maintain better visual consistency in the inpainted region having smooth transition from the bordering regions of the source image we use a quality measure to this end our experiments on a wide variety of images demonstrate that the proposed technique produces better inpainting images as compared with some other state of the art techniques

constructing a homogeneous lti descriptor system with desired properties using perturbation theory in the literature regarding linear systems and mathematical control theory several different techniques have been developed for obtaining the solution of homogeneous linear time invariant lti descriptor differential systems in this article applying the complex weierstrass canonical form we investigate the conditions under which a descriptor system with a specific structure and desired properties is being constructed using perturbation theory our approach is very general and as an example a stable homogeneous lti descriptor system is designed thus a proportional and derivative controller can be used such as the case where a family of perturbed pencils is defined and the solutions of the initial and the relative perturbed systems are close with respect to a frobenius distance a step algorithm and an illustrative example are also presented to illustrate the results of this article

no reference image quality assessment in the spatial domain we propose a natural scene statistic based distortion generic blind no reference nr image quality assessment iqa model that operates in the spatial domain the new model dubbed blind referenceless image spatial quality evaluator brisque does not compute distortion specific features such as ringing blur or blocking but instead uses scene statistics of locally normalized luminance coefficients to quantify possible losses of naturalness in the image due to the presence of distortions thereby leading to a holistic measure of quality the underlying features used derive from the empirical distribution of locally normalized luminances and products of locally normalized luminances under a spatial natural scene statistic model no transformation to another coordinate frame dct wavelet etc is required distinguishing it from prior nr iqa approaches despite its simplicity we are able to show that brisque is statistically better than the full reference peak signal to noise ratio and the structural similarity index and is highly competitive with respect to all present day distortion generic nr iqa algorithms brisque has very low computational complexity making it well suited for real time applications brisque features may be used for distortion identification as well to illustrate a new practical application of brisque we describe how a nonblind image denoising algorithm can be augmented with brisque in order to perform blind image denoising results show that brisque augmentation leads to performance improvements over state of the art methods a software release of brisque is available online http live ece utexas edu research quality brisque release zip for public use and evaluation

technical communique regularizability of linear time invariant descriptor systems under decentralized control regularizability conditions for linear time invariant descriptor systems under decentralized output feedback are presented in this paper the regularizability is characterized by rank conditions that involve the original system matrices only new rank conditions for regularity of the system are also derived

the concave convex procedure the concave convex procedure cccp is a way to construct discrete time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically this procedure can be applied to almost any optimization problem and many existing algorithms can be interpreted in terms of it in particular we prove that all expectation maximization algorithms and classes of legendre minimization and variational bounding algorithms can be reexpressed in terms of cccp we show that many existing neural network and mean field theory algorithms are also examples of cccp the generalized iterative scaling algorithm and sinkhorn s algorithm can also be expressed as cccp by changing variables cccp can be used both as a new way to understand and prove the convergence of existing optimization algorithms and as a procedure for generating new algorithms

comments on the properties of the operational matrices of integration and differentiation for fourier trigonometric functions the properties of the operational matrices of integration p and differentiation q are discussed for the case of fourier trigonometric functions in particular a theorem that establishes the relationship between the matrices p sup 1 and q is given and the meaning of using p sup 1 instead of q recently proposed in literature is clarified

a tutorial on the geometric analysis of linear time invariant implicit systems abstract a background of recent work in nonregular and regular implicit systems of the form e x ax bu is provided then the output nulling unknown input and composite subspaces are defined for singular systems with recursions given for their computation inner subspaces that is those in the domain of e and a are distinguished from outer subspaces those in the codomain of e and a the notions of composite preimage and composite image are introduced the result is a framework that unifies some of the recent work in the geometric theory of implicit systems system properties are discussed for both regular and nonregular systems and duality is investigated

sparse precision matrix estimation via lasso penalized d trace loss we introduce a constrained empirical loss minimization framework for estimating high dimensional sparse precision matrices and propose a new loss function called the d trace loss for that purpose a novel sparse precision matrix estimator is defined as the minimizer of the lasso penalized d trace loss under a positive definiteness constraint under a new irrepresentability condition the lasso penalized d trace estimator is shown to have the sparse recovery property examples demonstrate that the new condition can hold in situations where the irrepresentability condition for the lasso penalized gaussian likelihood estimator fails we establish rates of convergence for the new estimator in the elementwise maximum frobenius and operator norms we develop a very efficient algorithm based on alternating direction methods for computing the proposed estimator simulated and real data are used to demonstrate the computational efficiency of our algorithm and the finite sample performance of the new estimator the lasso penalized d trace estimator is found to compare favourably with the lasso penalized gaussian likelihood estimator

positive definite l1 penalized estimation of large covariance matrices the thresholding covariance estimator has nice asymptotic properties for estimating sparse large covariance matrices but it often has negative eigenvalues when used in real data analysis to fix this drawback of thresholding estimation we develop a positive definite l1 penalized covariance estimator for estimating sparse large covariance matrices we derive an efficient alternating direction method to solve the challenging optimization problem and establish its convergence properties under weak regularity conditions nonasymptotic statistical theory is also established for the proposed estimator the competitive finite sample performance of our proposal is demonstrated by both simulation and real applications

filling in by joint interpolation of vector fields and gray levels a variational approach for filling in regions of missing data in digital images is introduced the approach is based on joint interpolation of the image gray levels and gradient isophotes directions smoothly extending in an automatic fashion the isophote lines into the holes of missing data this interpolation is computed by solving the variational problem via its gradient descent flow which leads to a set of coupled second order partial differential equations one for the gray levels and one for the gradient orientations the process underlying this approach can be considered as an interpretation of the gestaltist s principle of good continuation no limitations are imposed on the topology of the holes and all regions of missing data can be simultaneously processed even if they are surrounded by completely different structures applications of this technique include the restoration of old photographs and removal of superimposed text like dates subtitles or publicity examples of these applications are given we conclude the paper with a number of theoretical results on the proposed variational approach and its corresponding gradient descent flow

disocclusion a variational approach using level lines object recognition robot vision image and film restoration may require the ability to perform disocclusion we call disocclusion the recovery of occluded areas in a digital image by interpolation from their vicinity it is shown in this paper how disocclusion can be performed by means of the level lines structure which offers a reliable complete and contrast invariant representation of images level lines based disocclusion yields a solution that may have strong discontinuities the proposed method is compatible with kanizsa s 1979 amodal completion theory

model selection and estimation in the gaussian graphical model summary we propose penalized likelihood methods for estimating the concentration matrix in the gaussian graphical model the methods lead to a sparse and shrinkage estimator of the concentration matrix that is positive definite and thus conduct model selection and estimation simultaneously the implementation of the methods is nontrivial because of the positive definite constraint on the concentration matrix but we show that the computation can be done effectively by taking advantage of the efficient maxdet algorithm developed in convex optimization we propose a bic type criterion for the selection of the tuning parameter in the penalized likelihood methods the connection between our methods and existing methods is illustrated simulations and real examples demonstrate the competitive performance of the new methods

convergence of an annealing algorithm the annealing algorithm is a stochastic optimization method which has attracted attention because of its success with certain difficult problems including np hard combinatorial problems such as the travelling salesman steiner trees and others there is an appealing physical analogy for its operation but a more formal model seems desirable in this paper we present such a model and prove that the algorithm converges with probability arbitrarily close to 1 we also show that there are cases where convergence takes exponentially long that is it is no better than a deterministic method we study how the convergence rate is affected by the form of the problem finally we describe a version of the algorithm that terminates in polynomial time and allows a good deal of practical confidence in the solution

a constrained l1 minimization approach to sparse precision matrix estimation this article proposes a constrained l1 minimization method for estimating a sparse inverse covariance matrix based on a sample of n iid p variate random variables the resulting estimator is shown to have a number of desirable properties in particular the rate of convergence between the estimator and the true s sparse precision matrix under the spectral norm is when the population distribution has either exponential type tails or polynomial type tails we present convergence rates under the elementwise l norm and frobenius norm in addition we consider graphical model selection the procedure is easily implemented by linear programming numerical performance of the estimator is investigated using both simulated and real data in particular the procedure is applied to analyze a breast cancer dataset and is found to perform favorably compared with existing methods

block clustering based on difference of convex functions dc programming and dc algorithms we investigate difference of convex functions dc programming and the dc algorithm dca to solve the block clustering problem in the continuous framework which traditionally requires solving a hard combinatorial optimization problem dc reformulation techniques and exact penalty in dc programming are developed to build an appropriate equivalent dc program of the block clustering problem they lead to an elegant and explicit dca scheme for the resulting dc program computational experiments show the robustness and efficiency of the proposed algorithm and its superiority over standard algorithms such as two mode k means two mode fuzzy clustering and block classification em

walsh function analysis of linear and bilinear discrete time systems a technique is presented for analysing linear and bilinear discrete systems in terms of walsh functions the method replaces the solution of a dynamical difference equation with the solution of an algebraic equation in the special case of a linear system a numerically stable recursive technique is derived to solve this algebraic equation the technique is illustrated with an example

exemplar based image inpainting using multiscale graph cuts we present a novel formulation of exemplar based inpainting as a global energy optimization problem written in terms of the offset map the proposed energy function combines a data attachment term that ensures the continuity of reconstruction at the boundary of the inpainting domain with a smoothness term that ensures a visually coherent reconstruction inside the hole this formulation is adapted to obtain a global minimum using the graph cuts algorithm to reduce the computational complexity we propose an efficient multiscale graph cuts algorithm to compensate the loss of information at low resolution levels we use a feature representation computed at the original image resolution this permits alleviation of the ambiguity induced by comparing only color information when the image is represented at low resolution levels our experiments show how well the proposed algorithm performs compared with other recent algorithms

deflation in eigenvalue assignment of descriptor systems using state feedback it is shown how deflation by unitary equivalence transformations can be used to solve the eigenvalue assignment problem for descriptor systems using state feedback the system is first transformed to a form that reveals its controllability the uncontrollable part of the system if any is then discarded and the assignments are continued with the completely controllable part of the system to give insight into the approach a mathematical method for the solution is derived without considering numerical issues using the idea of deflation it is shown how this method can be modified to derive a numerically efficient algorithm the numerical stability of the algorithm is proved and a numerical example is presented

optimization based dc programming and dca for hierarchical clustering one of the most promising approaches for clustering is based on methods of mathematical programming in this paper we propose new optimization methods based on dc difference of convex functions programming for hierarchical clustering a bilevel hierarchical clustering model is considered with different optimization formulations they are all nonconvex nonsmooth optimization problems for which we investigate attractive dc optimization algorithms called dca numerical results on some artificial and real world databases are reported the results demonstrate that the proposed algorithms are more efficient than related existing methods

sparsistency and rates of convergence in large covariance matrix estimation this paper studies the sparsistency and rates of convergence for estimating sparse covariance and precision matrices based on penalized likelihood with nonconvex penalty functions here sparsistency refers to the property that all parameters that are zero are actually estimated as zero with probability tending to one depending on the case of applications sparsity priori may occur on the covariance matrix its inverse or its cholesky decomposition we study these three sparsity exploration problems under a unified framework with a general penalty function we show that the rates of convergence for these problems under the frobenius norm are of order s n log p n n 1 2 where s n is the number of nonzero elements p n is the size of the covariance matrix and n is the sample size this explicitly spells out the contribution of high dimensionality is merely of a logarithmic factor the conditions on the rate with which the tuning parameter n goes to 0 have been made explicit and compared under different penalties as a result for the l 1 penalty to guarantee the sparsistency and optimal rate of convergence the number of nonzero elements should be small sn o pn at most among o pn2 parameters for estimating sparse covariance or correlation matrix sparse precision or inverse correlation matrix or sparse cholesky factor where sn is the number of the nonzero elements on the off diagonal entries on the other hand using the scad or hard thresholding penalty functions there is no such a restriction

sparse covariance matrix estimation by dca based algorithms this letter proposes a novel approach using the formula see text norm regularization for the sparse covariance matrix estimation scme problem the objective function of scme problem is composed of a nonconvex part and the formula see text term which is discontinuous and difficult to tackle appropriate dc difference of convex functions approximations of formula see text norm are used that result in approximation scme problems that are still nonconvex dc programming and dca dc algorithm powerful tools in nonconvex programming framework are investigated two dc formulations are proposed and corresponding dca schemes developed two applications of the scme problem that are considered are classification via sparse quadratic discriminant analysis and portfolio optimization a careful empirical experiment is performed through simulated and real data sets to study the performance of the proposed algorithms numerical results showed their efficiency and their superiority compared with seven state of the art methods

generalized functional observer a functional observer and state feedback are proposed for singular systems in the polynomial fraction form that requires no impulsive mode elimination the order of the compensator is determined by the newly defined generalized observability index that is associated with the mcmillan degree of the system a new generalized lyapunov equation is also proposed through a realization scheme that can be applied to both ordinary and singular systems the solution to the equation provides an algebraic approach to the observer of singular systems in the generalized state space form

a geometric approach to eigenstructure assignment for singular systems moore s well known result on the feedback assignment of eigenstructure in linear state space systems 5 was generalized to descriptor systems in 9 in this note we make some useful extensions to fill out the theory to begin with we do not assume controllability of the plant it is shown that the maximum number of finite closed loop eigenvalues is equal to dim el where l is the supremal a e r b invariant subspace in order to relate the possible closed loop structure to the open loop plant structure we define a set of indexes for descriptor systems which are in many respects similar to the controllability indexes of state space systems in terms of these controllability indexes we give a systematic approach to selecting the closed loop eigenvectors a major contribution of this note is the fact that it is geometric in nature and hence avoids any decomposition of the system matrices into a special form e g weierstrass form 1 the geometric setting introduced here should constitute a basis for further research in generalizing the well known results in geometric system theory disturbance decoupling input output decoupling etc to singular systems

positive definite estimators of large covariance matrices using convex optimization we construct a sparse estimator of the covariance matrix that is positive definite and performs well in high dimensional settings a lasso type penalty is used to encourage sparsity and a logarithmic barrier function is used to enforce positive definiteness consistency and convergence rate bounds are established as both the number of variables and sample size diverge an efficient computational algorithm is developed and the merits of the approach are illustrated with simulations and a speech signal classification example copyright 2012 oxford university press

structure analysis via orthogonal functions the use of orthogonal functions to analyse the structure of a system is investigated applying the definitions of observability and controllability to a system that is approximated with the help of orthogonal functions it is shown that the concepts of the state space and the space of orthogonal functions are equivalent provided that two weak conditions are met this result ensures that the observability and controllability properties remain invariant under the transformation introduced by the approximation furthermore new criteria to test observability and controllability are given in terms of the coefficient matrix of the orthogonal expansion because this test does not require the knowledge of the system matrices a b and c the results derived may be used for the identification of systems it is demonstrated that all the results obtained remain true even for an approximation with low accuracy these properties allow the application of orthogonal functions for the analysis of systems

penalized classification using fisher s linear discriminant we consider the supervised classification setting in which the data consist of p features measured on n observations each of which belongs to one of k classes linear discriminant analysis lda is a classical method for this problem however in the high dimensional setting where p n lda is not appropriate for two reasons first the standard estimate for the within class covariance matrix is singular and so the usual discriminant rule cannot be applied second when p is large it is difficult to interpret the classification rule obtained from lda since it involves all p features we propose penalized lda a general approach for penalizing the discriminant vectors in fisher s discriminant problem in a way that leads to greater interpretability the discriminant problem is not convex so we use a minorization maximization approach in order to efficiently optimize it when convex penalties are applied to the discriminant vectors in particular we consider the use of l 1 and fused lasso penalties our proposal is equivalent to recasting fisher s discriminant problem as a biconvex problem we evaluate the performances of the resulting methods on a simulation study and on three gene expression data sets we also survey past methods for extending lda to the high dimensional setting and explore their relationships with our proposal

generalized thresholding of large covariance matrices we propose a new class of generalized thresholding operators that combine thresholding with shrinkage and study generalized thresholding of the sample covariance matrix in high dimensions generalized thresholding of the covariance matrix has good theoretical properties and carries almost no computational burden we obtain an explicit convergence rate in the operator norm that shows the tradeoff between the sparsity of the true model dimension and the sample size and shows that generalized thresholding is consistent over a large class of models as long as the dimension p and the sample size n satisfy log p n 0 in addition we show that generalized thresholding has the sparsistency property meaning it estimates true zeros as zeros with probability tending to 1 and under an additional mild condition is sign consistent for nonzero elements we show that generalized thresholding covers as special cases hard and soft thresholding smoothly clipped absolute deviation and adaptive lasso and comp

approximation algorithms for classification problems with pairwise relationships metric labeling and markov random fields in a traditional classification problem we wish to assign one of k labels or classes to each of n objects in a way that is consistent with some observed data that we have about the problem an active line of research in this area is concerned with classification when one has information about pairwise relationships among the objects to be classified this issue is one of the principal motivations for the framework of markov random fields and it arises in areas such as image processing biometry and document analysis in its most basic form this style of analysis seeks to find a classification that optimizes a combinatorial function consisting of assignment costs based on the individual choice of label we make for each object and separation costs based on the pair of choices we make for two related objects we formulate a general classification problem of this type the metric labeling problem we show that it contains as special cases a number of standard classification frameworks including several arising from the theory of markov random fields from the perspective of combinatorial optimization our problem can be viewed as a substantial generalization of the multiway cut problem and equivalent to a type of uncapacitated quadratic assignment problem we provide the first nontrivial polynomial time approximation algorithms for a general family of classification problems of this type our main result is an o log k log log k approximation algorithm for the metric labeling problem with respect to an arbitrary metric on a set of k labels and an arbitrary weighted graph of relationships on a set of objects for the special case in which the labels are endowed with the uniform metric all distances are the same our methods provide a 2 approximation algorithm

model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data we consider the problem of estimating the parameters of a gaussian or binary distribution in such a way that the resulting undirected graphical model is sparse our approach is to solve a maximum likelihood problem with an added l1 norm penalty term the problem as formulated is convex but the memory requirements and complexity of existing interior point methods are prohibitive for problems with more than tens of nodes we present two new algorithms for solving problems with at least a thousand nodes in the gaussian case our first algorithm uses block coordinate descent and can be interpreted as recursive l1 norm penalized regression our second algorithm based on nesterov s first order method yields a complexity estimate with a better dependence on problem size than existing interior point methods using a log determinant relaxation of the log partition function wainwright and jordan 2006 we show that these same algorithms can be used to solve an approximate sparse maximum likelihood problem for the binary case we test our algorithms on synthetic data as well as on gene expression and senate voting records data