on perturbation bounds for the qr factorization abstract certain new perturbation bounds of the orthogonal factor in the qr factorization of a real matrix are derived the bounds of this note improve the known bounds in the literature

maximum likelihood based estimation of frequency and phase offset in dct ofdm systems under non circular transmissions algorithms analysis and comparisons recently the advantages of the discrete cosine transform dct based orthogonal frequency division multiplexing ofdm have come to the light we thus consider dct ofdm with non circular transmission our results cover circular transmission as well and present two blind joint maximum likelihood frequency offset and phase offset estimators both our theoretical analysis and numerical comparisons reveal new advantages of dct ofdm over the traditional discrete fourier transform dft based ofdm these advantages as well as those already uncovered in the early works on dct ofdm support the belief that dct ofdm is a promising multi carrier modulation scheme

a riemannian optimization approach to the matrix singular value decomposition the problem of the singular value decomposition of a matrix can be brought into an optimization problem on the product of two stiefel manifolds of different sizes the steepest descent the conjugate gradient and newton s methods for the problem are developed and applied with several numerical experiments these algorithms do not need the preconditioning that is inevitable in the usual singular value decomposition algorithm the present newton s method can serve to make more accurate the singular value decomposition obtained by other existing algorithms in addition degenerate optimal solutions are studied together with numerical experiments to show that those solutions form a disconnected submanifold

on the interchange of derivative and expectation for likelihood ratio derivative estimators sufficient conditions for the validity of interchange between derivative and expectation in the context of likelihood ratio gradient estimation were given in l ecuyer 1990 the aim of this paper is to shed additional light on these conditions and introduce specific variants of them which are often easier to check sufficient conditions for the derivative estimator to have finite moments up to a given order are also given and illustrated by examples in particular we give an example of an unbiased derivative estimator which satisfies the interchange conditions but which has infinite variance

a unified view of the ipa sf and lr gradient estimation techniques we study the links between the likelihood ratio lr gradient estimation technique sometimes called the score function sf method and infinitesimal perturbation analysis ipa we show how ipa can be viewed as a degenerate special case of the lr and sf techniques by selecting an appropriate representation of the underlying sample space for a given simulation experiment we also show how different definitions of the sample space yield different variants of the lr method some of them mixing ipa with more straightforward lr we illustrate this by many examples we also give sufficient conditions under which the gradient estimators are unbiased

quadratic optimization with orthogonality constraint explicit łojasiewicz exponent and linear convergence of retraction based line search and stochastic variance reduced gradient methods the problem of optimizing a quadratic form over an orthogonality constraint qp oc for short is one of the most fundamental matrix optimization problems and arises in many applications in this paper we characterize the growth behavior of the objective function around the critical points of the qp oc problem and demonstrate how such characterization can be used to obtain strong convergence rate results for iterative methods that exploit the manifold structure of the orthogonality constraint i e the stiefel manifold to find a critical point of the problem specifically our primary contribution is to show that the ojasiewicz exponent at any critical point of the qp oc problem is 1 2 such a result is significant as it expands the currently very limited repertoire of optimization problems for which the ojasiewicz exponent is explicitly known moreover it allows us to show in a unified manner and for the first time that a large family of retraction based line search methods will converge linearly to a critical point of the qp oc problem then as our secondary contribution we propose a stochastic variance reduced gradient svrg method called stiefel svrg for solving the qp oc problem and present a novel ojasiewicz inequality based linear convergence analysis of the method an important feature of stiefel svrg is that it allows for general retractions and does not require the computation of any vector transport on the stiefel manifold as such it is computationally more advantageous than other recently proposed svrg type algorithms for manifold optimization

convergence of neural networks for programming problems via a nonsmooth lojasiewicz inequality this paper considers a class of neural networks nns for solving linear programming lp problems convex quadratic programming qp problems and nonconvex qp problems where an indefinite quadratic objective function is subject to a set of affine constraints the nns are characterized by constraint neurons modeled by ideal diodes with vertical segments in their characteristic which enable to implement an exact penalty method a new method is exploited to address convergence of trajectories which is based on a nonsmooth lstrokojasiewicz inequality for the generalized gradient vector field describing the nn dynamics the method permits to prove that each forward trajectory of the nn has finite length and as a consequence it converges toward a singleton furthermore by means of a quantitative evaluation of the lstrokojasiewicz exponent at the equilibrium points the following results on convergence rate of trajectories are established 1 for nonconvex qp problems each trajectory is either exponentially convergent or convergent in finite time toward a singleton belonging to the set of constrained critical points 2 for convex qp problems the same result as in 1 holds moreover the singleton belongs to the set of global minimizers and 3 for lp problems each trajectory converges in finite time to a singleton belonging to the set of global minimizers these results which improve previous results obtained via the lyapunov approach are true independently of the nature of the set of equilibrium points and in particular they hold even when the nn possesses infinitely many nonisolated equilibrium points

adaptive image restoration using a generalized gaussian model for unknown noise a model adaptive method is proposed for restoring blurred and noise corrupted images the generalized p gaussian family of probability density functions is used as the approximating parametric noise model distribution shape parameters are estimated from the image and the resulting maximum likelihood optimization problem is solved an iterative algorithm for data directed restoration is presented and analyzed

convergence of the iterates of descent methods for analytic cost functions in the early eighties lojasiewicz in seminari di geometria 1982 1983 universita di bologna istituto di geometria dipartimento di matematica 1984 pp 115 117 proved that a bounded solution of a gradient flow for an analytic cost function converges to a well defined limit point in this paper we show that the iterates of numerical descent algorithms for an analytic cost function share this convergence property if they satisfy certain natural descent conditions the results obtained are applicable to a broad class of optimization schemes and strengthen classical weak convergence results for descent methods to strong limit point convergence for a large class of cost functions of practical interest the result does not require that the cost has isolated critical points and requires no assumptions on the convexity of the cost nor any nondegeneracy conditions on the hessian of the cost at critical points

extrema of sums of heterogeneous quadratic forms abstract we analyze the following problem arising in various situations in multivariate statistical analysis we are given k symmetric positive definite n n matrices a1 a2 ak k n and we would like to maximize the function ki 1 xtiaixi under the constraint that x1 x2 xk rn form an orthonormal system some theoretical results as well as an algorithm are presented

a framework of constraint preserving update schemes for optimization on stiefel manifold this paper considers optimization problems on the stiefel manifold x mathsf t x i p xtx ip where x in mathbb r n times p x rn p is the variable and i p ip is the p p by p p identity matrix a framework of constraint preserving update schemes is proposed by decomposing each feasible point into the range space of x x and the null space of x mathsf t xt while this general framework can unify many existing schemes a new update scheme with low complexity cost is also discovered then we study a feasible barzilai borwein like method under the new update scheme the global convergence of the method is established with an adaptive nonmonotone line search the numerical tests on the nearest low rank correlation matrix problem the kohn sham total energy minimization and a specific problem from statistics demonstrate the efficiency of the new method in particular the new method performs remarkably well for the nearest low rank correlation matrix problem in terms of speed and solution quality and is considerably competitive with the widely used scf iteration for the kohn sham total energy minimization

asymptotics of likelihood ratio derivative estimators in simulations of highly reliable markovian systems we discuss the estimation of derivatives of a performance measure using the likelihood ratio method in simulations of highly reliable markovian systems we compare the difficulties of estimating the performance measure and of estimating its partial derivatives with respect to component failure rates as the component failure rates tend to 0 and the component repair rates remain fixed we first consider the case when the quantities are estimated using naive simulation i e when no variance reduction technique is used in particular we prove that in the limit some of the partial derivatives can be estimated as accurately as the performance measure itself this result is of particular interest in light of the somewhat pessimistic empirical results others have obtained when applying the likelihood ratio method to other types of systems however the result only holds for certain partial derivatives of the performance measure when using naive simulation more specifically we can estimate a certain partial

stochastic blockmodels and community structure in networks stochastic blockmodels have been proposed as a tool for detecting community structure in networks as well as for generating synthetic networks for use as benchmarks most blockmodels however ignore variation in vertex degree making them unsuitable for applications to real world networks which typically display broad degree distributions that can significantly affect the results here we demonstrate how the generalization of blockmodels to incorporate this missing element leads to an improved objective function for community detection in complex networks we also propose a heuristic algorithm for community detection using this objective function or its non degree corrected counterpart and show that the degree corrected version dramatically outperforms the uncorrected one in both real world and synthetic networks

near optimal bounds for phase synchronization the problem of estimating the phases angles of a complex unit modulus vector z from their noisy pairwise relative measurements c zz sigma w where w is a complex valued gaussian rand

truncated power method for sparse eigenvalue problems this paper considers the sparse eigenvalue problem which is to extract dominant largest sparse eigenvectors with at most k non zero components we propose a simple yet effective solution called truncated power method that can approximately solve the underlying nonconvex optimization problem a strong sparse recovery result is proved for the truncated power method and this theory is our key motivation for developing the new algorithm the proposed method is tested on applications such as sparse principal component analysis and the densest k subgraph problem extensive experiments on several synthetic and real world data sets demonstrate the competitive empirical performance of our method

empirical arithmetic averaging over the compact stiefel manifold the aim of the present research work is to investigate algorithms to compute empirical averages of finite sets of sample points over the stiefel manifold by extending the notion of pythagoras arithmetic averaging over the real line to a curved manifold the idea underlying the developed algorithms is that sample points on the stiefel manifold get mapped onto a tangent space where the average is taken and then the average point on the tangent space is brought back to the stiefel manifold via appropriate maps numerical experimental results are shown and commented on in order to illustrate the numerical behaviour of the proposed procedure the obtained numerical results confirm that the developed algorithms converge steadily and in a few iterations and that they are able to cope with relatively large size problems

on the estimation performance and convergence rate of the generalized power method for phase synchronization an estimation problem of fundamental interest is that of phase or angular synchronization in which the goal is to recover a collection of phases or angles using noisy measurements of relative phases or angle offsets it is known that in the gaussian noise setting the maximum likelihood estimator mle is an optimal solution to a nonconvex quadratic optimization problem and can be found with high probability using semidefinite programming sdp provided that the noise power is not too large in this paper we study the estimation and convergence performance of a recently proposed low complexity alternative to the sdp based approach namely the generalized power method gpm our contribution is twofold first we show that the sequence of estimation errors associated with the gpm iterates is bounded above by a decreasing sequence as a corollary we show that all iterates achieve an estimation error that is on the same order as that of an mle our result holds under the least restrictive assumpti

a family of inexact sqa methods for non smooth convex minimization with provable convergence guarantees based on the luo tseng error bound property we propose a new family of inexact sequential quadratic approximation sqa methods which we call the inexact regularized proximal newton irpn method for minimizing the sum of two closed proper convex functions one of which is smooth and the other is possibly non smooth our proposed method features strong convergence guarantees even when applied to problems with degenerate solutions while allowing the inner minimization to be solved inexactly specifically we prove that when the problem possesses the so called luo tseng error bound eb property irpn converges globally to an optimal solution and the local convergence rate of the sequence of iterates generated by irpn is linear superlinear or even quadratic depending on the choice of parameters of the algorithm prior to this work such eb property has been extensively used to establish the linear convergence of various first order methods however to the best of our knowledge this work is the first to use the luo tseng eb property to establish the superlinear convergence of sqa type methods for non smooth convex minimization as a consequence of our result irpn is capable of solving regularized regression or classification problems under the high dimensional setting with provable convergence guarantees we compare our proposed irpn with several empirically efficient algorithms by applying them to the ell 1 regularized logistic regression problem experiment results show the competitiveness of our proposed method

convergence results for projected line search methods on varieties of low rank matrices via łojasiewicz inequality the aim of this paper is to derive convergence results for projected line search methods on the real algebraic variety mathcal m le k of real m times n matrices of rank at most k such methods extend riemannian optimization methods which are successfully used on the smooth manifold mathcal m k of rank k matrices to its closure by taking steps along gradient related directions in the tangent cone and afterwards projecting back to mathcal m le k considering such a method circumvents the difficulties which arise from the nonclosedness and the unbounded curvature of mathcal m k the pointwise convergence is obtained for real analytic functions on the basis of a ojasiewicz inequality for the projection of the antigradient to the tangent cone if the derived limit point lies on the smooth part of mathcal m le k i e in mathcal m k this boils down to more or less known results but with the benefit that asymptotic convergence rate estimates for specific step sizes

importance sampling for stochastic simulations importance sampling is one of the classical variance reduction techniques for increasing the efficiency of monte carlo algorithms for estimating integrals the basic idea is to replace the original random mechanism in the simulation by a new one and at the same time modify the function being integrated in this paper the idea is extended to problems arising in the simulation of stochastic systems discrete time markov chains continuous time markov chains and generalized semi markov processes are covered applications are given to a gi g 1 queueing problem and response surface estimation computation of the theoretical moments arising in importance sampling is discussed and some numerical examples given

phase retrieval via wirtinger flow theory and algorithms we study the problem of recovering the phase from magnitude measurements specifically we wish to reconstruct a complex valued signal boldsymbol x in mathbb c n about which we have phaseless samples of the form y r left langle boldsymbol a r boldsymbol x rangle right 2 r 1 ldots m knowledge of the phase of these samples would yield a linear system this paper develops a nonconvex formulation of the phase retrieval problem as well as a concrete solution algorithm in a nutshell this algorithm starts with a careful initialization obtained by means of a spectral method and then refines this initial estimate by iteratively applying novel update rules which have low computational complexity much like in a gradient descent scheme the main contribution is that this algorithm is shown to rigorously allow the exact retrieval of phase information from a nearly minimal number of random measurements indeed the sequence of successive iterates provably converges to the solution at a geometric rate so that the proposed scheme is efficient both in terms of computational and data resources in theory a variation on this scheme leads to a near linear time algorithm for a physically realizable model based on coded diffraction patterns we illustrate the effectiveness of our methods with various experiments on image data underlying our analysis are insights for the analysis of nonconvex optimization schemes that may have implications for computational problems beyond phase retrieval

an mdl framework for sparse coding and dictionary learning the power of sparse signal modeling with learned overcomplete dictionaries has been demonstrated in a variety of applications and fields from signal processing to statistical inference and machine learning however the statistical properties of these models such as underfitting or overfitting given sets of data are still not well characterized in the literature as a result the success of sparse modeling depends on hand tuning critical parameters for each data and application this work aims at addressing this by providing a practical and objective characterization of sparse models by means of the minimum description length mdl principle a well established information theoretic approach to model selection in statistical inference the resulting framework derives a family of efficient sparse coding and dictionary learning algorithms which by virtue of the mdl principle are completely parameter free furthermore such framework allows to incorporate additional prior information to existing models such as markovian dependencies or to define completely new problem formulations including in the matrix analysis area in a natural way these virtues will be demonstrated with parameter free algorithms for the classic image denoising and classification problems and for low rank matrix recovery in video applications however the framework is not limited to this imaging data and can be applied to a wide range of signal and data types and tasks

identification of transform coding chains transform coding is routinely used for lossy compression of discrete sources with memory the input signal is divided into n dimensional vectors which are transformed by means of a linear mapping then transform coefficients are quantized and entropy coded in this paper we consider the problem of identifying the transform matrix as well as the quantization step sizes first we study the case in which the only available information is a set of p transform decoded vectors we formulate the problem in terms of finding the lattice with the largest determinant that contains all observed vectors we propose an algorithm that is able to find the optimal solution and we formally study its convergence properties three potential realms of application are considered as example scenarios for the proposed theory 1 parameter retrieval in the presence of a chain of two transform coders 2 image tampering identification and 3 parameter estimation for predictive coders we show that despite their differences all three scenarios can be tackled by applying the same fundamental methodology experiments on both the synthetic data and the real images validate the proposed approach

likelihood ratio derivative estimation for finite time performance measures in generalized semi markov processes this paper investigates the likelihood ratio method for estimating derivatives of finite time performance measures in generalized semi markov processes gsmps we develop readily verifiable conditions for the applicability of this method our conditions mainly place restrictions on the basic building blocks i e the transition probabilities the distribution and density functions of the event lifetimes and the initial distribution of the gsmp which is in contrast to the structural conditions needed for infinitesimal perturbation analysis we explicitly show that our conditions hold in many practical settings and in particular for large classes of queueing and reliability models one intermediate result we obtain in this study which is of independent value is to formally show that the random variable representing the number of occurring events in a gsmp in a finite time horizon has finite exponential moments in a neighborhood of zero

distributions of the two dimensional dct coefficients for images for a two dimensional discrete cosine transform dct image coding system there have been different assumptions concerning the distributions of the transform coefficients this paper presents results of distribution tests that indicate that for many images the statistics of the coefficients are best approximated by a gaussian distribution for the dc coefficient and a laplacian distribution for the other coefficients furthermore from a simulation of the dct coding system it is shown that the assumption that the coefficients are laplacian yields a higher actual output signal to noise ratio and a much better agreement between theory and simulation than the gaussian assumption

nonparametric multiscale blind estimation of intensity frequency dependent noise the camera calibration parameters and the image processing chain which generated a given image are generally not available to the receiver this happens for example with scanned photographs and for most jpeg images these images have undergone various nonlinear contrast changes and also linear and nonlinear filters to deal with remnant noise in such images we introduce a general nonparametric intensity and frequency dependent noise model we demonstrate by simulated and experiments with real images that this model which requires the estimation of more than 1000 parameters performs an efficient noise estimation the proposed noise model is a patch model its estimation can therefore be used as a preliminary step to any patch based denoising method our noise estimation method introduces several new tools for performing this complex estimation one of them is a new sparse patch distance function permitting to find noisy patches with similar underlying geometry a validation of the noise model and of its estimation method is obtained by comparing its results to ground truth noise curves for both raw and jpeg encoded images and by visual inspection of the denoising results of real images a fair comparison with the state of the art is also performed

optimization algorithms exploiting unitary constraints this paper presents novel algorithms that iteratively converge to a local minimum of a real valued function f x subject to the constraint that the columns of the complex valued matrix x are mutually orthogonal and have unit norm the algorithms are derived by reformulating the constrained optimization problem as an unconstrained one on a suitable manifold this significantly reduces the dimensionality of the optimization problem pertinent features of the proposed framework are illustrated by using the framework to derive an algorithm for computing the eigenvector associated with either the largest or the smallest eigenvalue of a hermitian matrix

matrix completion from noisy entries given a matrix m of low rank we consider the problem of reconstructing it from noisy observations of a small random subset of its entries the problem arises in a variety of applications from collaborative filtering the netflix problem to structure from motion and positioning we study a low complexity algorithm introduced by keshavan montanari and oh 2010 based on a combination of spectral techniques and manifold optimization that we call here optspace we prove performance guarantees that are order optimal in a number of circumstances

projection like retractions on matrix manifolds this paper deals with constructing retractions a key step when applying optimization algorithms on matrix manifolds for submanifolds of euclidean spaces we show that the operation consisting of taking a tangent step in the embedding euclidean space followed by a projection onto the submanifold is a retraction we also show that the operation remains a retraction if the projection is generalized to a projection like procedure that consists of coming back to the submanifold along admissible directions and we give a sufficient condition on the admissible directions for the generated retraction to be second order this theory offers a framework in which previously proposed retractions can be analyzed as well as a toolbox for constructing new ones illustrations are given for projection like procedures on some specific manifolds for which we have an explicit easy to compute expression

matrix completion from a few entries let m be an n n matrix of rank r and assume that a uniformly random subset e of its entries is observed we describe an efficient algorithm which we call optspace that reconstructs m from e o rn observed entries with relative root mean square error 1 2 rmse c nr e 1 2 with probability larger than 1 1 n3 further if r o 1 and m is sufficiently unstructured then optspace reconstructs it exactly from e o n log n entries with probability larger than 1 1 n3 this settles in the case of bounded rank a question left open by candes and recht and improves over the guarantees for their reconstruction algorithm the complexity of our algorithm is o e r log n which opens the way to its use for massive data sets in the process of proving these statements we obtain a generalization of a celebrated result by friedman kahn szemeredi and feige ofek on the spectrum of sparse random matrices

phase transitions in semidefinite relaxations statistical inference problems arising within signal processing data mining and machine learning naturally give rise to hard combinatorial optimization problems these problems become intractable when the dimensionality of the data is large as is often the case for modern datasets a popular idea is to construct convex relaxations of these combinatorial problems which can be solved efficiently for large scale datasets semidefinite programming sdp relaxations are among the most powerful methods in this family and are surprisingly well suited for a broad range of problems where data take the form of matrices or graphs it has been observed several times that when the statistical noise is small enough sdp relaxations correctly detect the underlying combinatorial structures in this paper we develop asymptotic predictions for several detection thresholds as well as for the estimation error above these thresholds we study some classical sdp relaxations for statistical problems motivated by graph synchronization and community detection in networks we map these optimization problems to statistical mechanics models with vector spins and use nonrigorous techniques from statistical mechanics to characterize the corresponding phase transitions our results clarify the effectiveness of sdp relaxations in solving high dimensional statistical problems

moment inequalities for sums of random matrices and their applications in optimization in this paper we consider various moment inequalities for sums of random matrices which are well studied in the functional analysis and probability theory literature and demonstrate how they can be used to obtain the best known performance guarantees for several problems in optimization first we show that the validity of a recent conjecture of nemirovski is actually a direct consequence of the so called non commutative khintchine s inequality in functional analysis using this result we show that an sdp based algorithm of nemirovski which is developed for solving a class of quadratic optimization problems with orthogonality constraints has a logarithmic approximation guarantee this improves upon the polynomial approximation guarantee established earlier by nemirovski furthermore we obtain improved safe tractable approximations of a certain class of chance constrained linear matrix inequalities secondly we consider a recent result of delage and ye on the so called data driven distributionally robust stochastic programming problem one of the assumptions in the delage ye result is that the underlying probability distribution has bounded support however using a suitable moment inequality we show that the result in fact holds for a much larger class of probability distributions given the close connection between the behavior of sums of random matrices and the theoretical properties of various optimization problems we expect that the moment inequalities discussed in this paper will find further applications in optimization

noisy matrix completion understanding statistical guarantees for convex relaxation via nonconvex optimization this paper studies noisy low rank matrix completion given partial and noisy entries of a large low rank matrix the goal is to estimate the underlying matrix faithfully and efficiently arguably o

factoring polynomials with rational coefficients in this paper we present a polynomial time algorithm to solve the following problem given a non zero polynomial fe q x in one variable with rational coefficients find the decomposition of f into irreducible factors in q x it is well known that this is equivalent to factoring primitive polynomials fez x into irreducible factors in z x here we call f z x primitive if the greatest common divisor of its coefficients the content of f is 1 our algorithm performs well in practice cf 8 its running time measured in bit operations is o nl2 n9 log fd3

noise level estimation for natural images based on scale invariant kurtosis and piecewise stationarity noise level estimation is crucial in many image processing applications such as blind image denoising in this paper we propose a novel noise level estimation approach for natural images by jointly exploiting the piecewise stationarity and a regular property of the kurtosis in bandpass domains we design a k means based algorithm to adaptively partition an image into a series of non overlapping regions each of whose clean versions is assumed to be associated with a constant but unknown kurtosis throughout scales the noise level estimation is then cast into a problem to optimally fit this new kurtosis model in addition we develop a rectification scheme to further reduce the estimation bias through noise injection mechanism extensive experimental results show that our method can reliably estimate the noise level for a variety of noise types and outperforms some state of the art techniques especially for non gaussian noises

structural conditions for perturbation analysis derivative estimation finite time performance indices in recent years there has been a surge of research into methods for estimating derivatives of performance measures from sample paths of stochastic systems in the case of queueing systems typical performance measures are mean queue lengths throughputs etc and the derivatives estimated are with respect to system parameters such as parameters of service and interarrival time distributions derivative estimates potentially offer a general means of optimizing performance and are useful in sensitivity analysis this paper concerns one approach to derivative estimation known as infinitesimal perturbation analysis we first develop a general framework for these types of estimates then give simple sufficient conditions for them to be unbiased the key to our results is identifying conditions under which certain finite horizon performance measures are almost surely continuous functions of the parameter of differentiation throughout an interval the sufficient conditions we introduce are formulated in the setting of generalized semi markov processes but translate into readily verifiable conditions for queueing systems these results substantially extend the domain of problems in which infinitesimal perturbation analysis is provably applicable

likelihood ratio sensitivity analysis for markovian models of highly dependable systems this paper discusses the application of the likelihood ratio gradient estimator to simulations of large markovian models of highly dependable systems extensive empirical work as well as some mathematical analysis of small dependability models suggests that in this model setting the gradient estimators are not significantly more noisy than the estimates of the performance measures themselves the paper also discusses implementation issues associated with likelihood ratio gradient estimation as well as some theoretical complements associated with application of the technique to continuous time markov chains

monotonicity in generalized semi markov processes we establish stochastic monotonicity of the event epoch sequences of generalized semi markov processes through the structure of the generalized semi markov schemes on which they are based our main condition states roughly that the occurrence of more events in the short run never leads to the activation of less events in the long run we consider monotonicity with respect to two types of inputs clock times which translate to e g service and interarrival times in queueing systems and structural parameters which translate to e g buffer size number of servers and job population for the second type of comparison we replace a structural change with an equivalent change in clock times to reduce the comparison to one of the first type when applied to queueing systems our results yield new comparisons and also unify several existing results previously established using special properties of individual systems

closest point search in lattices in this semitutorial paper a comprehensive survey of closest point search methods for lattices without a regular structure is presented the existing search strategies are described in a unified framework and differences between them are elucidated an efficient closest point search algorithm based on the schnorr euchner 1995 variation of the pohst 1981 method is implemented given an arbitrary point x spl isin spl ropf sup m and a generator matrix for a lattice spl lambda the algorithm computes the point of spl lambda that is closest to x the algorithm is shown to be substantially faster than other known methods by means of a theoretical comparison with the kannan 1983 1987 algorithm and an experimental comparison with the pohst 1981 algorithm and its variants such as the viterbo boutros see ibid vol 45 p 1639 42 1999 decoder modifications of the algorithm are developed to solve a number of related search problems for lattices such as finding a shortest vector determining the kissing number computing the voronoi 1908 relevant vectors and finding a korkine zolotareff 1873 reduced basis

a feasible method for optimization with orthogonality constraints minimization with orthogonality constraints e g x top x i and or spherical constraints e g vert x vert 2 1 has wide applications in polynomial optimization combinatorial optimization eigenvalue problems sparse pca p harmonic flows 1 bit compressive sensing matrix rank minimization etc these problems are difficult because the constraints are not only non convex but numerically expensive to preserve during iterations to deal with these difficulties we apply the cayley transform a crank nicolson like update scheme to preserve the constraints and based on it develop curvilinear search algorithms with lower flops compared to those based on projections and geodesics the efficiency of the proposed algorithms is demonstrated on a variety of test problems in particular for the maxcut problem it exactly solves a decomposition formulation for the sdp relaxation for polynomial optimization nearest correlation matrix estimation and extreme eigenvalue problems the proposed algorithms run very fast and return solutions no worse than those from their state of the art algorithms for the quadratic assignment problem a gap 0 842 to the best known solution on the largest problem tai256c in qaplib can be reached in 5 min on a typical laptop

a unified approach to error bounds for structured convex optimization problems error bounds which refer to inequalities that bound the distance of vectors in a test set to a given set by a residual function have proven to be extremely useful in analyzing the convergence rates of a host of iterative methods for solving optimization problems in this paper we present a new framework for establishing error bounds for a class of structured convex optimization problems in which the objective function is the sum of a smooth convex function and a general closed proper convex function such a class encapsulates not only fairly general constrained minimization problems but also various regularized loss minimization formulations in machine learning signal processing and statistics using our framework we show that a number of existing error bound results can be recovered in a unified and transparent manner to further demonstrate the power of our framework we apply it to a class of nuclear norm regularized loss minimization problems and establish a new error bound for this class under a strict complementarity type regularity condition we then complement this result by constructing an example to show that the said error bound could fail to hold without the regularity condition we believe that our approach will find further applications in the study of error bounds for structured convex optimization problems

convergence properties of infinitesimal perturbation analysis infinitesimal perturbation analysis ipa is a method for computing a sample path derivative with respect to an input parameter in a discrete event simulation the ipa algorithm is based on the fact that for certain parameters and any realization of a simulation the change in parameter can be made small enough so that only the times of events get shifted but their order does not change this paper considers the convergence properties of the ipa sample path derivatives in particular the question of when an ipa estimate converges to the derivative of a steady state performance measure is studied necessary and sufficient conditions for this convergence are derived for a class of regenerative processes although these conditions are not guaranteed to be satisfied in general they are satisfied for the mean stationary response time in the m g 1 queue a necessary condition for multiple ipa estimates to simultaneously converge to the derivatives of steady state throughputs in a queueing network is determined the implications of this necessary condition are that except in special cases the original ipa algorithm cannot be used to consistently estimate steady state throughput derivatives in queueing networks with multiple types of customers state dependent routing or blocking numerical studies on ipa convergence properties are also presented

clustering partially observed graphs via convex optimization this paper considers the problem of clustering a partially observed unweighted graph i e one where for some node pairs we know there is an edge between them for some others we know there is no edge and for the remaining we do not know whether or not there is an edge we want to organize the nodes into disjoint clusters so that there is relatively dense observed connectivity within clusters and sparse across clusters we take a novel yet natural approach to this problem by focusing on finding the clustering that minimizes the number of disagreements i e the sum of the number of observed missing edges within clusters and observed present edges across clusters our algorithm uses convex optimization its basis is a reduction of disagreement minimization to the problem of recovering an unknown low rank matrix and an unknown sparse matrix from their partially observed sum we evaluate the performance of our algorithm on the classical planted partition stochastic block model our main theorem provides sufficient conditions for the success of our algorithm as a function of the minimum cluster size edge density and observation probability in particular the results characterize the tradeoff between the observation probability and the edge density gap when there are a constant number of clusters of equal size our results are optimal up to logarithmic factors

community detection in sparse networks via grothendieck s inequality we present a simple and flexible method to prove consistency of semidefinite optimization problems on random graphs the method is based on grothendieck s inequality unlike the previous uses of this inequality that lead to constant relative accuracy we achieve any given relative accuracy by leveraging randomness we illustrate the method with the problem of community detection in sparse networks those with bounded average degrees we demonstrate that even in this regime various simple and natural semidefinite programs can be used to recover the community structure up to an arbitrarily small fraction of misclassified vertices the method is general it can be applied to a variety of stochastic models of networks and semidefinite programs

complete dictionary recovery over the sphere ii recovery by riemannian trust region method we consider the problem of recovering a complete i e square and invertible matrix a 0 from y in mathbb r n times p with y boldsymbol a 0 x 0 provided x 0 is sufficiently sparse this recovery problem is central to theoretical understanding of dictionary learning which seeks a sparse representation for a collection of input signals and finds numerous applications in modern signal processing and machine learning we give the first efficient algorithm that provably recovers a 0 when x 0 has o left n right nonzeros per column under suitable probability model for x 0 our algorithmic pipeline centers around solving a certain nonconvex optimization problem with a spherical constraint and hence is naturally phrased in the language of manifold optimization in a companion paper we have showed that with high probability our nonconvex formulation has no spurious local minimizers and around any saddle point the objective function has a negative directional curvature in this paper we take advantage of the particular geometric structure and describe a riemannian trust region algorithm that provably converges to a local minimizer with from arbitrary initializations such minimizers give excellent approximations to the rows of x 0 the rows are then recovered by a linear programming rounding and deflation

phase retrieval using alternating minimization phase retrieval problems involve solving linear equations but with missing sign or phase for complex numbers information more than four decades after it was first proposed the seminal error reduction algorithm of gerchberg and saxton and fienup is still the popular choice for solving many variants of this problem the algorithm is based on alternating minimization i e it alternates between estimating the missing phase information and the candidate solution despite its wide usage in practice no global convergence guarantees for this algorithm are known in this paper we show that a resampling variant of this approach converges geometrically to the solution of one such problem finding a vector bf x from bf y bf a where bf y vert bf a t bf x vert and vert bf z vert denotes a vector of element wise magnitudes of bf z under the assumption that bf a is gaussian empirically we demonstrate that alternating minimization performs similar to recently proposed convex techniques for this problem which are based on lifting to a convex matrix problem in sample complexity and robustness to noise however it is much more efficient and can scale to large problems analytically for a resampling version of alternating minimization we show geometric convergence to the solution and sample complexity that is off by log factors from obvious lower bounds we also establish close to optimal scaling for the case when the unknown vector is sparse our work represents the first theoretical guarantee for alternating minimization albeit with resampling for any variant of phase retrieval problems in the non convex setting

a mathematical analysis of the dct coefficient distributions for images over the past two decades there have been various studies on the distributions of the dct coefficients for images however they have concentrated only on fitting the empirical data from some standard pictures with a variety of well known statistical distributions and then comparing their goodness of fit the laplacian distribution is the dominant choice balancing simplicity of the model and fidelity to the empirical data yet to the best of our knowledge there has been no mathematical justification as to what gives rise to this distribution we offer a rigorous mathematical analysis using a doubly stochastic model of the images which not only provides the theoretical explanations necessary but also leads to insights about various other observations from the literature this model also allows us to investigate how certain changes in the image statistics could affect the dct coefficient distributions

optimal rates of convergence for noisy sparse phase retrieval via thresholded wirtinger flow this paper considers the noisy sparse phase retrieval problem recovering a sparse signal mathbf x in mathbb r p from noisy quadratic measurements y j mathbf a j mathbf x 2 varepsilon j j 1 ldots m with independent sub exponential noise varepsilon j the goals are to understand the effect of the sparsity of mathbf x on the estimation precision and to construct a computationally feasible estimator to achieve the optimal rates adaptively inspired by the wirtinger flow ieee trans inform theory61 2015 1985 2007 proposed for non sparse and noiseless phase retrieval a novel thresholded gradient descent algorithm is proposed and it is shown to adaptively achieve the minimax optimal rates of convergence over a wide range of sparsity levels when the mathbf a j s are independent standard gaussian random vectors provided that the sample size is sufficiently large compared to the sparsity of mathbf x

new error bounds and their applications to convergence analysis of iterative algorithms we present two new error bounds for optimization problems over a convex set whose objective function f is either semianalytic or strictly convex with 1 we then apply these error bounds to analyze the rate of convergence of a wide class of iterative descent algorithms for the aforementioned optimization problem our analysis shows that the function sequence f xk converges at least at the sublinear rate of k for some positive constant where k is the iteration index moreover the distances from the iterate sequence xk to the set of stationary points of the optimization problem converge to zero at least sublinearly

some guidelines and guarantees for common random numbers common random numbers crn is a widely used technique for reducing variance in comparing stochastic systems through simulation its popularity derives from its intuitive appeal and ease of implementation however though crn has been observed to work well with a broad range of models the class of systems for which it is provably advantageous has remained rather limited this paper has two purposes we first discuss the effectiveness and optimality of crn in a general setting stressing the roles played by monotonicity and continuity properties we then present specific new classes of systems and comparisons for which crn is beneficial and even optimal our conclusions for these systems are largely consistent with simulation practice and lend further theoretical support to folklore our results differ from those of previous analyses primarily because we put conditions on the timing of events rather than the sequence of states in a discrete event simulation we formulate our results in three settings corresponding to three applications of crn distributional comparisons structural comparisons and sensitivity analysis in each case we make use of conditions that simultaneously ensure monotonicity and continuity in the timing of events these properties are established through explicit recursions for event epochs in terms of increasing continuous functions

stochastic gradient descent on riemannian manifolds stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise in this paper we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a riemannian manifold we prove that as in the euclidian case the gradient descent algorithm converges to a critical point of the cost function the algorithm has numerous potential applications and is illustrated here by four examples in particular a novel gossip algorithm on the set of covariance matrices is derived and tested numerically

generalized semi markov processes antimatroid structure and second order properties a generalized semi markov scheme models the structure of a discrete event system such as a network of queues by studying combinatorial and geometric representations of schemes we find conditions for second order properties convexity concavity sub supermodularity of their event epochs and event counting processes a scheme generates a language of feasible strings of events we show that monotonicity of the event epochs is equivalent to this language forming an antimatroid with repetition this connection gives rise to a rich combinatorial structure and serves as a starting point for other properties for example by strengthening the antimatroid condition we give several equivalent characterizations of the convexity of event epochs within a scheme all of these correspond in slightly different ways to making a certain score space a lattice to closing an ordinary antimatroid under intersections we also establish second order properties across schemes tied together through a synchronization mechanism a geometric view based on the score space facilitates verification of these properties in certain queueing systems

recompression of jpeg images by requantization we report a novel heuristic for requantizing jpeg images the resulting images are generally smaller and often have improved perceptual image quality over a blind requantization approach that is one that does not consider the properties of the quantization matrices the heuristic is supported by a detailed mathematical treatment which incorporates the well known laplacian distribution of the ac discrete cosine transform dct coefficients with an analysis of the error introduced by requantization we note that the technique is applicable to any image compression method which employs discrete cosine transforms and quantization

jpeg compression history estimation for color images we routinely encounter digital color images that were previously compressed using the joint photographic experts group jpeg standard en route to the image s current representation the previous jpeg compression s various settings termed its jpeg compression history ch are often discarded after the jpeg decompression step given a jpeg decompressed color image this paper aims to estimate its lost jpeg ch we observe that the previous jpeg compression s quantization step introduces a lattice structure in the discrete cosine transform dct domain this paper proposes two approaches that exploit this structure to solve the jpeg compression history estimation chest problem first we design a statistical dictionary based chest algorithm that tests the various chs in a dictionary and selects the maximum a posteriori estimate second for cases where the dct coefficients closely conform to a 3 d parallelepiped lattice we design a blind lattice based chest algorithm the blind algorithm exploits the fact that the jpeg ch is encoded in the nearly orthogonal bases for the 3 d lattice and employs novel lattice algorithms and recent results on nearly orthogonal lattice bases to estimate the ch both algorithms provide robust jpeg chest performance in practice simulations demonstrate that jpeg chest can be useful in jpeg recompression the estimated ch allows us to recompress a jpeg decompressed image with minimal distortion large signal to noise ratio and simultaneously achieve a small file size

f divergence inequalities this paper develops systematic approaches to obtain f divergence inequalities dealing with pairs of probability measures defined on arbitrary alphabets functional domination is one such approach where special emphasis is placed on finding the best possible constant upper bounding a ratio of f divergences another approach used for the derivation of bounds among f divergences relies on moment inequalities and the logarithmic convexity property which results in tight bounds on the relative entropy and bhattacharyya distance in terms of chi 2 divergences a rich variety of bounds are shown to hold under boundedness assumptions on the relative information special attention is devoted to the total variation distance and its relation to the relative information and relative entropy including reverse pinsker inequalities as well as on the e gamma divergence which generalizes the total variation distance pinsker s inequality is extended for this type of f divergence a result which leads to an inequality linking the relative entropy and relative information spectrum integral expressions of the renyi divergence in terms of the relative information spectrum are derived leading to bounds on the renyi divergence in terms of either the variational distance or relative entropy

low rank matrix recovery from errors and erasures this paper considers the recovery of a low rank matrix from an observed version that simultaneously contains both 1 erasures most entries are not observed and 2 errors values at a constant fraction of unknown locations are arbitrarily corrupted we provide a new unified performance guarantee on when minimizing nuclear norm plus l1 norm succeeds in exact recovery our result allows for the simultaneous presence of random and deterministic components in both the error and erasure patterns by specializing this one single result in different ways we recover up to poly log factors as corollaries all the existing results in exact matrix completion and exact sparse and low rank matrix decomposition our unified result also provides the first guarantees for 1 recovery when we observe a vanishing fraction of entries of a corrupted matrix and 2 deterministic matrix completion

steepest descent algorithms for optimization under unitary matrix constraint in many engineering applications we deal with constrained optimization problems with respect to complex valued matrices this paper proposes a riemannian geometry approach for optimization of a real valued cost function t of complex valued matrix argument w under the constraint that w is an n times n unitary matrix we derive steepest descent sd algorithms on the lie group of unitary matrices u n the proposed algorithms move towards the optimum along the geodesics but other alternatives are also considered we also address the computational complexity and the numerical stability issues considering both the geodesic and the nongeodesic sd algorithms armijo step size 1 adaptation rule is used similarly to 2 but with reduced complexity the theoretical results are validated by computer simulations the proposed algorithms are applied to blind source separation in mimo systems by using the joint diagonalization approach 3 we show that the proposed algorithms outperform other widely used algorithms

information recovery from pairwise measurements this paper is concerned with jointly recovering n node variables left x i right 1 leq i leq n from a collection of pairwise difference measurements imagine we acquire a few observations taking the form of x i x j the observation pattern is represented by a measurement graph mathcal g with an edge set mathcal e such that x i x j is observed if and only if i j in mathcal e to account for noisy measurements in a general manner we model the data acquisition process by a set of channels with given input output transition measures employing information theoretic tools applied to channel decoding problems we develop a unified framework to characterize the fundamental recovery criterion which accommodates general graph structures alphabet sizes and channel transition measures in particular our results isolate a family of minimum channel divergence measures to characterize the degree of measurement corruption which together with the size of the minimum cut of mathcal g dictates the feasibility of exact information recovery for various homogeneous graphs the recovery condition depends almost only on the edge sparsity of the measurement graph irrespective of other graphical metrics alternatively the minimum sample complexity required for these graphs scales like n log n mathsf hel 1 2 min for certain information metric mathsf hel 1 2 min defined in the main text as long as the alphabet size is not super polynomial in n we apply our general theory to three concrete applications including the stochastic block model the random corruption model and the haplotype assembly problem our theory leads to orderwise tight recovery conditions for all these scenarios

solving systems of random quadratic equations via truncated amplitude flow this paper presents a new algorithm termed truncated amplitude flow taf to recover an unknown vector x from a system of quadratic equations of the form y i langle a i x rangle 2 where a i s are given random measurement vectors this problem is known to be np hard in general we prove that as soon as the number of equations is on the order of the number of unknowns taf recovers the solution exactly up to a global unimodular constant with high probability and complexity growing linearly with both the number of unknowns and the number of equations our taf approach adapts the amplitude based empirical loss function and proceeds in two stages in the first stage we introduce an orthogonality promoting initialization that can be obtained with a few power iterations stage two refines the initial estimate by successive updates of scalable truncated generalized gradient iterations which are able to handle the rather challenging nonconvex and nonsmooth amplitude based objective function in particular when vectors x and a i s are real valued our gradient truncation rule provably eliminates erroneously estimated signs with high probability to markedly improve upon its untruncated version numerical tests using synthetic data and real images demonstrate that our initialization returns more accurate and robust estimates relative to spectral initializations furthermore even under the same initialization the proposed amplitude based refinement outperforms existing wirtinger flow variants corroborating the superior performance of taf over state of the art algorithms

complete dictionary recovery over the sphere i overview and the geometric picture we consider the problem of recovering a complete i e square and invertible matrix a 0 from y in mathbb r n times p with y a 0 x 0 provided x 0 is sufficiently sparse this recovery problem is central to theoretical understanding of dictionary learning which seeks a sparse representation for a collection of input signals and finds numerous applications in modern signal processing and machine learning we give the first efficient algorithm that provably recovers a 0 when x 0 has o left n right nonzeros per column under suitable probability model for x 0 in contrast prior results based on efficient algorithms either only guarantee recovery when x 0 has o sqrt n zeros per column or require multiple rounds of semidefinite programming relaxation to work when x 0 has o n nonzeros per column our algorithmic pipeline centers around solving a certain nonconvex optimization problem with a spherical constraint in this paper we provide a geometric characterization of the objective landscape in particular we show that the problem is highly structured with high probability 1 there are no spurious local minimizers and 2 around all saddle points the objective has a negative directional curvature this distinctive structure makes the problem amenable to efficient optimization algorithms in a companion paper we design a second order trust region algorithm over the sphere that provably converges to a local minimizer from arbitrary initializations despite the presence of saddle points

statistical model of jpeg noises and its application in quantization step estimation in this paper we present a statistical analysis of jpeg noises including the quantization noise and the rounding noise during a jpeg compression cycle the jpeg noises in the first compression cycle have been well studied however so far less attention has been paid on the statistical model of jpeg noises in higher compression cycles our analysis reveals that the noise distributions in higher compression cycles are different from those in the first compression cycle and they are dependent on the quantization parameters used between two successive cycles to demonstrate the benefits from the analysis we apply the statistical model in jpeg quantization step estimation we construct a sufficient statistic by exploiting the derived noise distributions and justify that the statistic has several special properties to reveal the ground truth quantization step experimental results demonstrate that the proposed estimator can uncover jpeg compression history with a satisfactory performance

achieving exact cluster recovery threshold via semidefinite programming the binary symmetric stochastic block model deals with a random graph of n vertices partitioned into two equal sized clusters such that each pair of vertices is independently connected with probability p within clusters and q across clusters in the asymptotic regime of p a log n n and q b log n n for fixed a b and n to infty we show that the semidefinite programming relaxation of the maximum likelihood estimator achieves the optimal threshold for exactly recovering the partition from the graph with probability tending to one resolving a conjecture of abbe et al furthermore we show that the semidefinite programming relaxation also achieves the optimal recovery threshold in the planted dense subgraph model containing a single cluster of size proportional to n

sensitivity analysis for simulations via likelihood ratios we present a simple method of estimating the sensitivity of quantities obtained from simulation with respect to a class of parameters here sensitivity means the derivative of an expectation with respect to a parameter the class of parameters includes for example poisson rates discrete probabilities and the mean and variance of a normal distribution the method is extremely well suited to regenerative simulation and can be implemented on extant simulations with little effort increase in running time or memory requirements it is based on some change of measure formulas derived from likelihood ratios in addition to the theorems that underly the technique we present some numerical examples

gradient descent with random initialization fast global convergence for nonconvex phase retrieval this paper considers the problem of solving systems of quadratic equations namely recovering an object of interest varvec x natural in mathbb r n from m quadratic equations samples y i varvec a i top varvec x natural 2 1 le i le m this problem also dubbed as phase retrieval spans multiple domains including physical sciences and machine learning we investigate the efficacy of gradient descent or wirtinger flow designed for the nonconvex least squares problem we prove that under gaussian designs gradient descent when randomly initialized yields an epsilon accurate solution in o big log n log 1 epsilon big iterations given nearly minimal samples thus achieving near optimal computational and sample complexities at once this provides the first global convergence guarantee concerning vanilla gradient descent for phase retrieval without the need of i carefully designed initialization ii sample splitting or iii sophisticated saddle point escaping schemes all of these are achieved by exploiting the statistical models in analyzing optimization algorithms via a leave one out approach that enables the decoupling of certain statistical dependency between the gradient descent iterates and the data

sharp nonasymptotic bounds on the norm of random matrices with independent entries we obtain nonasymptotic bounds on the spectral norm of random matrices with independent entries that improve significantly on earlier results if xx is the n nn n symmetric matrix with xij n 0 b2ij xij n 0 bij2 we show that this bound is optimal in the sense that a matching lower bound holds under mild assumptions and the constants are sufficiently sharp that we can often capture the precise edge of the spectrum analogous results are obtained for rectangular matrices and for more general sub gaussian or heavy tailed distributions of the entries and we derive tail bounds in addition to bounds on the expected norm the proofs are based on a combination of the moment method and geometric functional analysis techniques as an application we show that our bounds immediately yield the correct phase transition behavior of the spectral edge of random band matrices and of sparse wigner matrices we also recover a result of seginer on the norm of rademacher matrices

nonconvex phase synchronization we estimate n phases angles from noisy pairwise relative phase measurements the task is modeled as a nonconvex least squares optimization problem it was recently shown that this problem can be solved in polynomial time via convex relaxation under some conditions on the noise in this paper under similar but more restrictive conditions we show that a modified version of the power method converges to the global optimum this is simpler and empirically faster than convex approaches empirically they both succeed in the same regime further analysis shows that in the same noise regime as previously studied second order necessary optimality conditions for this quadratically constrained quadratic program are also sufficient despite nonconvexity

a theory for multiresolution signal decomposition the wavelet representation multiresolution representations are effective for analyzing the information content of images the properties of the operator which approximates a signal at a given resolution were studied it is shown that the difference of information between the approximation of a signal at the resolutions 2 sup j 1 and 2 sup j where j is an integer can be extracted by decomposing this signal on a wavelet orthonormal basis of l sup 2 r sup n the vector space of measurable square integrable n dimensional functions in l sup 2 r a wavelet orthonormal basis is a family of functions which is built by dilating and translating a unique function psi x this decomposition defines an orthogonal multiresolution representation called a wavelet representation it is computed with a pyramidal algorithm based on convolutions with quadrature mirror filters wavelet representation lies between the spatial and fourier domains for images the wavelet representation differentiates several spatial orientations the application of this representation to data compression in image coding texture discrimination and fractal analysis is discussed