low rank doubly stochastic matrix decomposition for cluster analysis cluster analysis by nonnegative low rank approximations has experienced a remarkable progress in the past decade however the majority of such approximation approaches are still restricted to nonnegative matrix factorization nmf and suffer from the following two drawbacks 1 they are unable to produce balanced partitions for large scale manifold data which are common in real world clustering tasks 2 most existing nmf type clustering methods cannot automatically determine the number of clusters we propose a new low rank learning method to address these two problems which is beyond matrix factorization our method approximately decomposes a sparse input similarity in a normalized way and its objective can be used to learn both cluster assignments and the number of clusters for efficient optimization we use a relaxed formulation based on data cluster data random walk which is also shown to be equivalent to low rank factorization of the doubly stochastically normalized cluster incidence matrix the probabilistic cluster assignments can thus be learned with a multiplicative majorization minimization algorithm experimental results show that the new method is more accurate both in terms of clustering large scale manifold data sets and of selecting the number of clusters

nonnegative matrix factorization for spectral data analysis data analysis is pervasive throughout business engineering and science very often the data to be analyzed is nonnegative and it is often preferable to take this constraint into account in the analysis process here we are concerned with the application of analyzing data obtained using astronomical spectrometers which provide spectral data which is inherently nonnegative the identification and classification of space objects that cannot be imaged in the normal way with telescopes is an important but difficult problem for tracking thousands of objects including satellites rocket bodies debris and asteroids in orbit around the earth in this paper we develop an effective nonnegative matrix factorization algorithm with novel smoothness constraints for unmixing spectral reflectance data for space object identification and classification purposes promising numerical results are presented using laboratory and simulated datasets

learning the information divergence information divergence that measures the difference between two nonnegative matrices or tensors has found its use in a variety of machine learning problems examples are nonnegative matrix tensor factorization stochastic neighbor embedding topic models and bayesian network optimization the success of such a learning task depends heavily on a suitable divergence a large variety of divergences have been suggested and analyzed but very few results are available for an objective choice of the optimal divergence for a given task here we present a framework that facilitates automatic selection of the best divergence among a given family based on standard maximum likelihood estimation we first propose an approximated tweedie distribution for the beta divergence family selecting the best beta then becomes a machine learning problem solved by maximum likelihood next we reformulate alpha divergence in terms of beta divergence which enables automatic selection of alpha by maximum likelihood with reuse of the learning principle for beta divergence furthermore we show the connections between gamma and beta divergences as well as renyi and alpha divergences such that our automatic selection framework is extended to non separable divergences experiments on both synthetic and real world data demonstrate that our method can quite accurately select information divergence across different learning problems and various divergence families

convex and semi nonnegative matrix factorizations we present several new variations on the theme of nonnegative matrix factorization nmf considering factorizations of the form x fgt we focus on algorithms in which g is restricted to containing nonnegative entries but allowing the data matrix x to have mixed signs thus extending the applicable range of nmf methods we also consider algorithms in which the basis vectors of f are constrained to be convex combinations of the data points this is used for a kernel extension of nmf we provide algorithms for computing these new factorizations and we provide supporting theoretical analysis we also analyze the relationships between our algorithms and clustering algorithms and consider the implications for sparseness of solutions finally we present experimental results that explore the properties of these new methods

a second order method for fitting the canonical polyadic decomposition with non least squares cost the canonical polyadic decomposition cpd can be used to extract meaningful components from a tensor most existing optimization methods for fitting the cpd use as cost function the least squares distance between the tensor and its cpd while the minimum of this cost function coincides with the maximum likelihood estimator for data with additive i i d gaussian distributed noise for other noise distributions better suited cost functions exist for such cost functions first order gradient based optimization methods have been proposed however approximate second order methods which additionally use information from the hessian of the cost function to achieve faster convergence are still largely unexplored in this paper we generalize the gauss newton nonlinear least squares algorithm to twice differentiable entry wise cost functions the low rank structure of the problem is exploited to keep the computational cost low as a special case beta divergence cost functions are examined we show that quadratic convergence can be obtained close to the solution with a reasonable extra cost in memory and computation time making the proposed method particularly useful when high accuracy of the decomposition is desired

solving low density subset sum problems the subset sum problem is to decide whether or not the 0 l integer programming problem s n i l a i x i m i x i 0 or 1 has a solution where the a i and m are given positive integers this problem is np complete and the difficulty of solving it is the basis of public key cryptosystems of knapsack type an algorithm is proposed that searches for a solution when given an instance of the subset sum problem this algorithm always halts in polynomial time but does not always find a solution when one exists it converts the problem to one of finding a particular short vector v in a lattice and then uses a lattice basis reduction algorithm due to a k lenstra h w lenstra jr and l lovasz to attempt to find v the performance of the proposed algorithm is analyzed let the density d of a subset sum problem be defined by d n log 2 max i a i then for almost all problems of density d d n it is proved that the lattice basis reduction algorithm locates v extensive computational tests of the algorithm suggest that it works for densities d d c n where d c n is a cutoff value that is substantially larger than 1 n this method gives a polynomial time attack on knapsack public key cryptosystems that can be expected to break them if they transmit information at rates below d c n as n

stability results for polyhedral complementarity problems in this work we study the multivalued complementarity problem for polyhedral multifunctions under homogeneity assumptions we employ an approach that consists in approximating the equivalent variational inequality formulation of the problem and studying the asymptotic behavior of sequences of solutions to these approximation problems to do this we employ results and the language of variational analysis the novelty of this approach lies in the fact that it allows us to obtain not only existence results but also stability ones we consider that our results can be used for developing numerical algorithms for solving multivalued complementarity problems

an iterative image space reconstruction algorthm suitable for volume ect the trend in the design of scanners for positron emission computed tomography has traditionally been to improve the transverse spatial resolution to several millimeters while maintaining relatively coarse axial resolution 1 2 cm several scanners are being built with fine sampling in the axial as well as transverse directions leading to the possibility of the true volume imaging the number of possible coincidence pairs in these scanners is quite large the usual methods of image reconstruction cannot handle these data without making approximations it is computationally most efficient to reduce the size of this large sparsely populated array by back projecting the coincidence data prior to reconstruction while analytic reconstruction techniques exist for back projected data an iterative algorithm may be necessary for those cases where the point spread function is spatially variant a modification of the maximum likelihood algorithm is proposed to reconstruct these back projected data the method the iterative image space reconstruction algorithm isra is able to reconstruct data from a scanner with a spatially variant point spread function in less time than other proposed algorithms results are presented for single slice data simulated and actual from the penn pet scanner

linearly perturbed polyhedral normal cone mappings and applications abstract under a mild regularity assumption we derive an exact formula for the frechet coderivative and some estimates for the mordukhovich coderivative of the normal cone mappings of perturbed polyhedra in reflexive banach spaces our focus point is a positive linear independence condition which is a relaxed form of the linear independence condition employed recently by henrion et al 2010 1 and nam 2010 3 the formulae obtained allow us to get new results on solution stability of affine variational inequalities under linear perturbations thus our paper develops some aspects of the work of henrion et al 2010 1 nam 2010 3 qui in press 12 and yao and yen 2009 6 7

on the iterative image space reconstruction algorthm for ect the isra of 1 is shown to be an iterative algorithm that aims to converge to the least squares estimates of emission densities convergence is established in the case where a unique least squares estimate exists that is elementwise strictly positive it is pointed out that in terms of asymptotic theory the resulting estimators are inferior to the maximum likelihood estimators for which the em algorithm is a computational procedure potential difficulties with the behavior of the isra are illustrated using very simple examples

a generalized divergence measure for nonnegative matrix factorization this letter presents a general parametric divergence measure the metric includes as special cases quadratic error and kullback leibler divergence a parametric generalization of the two different multiplicative update rules for nonnegative matrix factorization by lee and seung 2001 is shown to lead to locally optimal solutions of the nonnegative matrix factorization problem with this new cost function numeric simulations demonstrate that the new update rule may improve the quadratic distance convergence speed a proof of convergence is given that as in lee and seung uses an auxiliary function known from the expectation maximization theoretical framework

stability analysis of multiplicative update algorithms and application to nonnegative matrix factorization multiplicative update algorithms have proved to be a great success in solving optimization problems with nonnegativity constraints such as the famous nonnegative matrix factorization nmf and its many variants however despite several years of research on the topic the understanding of their convergence properties is still to be improved in this paper we show that lyapunov s stability theory provides a very enlightening viewpoint on the problem we prove the exponential or asymptotic stability of the solutions to general optimization problems with nonnegative constraints including the particular case of supervised nmf and finally study the more difficult case of unsupervised nmf the theoretical results presented in this paper are confirmed by numerical simulations involving both supervised and unsupervised nmf and the convergence speed of nmf multiplicative updates is investigated

the hardness of decoding linear codes with preprocessing the problem of maximum likelihood decoding of linear block codes is known to be hard the fact that the problem remains hard even if the code is known in advance and can be preprocessed for as long as desired in order to device a decoding algorithm is shown the hardness is based on the fact that existence of a polynomial time algorithm implies that the polynomial hierarchy collapses thus some linear block codes probably do not have an efficient decoder the proof is based on results in complexity theory that relate uniform and nonuniform complexity classes

a framework for mining signatures from event sequences and its applications in healthcare data this paper proposes a novel temporal knowledge representation and learning framework to perform large scale temporal signature mining of longitudinal heterogeneous event data the framework enables the representation extraction and mining of high order latent event structure and relationships within single and multiple event sequences the proposed knowledge representation maps the heterogeneous event sequences to a geometric image by encoding events as a structured spatial temporal shape process we present a doubly constrained convolutional sparse coding framework that learns interpretable and shift invariant latent temporal event signatures we show how to cope with the sparsity in the data as well as in the latent factor model by inducing a double sparsity constraint on the divergence to learn an overcomplete sparse latent factor model a novel stochastic optimization scheme performs large scale incremental learning of group specific temporal event signatures we validate the framework on synthetic data and on an electronic health record dataset

automatic relevance determination in nonnegative matrix factorization with the beta divergence this paper addresses the estimation of the latent dimensionality in nonnegative matrix factorization nmf with the divergence the divergence is a family of cost functions that includes the squared euclidean distance kullback leibler kl and itakura saito is divergences as special cases learning the model order is important as it is necessary to strike the right balance between data fidelity and overfitting we propose a bayesian model based on automatic relevance determination ard in which the columns of the dictionary matrix and the rows of the activation matrix are tied together through a common scale parameter in their prior a family of majorization minimization mm algorithms is proposed for maximum a posteriori map estimation a subset of scale parameters is driven to a small lower bound in the course of inference with the effect of pruning the corresponding spurious components we demonstrate the efficacy and robustness of our algorithms by performing extensive experiments on synthetic data the swimmer dataset a music decomposition example and a stock price prediction task

k approximating circuits in this paper we define and study the k approximating circuits a circuit accepting a given set of inputs a is k approximated by accepting inputs that lifter from one of a by at most k bits we show that the existence of polynomial size k approximating circuits depends on the relation between k and the number of inputs

identity based identification from algebraic coding theory cryptographic identification schemes allow a remote user to prove his her identity to a verifier who holds some public information of the user such as the user public key or identity most of the existing cryptographic identification schemes are based on number theoretic hard problems such as discrete log and factorization this paper focuses on the design and analysis of identity based identification ibi schemes based on algebraic coding theory we first revisit an existing code based ibi scheme which is derived by combining the courtois finiasz sendrier signature scheme and the stern zero knowledge identification scheme previous results have shown that this ibi scheme is secure under passive attacks in this paper we prove that the scheme in fact can resist active attacks however whether the scheme can be proven secure under concurrent attacks the most powerful attacks against identification schemes remains open in addition we show that it is difficult to apply the conventional or proof approach to this particular ibi scheme in order to obtain concurrent security we then construct a special or proof variant of this scheme and prove that the resulting ibi scheme is secure under concurrent attacks

lipschitz behavior of the robust regularization to minimize or upper bound the value of a function robustly we might instead minimize or upper bound the epsilon robust regularization defined as the map from a point to the maximum value of the function within an epsilon radius this regularization may be easy to compute convex quadratics lead to semidefinite representable regularizations for example and the spectral radius of a matrix leads to pseudospectral computations for favorable classes of functions we show that the robust regularization is lipschitz around any given point for all small epsilon 0 even if the original function is non lipschitz like the spectral radius one such favorable class consists of the semi algebraic functions such functions have graphs that are finite unions of sets defined by finitely many polynomial inequalities and are commonly encountered in applications

robust and efficient estimation by minimising a density power divergence a minimum divergence estimation method is developed for robust parameter estimation the proposed approach uses new density based divergences which unlike existing methods of this type such as minimum hellinger distance estimation avoid the use of nonparametric density estimation and associated complications such as bandwidth selection the proposed class of density power divergences is indexed by a single parameter which controls the trade off between robustness and efficiency the methodology affords a robust extension of maximum likelihood estimation for which 0 choices of near zero afford considerable robustness while retaining efficiency close to that of maximum likelihood

a new paradigm for public key identification the present paper investigates the possibility of designing zero knowledge identification schemes based on hard problems from coding theory zero knowledge proofs were introduced by goldwasser micali and rackoff 1985 their practical significance was soon demonstrated in the work of fiat and shamir 1986 who turned zero knowledge proofs of quadratic residuosity into efficient means of establishing user identities in the present paper we propose a new identification scheme based on error correcting codes which is zero knowledge and seems of practical value furthermore we describe several variants including one which has an identity based character the security of our schemes depends on the hardness of finding a word of given syndrome and prescribed small weight with respect to some randomly generated binary linear error correcting code this is of course not the first attempt to design a cryptographic scheme using tools from coding theory the difference is that identification protocols do not follow the public key paradigm based on trap door functions and described in the seminal diffie hellman paper 1976 rather they only require one way functions which opens the way to using in a rather direct manner simple combinatorial problems of the kind provided by coding theory the resulting schemes compare favorably to their number theoretic analogs

nonnegative matrix factorization with the itakura saito divergence with application to music analysis this letter presents theoretical algorithmic and experimental results about nonnegative matrix factorization nmf with the itakura saito is divergence we describe how is nmf is underlaid by a well defined statistical model of superimposed gaussian components and is equivalent to maximum likelihood estimation of variance parameters this setting can accommodate regularization constraints on the factors through bayesian priors in particular inverse gamma and gamma markov chain priors are considered in this work estimation can be carried out using a space alternating generalized expectation maximization sage algorithm this leads to a novel type of nmf algorithm whose convergence to a stationary point of the is cost function is guaranteed we also discuss the links between the is divergence and other cost functions used in nmf in particular the euclidean distance and the generalized kullback leibler kl divergence as such we describe how is nmf can also be performed using a gradient multiplicative algorithm a standard algorithm structure in nmf whose convergence is observed in practice though not proven finally we report a furnished experimental comparative study of euclidean nmf kl nmf and is nmf algorithms applied to the power spectrogram of a short piano sequence recorded in real conditions with various initializations and model orders then we show how is nmf can successfully be employed for denoising and upmix mono to stereo conversion of an original piece of early jazz music these experiments indicate that is nmf correctly captures the semantics of audio and is better suited to the representation of music signals than nmf with the usual euclidean and kl costs

on the convergence of multiplicative update algorithms for nonnegative matrix factorization nonnegative matrix factorization nmf is useful to find basis information of nonnegative data currently multiplicative updates are a simple and popular way to find the factorization however for the common nmf approach of minimizing the euclidean distance between approximate and true values no proof has shown that multiplicative updates converge to a stationary point of the nmf optimization problem stationarity is important as it is a necessary condition of a local minimum this paper discusses the difficulty of proving the convergence we propose slight modifications of existing updates and prove their convergence techniques invented in this paper may be applied to prove the convergence for other bound constrained optimization problems

variational inequalities over perturbed polyhedral convex sets this paper provides conditions for existence of a locally unique lipschitzian solution of a linear variational inequality posed over a polyhedral convex set in rn under perturbation of either or both of the constant term in the variational inequality and the right hand side of the system of linear constraints defining its feasible set conditions for perturbation of just the constant term are well known here we show that a suitable extension of those conditions suffices for the more general case in which the right hand side of the constraints varies also as a consequence we obtain existence uniqueness and lipschitz continuity properties of solutions of nonlinear variational inequalities posed over perturbed polyhedral convex sets

on the relation between the isra and the em algorithm for positron emission tomography the image space reconstruction algorithm isra was proposed as a modification of the expectation maximization em algorithm based on physical considerations for application in volume emission computered tomography as a consequence of this modification isra searches for least squares solutions instead of maximizing poisson likelihoods as the em algorithm it is shown that both algorithms may be obtained from a common mathematical framework this fact is used to extend isra for penalized likelihood estimates

a new efficient threshold ring signature scheme based on coding theory ring signatures were introduced by rivest shamir and tauman in 2001 these signatures allow a signer to anonymously authenticate a message on behalf of a group of his choice this concept was then extended by bresson stern and szydlo into t out of n threshold ring signatures in 2002 we propose in this article a generalization of stern s code based identification and signature scheme to design a practical t out of n threshold ring signature scheme the size of the resulting signatures is in o n and does not depend on t contrary to most of the existing protocols our scheme is existentially unforgeable under a chosen message attack in the random oracle model assuming the hardness of the minimum distance problem is unconditionally source hiding has a very short public key and has an overall complexity in o n this protocol is the first efficient code based ring signature scheme and the first code based threshold ring signature scheme moreover it has a better complexity than number theory based schemes which have a complexity in o nt this paper is an extended version of a paper published in the conference pqcrypto 2008 with complete proofs and definitions

a unified convergence analysis of the multiplicative update algorithm for regularized nonnegative matrix factorization the multiplicative update mu algorithm has been extensively used to estimate the basis and coefficient matrices in nonnegative matrix factorization nmf problems under a wide range of divergences and regularizers however theoretical convergence guarantees have only been derived for a few special divergences without regularization in this work we provide a conceptually simple self contained and unified proof for the convergence of the mu algorithm applied on nmf with a wide range of divergences and regularizers our main result shows the sequence of iterates i e pairs of basis and coefficient matrices produced by the mu algorithm converges to the set of stationary points of the nonconvex nmf optimization problem our proof strategy has the potential to open up new avenues for analyzing similar problems in machine learning and signal processing

majorization minimization algorithms in signal processing communications and machine learning this paper gives an overview of the majorization minimization mm algorithmic framework which can provide guidance in deriving problem driven algorithms with low computational cost a general introduction of mm is presented including a description of the basic principle and its convergence results the extensions acceleration schemes and connection to other algorithmic frameworks are also covered to bridge the gap between theory and practice upperbounds for a large number of basic functions derived based on the taylor expansion convexity and special inequalities are provided as ingredients for constructing surrogate functions with the pre requisites established the way of applying mm to solving specific problems is elaborated by a wide range of applications in signal processing communications and machine learning

solution continuity in monotone affine variational inequalities in this paper we study the behavior of solutions of finite dimensional monotone affine variational inequalities posed over graph convex polyhedral multifunctions we identify precisely the class of positive semidefinite linear transformations appearing in these variational inequalities for which the solution sets will be lipschitzian in the argument of the underlying multifunction this class is that of cocoercive linear transformations which include but are not limited to those appearing in problems of linear or of convex quadratic programming

algorithms for nonnegative matrix factorization with the β divergence this letter describes algorithms for nonnegative matrix factorization nmf with the divergence nmf the divergence is a family of cost functions parameterized by a single shape parameter that takes the euclidean distance the kullback leibler divergence and the itakura saito divergence as special cases 2 1 0 respectively the proposed algorithms are based on a surrogate auxiliary function a local majorization of the criterion function we first describe a majorization minimization algorithm that leads to multiplicative updates which differ from standard heuristic multiplicative updates by a dependent power exponent the monotonicity of the heuristic algorithm can however be proven for 0 1 using the proposed auxiliary function then we introduce the concept of the majorization equalization me algorithm which produces updates that move along constant level sets of the auxiliary function and lead to larger steps than mm simulations on synthetic and real data illustrate the faster convergence of the me approach the letter also describes how the proposed algorithms can be adapted to two common variants of nmf penalized nmf when a penalty function of the factors is added to the criterion function and convex nmf when the dictionary is assumed to belong to a known subspace

maximum marginal likelihood estimation for nonnegative dictionary learning in the gamma poisson model in this paper we describe an alternative to standard nonnegative matrix factorization nmf for nonnegative dictionary learning i e the task of learning a dictionary with nonnegative values from nonnegative data under the assumption of nonnegative expansion coefficients a popular cost function used for nmf is the kullback leibler divergence which underlies a poisson observation model nmf can thus be considered as maximization of the joint likelihood of the dictionary and the expansion coefficients this approach lacks optimality because the number of parameters which include the expansion coefficients grows with the number of observations in this paper we describe variational bayes and monte carlo em algorithms for optimization of the marginal likelihood i e the likelihood of the dictionary where the expansion coefficients have been integrated out given a gamma prior we compare the output of both maximum joint likelihood estimation i e standard nmf and maximum marginal likelihood estimation mmle on real and synthetical datasets in particular we present face reconstruction results on cbcl dataset and text retrieval results over the musixmatch dataset a collection of word counts in song lyrics the mmle approach is shown to prevent overfitting by automatically pruning out irrelevant dictionary columns i e embedding automatic model order selection

projected gradient methods for nonnegative matrix factorization nonnegative matrix factorization nmf can be formulated as a minimization problem with bound constraints although bound constrained optimization has been studied extensively in both theory and practice so far no study has formally applied its techniques to nmf in this letter we propose two projected gradient methods for nmf both of which exhibit strong optimization properties we discuss efficient implementations and demonstrate that one of the proposed methods converges faster than the popular multiplicative update approach a simple matlab code is also provided