unified approach to h optimization hankel approximation and balanced realization problems a unified approach is proposed here to solve three problems simultaneously namely h optimization hankel approximation and minimal balanced realization for scalar systems it is shown that all these problems can be reduced to the same standard form wherein merely the singular value decomposition svd of an easily constructed square matrix is required to establish the final solutions the formulation is solely in terms of the coefficients of a transfer function and the resulting algorithm can be performed in a very computationally efficient way

a polynomial method for the singular value decomposition of block hankel operators this paper addresses the representation and analysis by polynomial methods of the block hankel operator corresponding to a rational transfer function matrix g z an algorithm for the singular values and vectors of is described and the numerical performance of an implementation is reported the algorithm involves no truncation error does not require knowledge of the poles of g and makes use of a minimal representation of if has rank d it is represented by a d d matrix

learning overcomplete representations in an overcomplete basis the number of basis vectors is greater than the dimensionality of the input and the representation of an input is not a unique combination of basis vectors overcomplete representations have been advocated because they have greater robustness in the presence of noise can be sparser and can have greater flexibility in matching structure in the data overcomplete codes have also been proposed as a model of some of the response properties of neurons in primary visual cortex previous work has focused on finding the best representation of a signal using a fixed overcomplete basis or dictionary we present an algorithm for learning an overcomplete basis by viewing it as probabilistic model of the observed data we show that overcomplete bases can yield a better approximation of the underlying statistical distribution of the data and can thus lead to greater coding efficiency this can be viewed as a generalization of the technique of independent component analysis and provides a method for bayesian reconstruction of signals in the presence of noise and for blind source separation when there are more sources than mixtures

one step extension approach to optimal hankel norm approximation and h sup infinity optimization problems a methodology is presented for hankel approximation and h sup infinity optimization problems that is based on a new formulation of a one step extension problem which is solved by the sarason interpolation theorem the parameterization of all optimal hankel approximants for multivariable systems is given in terms of the eigenvalue decomposition of an hermitian matrix composed directly from the coefficients of a given transfer function matrix phi rather than starting with the state space realization of phi the authors use polynomial coefficients of phi as input data in terms of these data a natural basis is given for the finite dimensional sarason model space and all computations involve only manipulations with finite matrices

direct learning of sparse changes in markov networks by density ratio estimation we propose a new method for detecting changes in markov network structure between two sets of samples instead of naively fitting two markov network models separately to the two data sets and figuring out their difference we directly learn the network structure change by estimating the ratio of markov network models this density ratio formulation naturally allows us to introduce sparsity in the network structure change which highly contributes to enhancing interpretability furthermore computation of the normalization term a critical bottleneck of the naive approach can be remarkably mitigated we also give the dual formulation of the optimization problem which further reduces the computation cost for large scale markov networks through experiments we demonstrate the usefulness of our method

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

minimax entropy principle and its application to texture modeling this article proposes a general theory and methodology called the minimax entropy principle for building statistical models for images or signals in a variety of applications this principle consists of two parts the first is the maximum entropy principle for feature binding or fusion for a given set of observed feature statistics a distribution can be built to bind these feature statistics together by maximizing the entropy over all distributions that reproduce them the second part is the minimum entropy principle for feature selection among all plausible sets of feature statistics we choose the set whose maximum entropy distribution has the minimum entropy computational and inferential issues in both parts are addressed in particular a feature pursuit procedure is proposed for approximately selecting the optimal set of features the minimax entropy principle is then corrected by considering the sample variation in the observed feature statistics and an information criterion for feature pursuit is derived the minimax entropy principle is applied to texture modeling where a novel markov random field mrf model called frame filter random field and minimax entropy is derived and encouraging results are obtained in experiments on a variety of texture images the relationship between our theory and the mechanisms of neural computation is also discussed

connections between score matching contrastive divergence and pseudolikelihood for continuous valued variables score matching sm and contrastive divergence cd are two recently proposed methods for estimation of nonnormalized statistical methods without computation of the normalization constant partition function although they are based on very different approaches we show in this letter that they are equivalent in a special case in the limit of infinitesimal noise in a specific monte carlo method further we show how these methods can be interpreted as approximations of pseudolikelihood

nonsmooth ica contrast minimization using a riemannian nelder mead method this brief concerns the design and application of a riemannian nelder mead algorithm to minimize a hartley entropy based contrast function to reliably estimate the sources from their mixtures despite its nondifferentiability the contrast function is endowed with attractive properties such as discriminacy and hence warrants an effort to be effectively handled by a derivative free optimizer aside from tailoring the nelder mead technique to the constraint set namely oblique manifold the source separation results attained in an empirical study with quasi correlated synthetic signals and digital images are presented which favor the proposed method on a comparative basis

the singular value decomposition of an infinite hankel matrix abstract let h be an infinite hankel matrix of known finite rank r a new algorithm for the numerical calculation of the singular values and vectors of h is presented the method proceeds by reduction to the singular value problem for an r r matrix this is achieved without solving for the poles of the symbol of h the resulting algorithm is of order r 3

numerical algorithms for the nevanlinna pick problem based on the formulation in 14 efficient algorithms are devised for two variants of the nevanlinna pick problem their implementation is validated by several examples the method depends on establishing the equivalence of two different operator theoretic approaches to the solution of classical interpolation problems 1 14

learning the information divergence information divergence that measures the difference between two nonnegative matrices or tensors has found its use in a variety of machine learning problems examples are nonnegative matrix tensor factorization stochastic neighbor embedding topic models and bayesian network optimization the success of such a learning task depends heavily on a suitable divergence a large variety of divergences have been suggested and analyzed but very few results are available for an objective choice of the optimal divergence for a given task here we present a framework that facilitates automatic selection of the best divergence among a given family based on standard maximum likelihood estimation we first propose an approximated tweedie distribution for the beta divergence family selecting the best beta then becomes a machine learning problem solved by maximum likelihood next we reformulate alpha divergence in terms of beta divergence which enables automatic selection of alpha by maximum likelihood with reuse of the learning principle for beta divergence furthermore we show the connections between gamma and beta divergences as well as renyi and alpha divergences such that our automatic selection framework is extended to non separable divergences experiments on both synthetic and real world data demonstrate that our method can quite accurately select information divergence across different learning problems and various divergence families

modeling multiscale subbands of photographic images with fields of gaussian scale mixtures the local statistical properties of photographic images when represented in a multi scale basis have been described using gaussian scale mixtures here we use this local description as a substrate for constructing a global field of gaussian scale mixtures fogsm specifically we model multi scale subbands as a product of an exponentiated homogeneous gaussian markov random field hgmrf and a second independent hgmrf we show that parameter estimation for this model is feasible and that samples drawn from a fogsm model have marginal and joint statistics similar to subband coefficients of photographic images we develop an algorithm for removing additive gaussian white noise based on the fogsm model and demonstrate denoising performance comparable with state of the art methods

generalized inversion of finite rank hankel and toeplitz operators with rational matrix symbols abstract the goal of the paper is a generalized inversion of finite rank hankel operators and hankel or toeplitz operators with block matrices having finitely many rows to attain it a left coprime fractional factorization of a strictly proper rational matrix function and the bezout equation are used generalized inverses of these operators and generating functions for the inverses are explicitly constructed in terms of the fractional factorization

ica using spacings estimates of entropy this paper presents a new algorithm for the independent components analysis ica problem based on an efficient entropy estimator like many previous methods this algorithm directly minimizes the measure of departure from independence according to the estimated kullback leibler divergence between the joint distribution and the product of the marginal distributions we pair this approach with efficient entropy estimators from the statistics literature in particular the entropy estimator we use is consistent and exhibits rapid convergence the algorithm based on this estimator is simple computationally efficient intuitively appealing and outperforms other well known algorithms in addition the estimator s relative insensitivity to outliers translates into superior performance by our ica algorithm on outlier tests we present favorable comparisons to the kernel ica fast ica jade and extended infomax algorithms in extensive simulations we also provide public domain source code for our algorithms

a derivative free riemannian powell s method minimizing hartley entropy based ica contrast even though the hartley entropy based contrast function guarantees an unmixing local minimum the reported nonsmooth optimization techniques that minimize this nondifferentiable function encounter computational bottlenecks toward this powell s derivative free optimization method has been extended to a riemannian manifold namely oblique manifold for the recovery of quasi correlated sources by minimizing this contrast function the proposed scheme has been demonstrated to converge faster than the related algorithms in the literature besides the impressive source separation results in simulations involving synthetic sources having finite support distributions and correlated images

fast kernel based independent component analysis recent approaches to independent component analysis ica have used kernel independence measures to obtain highly accurate solutions particularly where classical methods experience difficulty for instance sources with near zero kurtosis fastkica fast hsic based kernel ica is a new optimization method for one such kernel independence measure the hilbert schmidt independence criterion hsic the high computational efficiency of this approach is achieved by combining geometric optimization techniques specifically an approximate newton like method on the orthogonal group with accurate estimates of the gradient and hessian based on an incomplete cholesky decomposition in contrast to other efficient kernel based ica algorithms fastkica is applicable to any twice differentiable kernel function experimental results for problems with large numbers of sources and observations indicate that fastkica provides more accurate solutions at a given cost than gradient descent on hsic comparing with other recently published ica methods fastkica is competitive in terms of accuracy relatively insensitive to local minima when initialized far from independence and more robust towards outliers an analysis of the local convergence properties of fastkica is provided

noise contrastive estimation of unnormalized statistical models with applications to natural image statistics we consider the task of estimating from observed data a probabilistic model that is parameterized by a finite number of parameters in particular we are considering the situation where the model probability density function is unnormalized that is the model is only specified up to the partition function the partition function normalizes a model so that it integrates to one for any choice of the parameters however it is often impossible to obtain it in closed form gibbs distributions markov and multi layer networks are examples of models where analytical normalization is often impossible maximum likelihood estimation can then not be used without resorting to numerical approximations which are often computationally expensive we propose here a new objective function for the estimation of both normalized and unnormalized models the basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise with this approach the normalizing partition function can be estimated like any other parameter we prove that the new estimation method leads to a consistent convergent estimator of the parameters for large noise sample sizes the new estimator is furthermore shown to behave like the maximum likelihood estimator in the estimation of unnormalized models there is a trade off between statistical and computational performance we show that the new method strikes a competitive trade off in comparison to other estimation methods for unnormalized models as an application to real data we estimate novel two layer models of natural image statistics with spline nonlinearities

consistency of pseudolikelihood estimation of fully visible boltzmann machines a boltzmann machine is a classic model of neural computation and a number of methods have been proposed for its estimation most methods are plagued by either very slow convergence or asymptotic bias in the resulting estimates here we consider estimation in the basic case of fully visible boltzmann machines we show that the old principle of pseudolikelihood estimation provides an estimator that is computationally very simple yet statistically consistent

tracking poles and representing hankel operators directly from data we propose and analyse a method of estimating the poles near the unit circlet of a functiong whose values are given at a grid of points ont we give an algorithm for performing this estimation and prove a convergence theorem the method is to identify the phase for an estimate by considering the peaks of the absolute value ofg ont and then to estimate the modulus by seeking a bestl 2 fit tog over a small arc by a first order rational function these pole estimates lead to the construction of a basis ofl 2 which is well suited to the numerical representation of the hankel operator with symbolg and thereby to the numerical solution of the nehari problem computing the besth analytic approximation tog relative to thel norm as analysed in hy we present the results of numerical tests of these algorithms

estimation of non normalized statistical models by score matching one often wants to estimate statistical models where the probability density function is known only up to a multiplicative normalization constant typically one then has to resort to markov chain monte carlo methods or approximations of the normalization constant here we propose that such models can be estimated by minimizing the expected squared distance between the gradient of the log density given by the model and the gradient of the log density of the observed data while the estimation of the gradient of log density function is in principle a very difficult non parametric problem we prove a surprising result that gives a simple formula for this objective function the density function of the observed data does not appear in this formula which simplifies to a sample average of a sum of some derivatives of the log density given by the model the validity of the method is demonstrated on multivariate gaussian and independent component analysis models and by estimating an overcomplete filter set for natural image data

a computational algorithm for the super optimal solution of the model matching problem abstract in this paper we develop a computational algorithm to implement the strengthened minimization technique 1 in the model matching problem which arises in robust controller design and weighted sensitivity minimization a general algorithm is proposed for constructing unitary matrices and the extremal functions for the h optimization problem are diagonalized using the constructed unitary matrices we impose stronger minimization conditions to force the solution of the optimization problem to be unique and explain their physical significance

an estimator for the relative entropy rate of path measures for stochastic differential equations we address the problem of estimating the relative entropy rate rer for two stochastic processes described by stochastic differential equations for the case where the drift of one process is known analytically but one has only observations from the second process we use a variational bound on the rer to construct an estimator

a generalized gaussian image model for edge preserving map estimation the authors present a markov random field model which allows realistic edge modeling while providing stable maximum a posterior map solutions the model referred to as a generalized gaussian markov random field ggmrf is named for its similarity to the generalized gaussian distribution used in robust detection and estimation the model satisfies several desirable analytical and computational properties for map estimation including continuous dependence of the estimate on the data invariance of the character of solutions to scaling of data and a solution which lies at the unique global minimum of the a posteriori log likelihood function the ggmrf is demonstrated to be useful for image reconstruction in low dosage transmission tomography

on identification of stable systems and optimal approximation abstract approximative modelling of stable continuous time possibly infinite dimensional systems is studied based on an optimal approximation approach both approximation of analytical system representations system approximation as well as approximation of input output data based system estimates system identification are considered while special emphasis is given to approximative modelling in the h and hankel norms the l1 and l2 norm cases are also discussed the model sets considered here are finite dimensional systems and time shifted systems simple delay systems the theory of approximation numbers is shown to provide a convenient tool to study problems of identification of stable continuous time systems in a deterministic framework with close connections to complexity considerations laguerre fourier series methods and hankel operator techniques can be utilized to develop fully practical identification methods for continuous time possibly infinite dimensional systems

complex ica using nonlinear functions we introduce a framework based on wirtinger calculus for nonlinear complex valued signal processing such that all computations can be directly carried out in the complex domain the two main approaches for performing independent component analysis maximum likelihood and maximization of non gaussianity which are intimately related to each other are studied using this framework the main update rules for the two approaches are derived their properties and density matching strategies are discussed along with numerical examples to highlight their relationships

a modification of the nevanlinna algorithm on the solution of the h sup infinity sub m n optimization problem a modification of the nevanlinna algorithm is presented for the solution of the h sup infinity sub m m optimization problem this is an essential modification when the nevanlinna algorithm is applied to find the extremal function of the matrix h sup infinity interpolation problem which deals with the case that the interpolation points have multiplicities for the computation of the minimal h sup infinity norm a pure matrices computational algorithm is used where the formulation is solely in terms of the coefficients of the inner function

least squares estimation without priors or supervision selection of an optimal estimator typically relies on either supervised training samples pairs of measurements and their associated true values or a prior probability model for the true values here we consider the problem of obtaining a least squares estimator given a measurement process with known statistics i e a likelihood function and a set of unsupervised measurements each arising from a corresponding true value drawn randomly from an unknown distribution we develop a general expression for a nonparametric empirical bayes least squares nebls estimator which expresses the optimal least squares estimator in terms of the measurement density with no explicit reference to the unknown prior density we study the conditions under which such estimators exist and derive specific forms for a variety of different measurement processes we further show that each of these nebls estimators may be used to express the mean squared estimation error as an expectation over the measurement density alone thus generalizing stein s unbiased risk estimator sure which provides such an expression for the additive gaussian noise case this error expression may then be optimized over noisy measurement samples in the absence of supervised training data yielding a generalized sure optimized parametric least squares sure2pls estimator in the special case of a linear parameterization i e a sum of nonlinear kernel functions the objective function is quadratic and we derive an incremental form for learning this estimator from data we also show that combining the nebls form with its corresponding generalized sure expression produces a generalization of the score matching procedure for parametric density estimation finally we have implemented several examples of such estimators and we show that their performance is comparable to their optimal bayesian or supervised regression counterparts for moderate to large amounts of data

a unified framework for quadratic measures of independence this paper proposes a unified framework for several available measures of independence by generalizing the concept of information theoretic learning itl the key component of itl is the use of inner product between two density functions as a measure of similarity between two random variables we show that by generalizing the inner product using a symmetric strictly positive definite kernel and by choosing appropriate kernels it is possible to reproduce a number of popular measures of independence this unified framework also allows the design of new strictly positive definite kernels and corresponding measures of independence following this framework we explore a new measure of independence and apply it in the context of linear independent component analysis ica an attractive property of the proposed method is that it does not involve any free parameter and we demonstrate that it performs equally well compared to the existing methods for ica

tracking poles representing hankel operators and the nehari problem we present an algorithm which locates the poles and zeros of a rational function given the values at the roots of unity so long as enough values are specified to make the problem well posed the algorithm is robust in a strong sense if the sample values are perturbed slightly it will identify the correct number of candidate poles and zeros and they will be close to the correct poles and zeros the algorithm proceeds by first calculating the discrete fourier transform from the given sample values and then examining the singular value decompositions of truncations of four hankel matrices formed from them we then show how given such sample values for a rational function o we may exploit the poles and zeros to construct the szego bases for ho the hankel operator with symbol o and hence solve the nehari problem the algorithms are shown to be robust and they are very accurate the results improve considerably on those of helton spain and young

bayesian model comparison with the hyvarinen score computation and consistency the bayes factor is a widely used criterion in model comparison and its logarithm is a difference of out of sample predictive scores under the logarithmic scoring rule however when some of the ca

on the risk of using renyi s entropy for blind source separation recently some researchers have suggested renyi s entropy in its general form as a blind source separation bss objective function this was motivated by two arguments 1 shannon s entropy which is known to be a suitable criterion for bss is a particular case of renyi s entropy and 2 some practical advantages can be obtained by choosing another specific value for the renyi exponent yielding to e g quadratic entropy unfortunately by doing so there is no longer guarantee that optimizing this generalized criterion would lead to recovering the original sources in this paper we show that renyi s entropy in its exact form i e out of any consideration about its practical estimation or computation might lead to not recovering the sources depending on the source densities and on renyi s exponent value this is illustrated on specific examples we also compare our conclusions with previous works involving renyi s entropies for blind deconvolution

on singular values of hankel operators of finite rank abstract let h be a hankel operator defined by its symbol where is a monic polynomial of degree n and is a polynomial of degree less than n then h has rank n we derive a generalized takagi singular value problem defined by two n x n matrices such that its n generalized takagi singular values are the positive singular values of h if is real then the generalized takagi singular value problem reduces to a generalized symmetric eigenvalue problem the computations can be carried out so that the lanczos method applied to the latter problem requires only o n log n arithmetic operations for each iteration if and are given in power form then the elements of all n x n matrices required can be determined in o n 2 arithmetic operations

a new algorithm for the minimal balanced realization of a transfer function matrix abstract a new algorithm is proposed here to obtain a minimal balanced realization directly from the transfer function matrix tfm this method which employs the singular value decomposition svd of an infinite block hankel matrix requires neither an initial minimal realization nor the solution of a lyapunov matrix equation the formulation is solely in terms of the coefficients of the transfer function matrix

a fast learning algorithm for deep belief nets we show how to use complementary priors to eliminate the explaining away effects that make inference difficult in densely connected belief nets that have many hidden layers using complementary priors we derive a fast greedy algorithm that can learn deep directed belief networks one layer at a time provided the top two layers form an undirected associative memory the fast greedy algorithm is used to initialize a slower learning procedure that fine tunes the weights using a contrastive version of the wake sleep algorithm after fine tuning a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels this generative model gives better digit classification than the best discriminative learning algorithms the low dimensional manifolds on which the digits lie are modeled by long ravines in the free energy landscape of the top level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind

spherical mesh adaptive direct search for separating quasi uncorrelated sources by range based independent component analysis it is seemingly paradoxical to the classical definition of the independent component analysis ica that in reality the true sources are often not strictly uncorrelated with this in mind this letter concerns a framework to extract quasi uncorrelated sources with finite supports by optimizing a range based contrast function under unit norm constraints to handle the inherent scaling indeterminacy of ica but without orthogonality constraints albeit the appealing contrast properties of the range based function e g the absence of mixing local optima the function is not differentiable everywhere unfortunately there is a dearth of literature on derivative free optimizers that effectively handle such a nonsmooth yet promising contrast function this is the compelling reason for the design of a nonsmooth optimization algorithm on a manifold of matrices having unit norm columns with the following objectives to ascertain convergence to a clarke stationary point of the contrast function and adhere to the necessary unit norm constraints more naturally the proposed nonsmooth optimization algorithm crucially relies on the design and analysis of an extension of the mesh adaptive direct search mads method to handle locally lipschitz objective functions defined on the sphere the applicability of the algorithm in the ica domain is demonstrated with simulations involving natural face aerial and texture images

new method for parameter estimation in probabilistic models minimum probability flow fitting probabilistic models to data is often difficult due to the general intractability of the partition function we propose a new parameter fitting method minimum probability flow mpf which is applicable to any parametric model we demonstrate parameter estimation using mpf in two cases a continuous state space model and an ising spin glass in the latter case mpf outperforms current techniques by at least an order of magnitude in convergence time with lower error in the recovered coupling parameters

energy based models for sparse overcomplete representations we present a new way of extending independent components analysis ica to overcomplete representations in contrast to the causal generative extensions of ica which maintain marginal independence of sources we define features as deterministic linear functions of the inputs this assumption results in marginal dependencies among the features but conditional independence of the features given the inputs by assigning energies to the features a probability distribution over the input states is defined through the boltzmann distribution free parameters of this model are trained using the contrastive divergence objective hinton 2002 when the number of features is equal to the number of input dimensions this energy based model reduces to noiseless ica and we show experimentally that the proposed learning algorithm is able to perform blind source separation on speech data in additional experiments we train overcomplete energy based models to extract features from various standard data sets containing speech natural images hand written digits and faces

independent component analysis by entropy bound minimization a novel differential entropy estimator is introduced where the maximum entropy bound is used to approximate the entropy given the observations and is computed using a numerical procedure thus resulting in accurate estimates for the entropy we show that such an estimator exists for a wide class of measuring functions and provide a number of design examples to demonstrate its flexible nature we then derive a novel independent component analysis ica algorithm that uses the entropy estimate thus obtained ica by entropy bound minimization ica ebm the algorithm adopts a line search procedure and initially uses updates that constrain the demixing matrix to be orthogonal for robust performance we demonstrate the superior performance of ica ebm and its ability to match sources that come from a wide range of distributions using simulated and real world data

what regularized auto encoders learn from the data generating distribution what do auto encoders learn about the underlying data generating distribution recent work suggests that some auto encoder variants do a good job of capturing the local manifold structure of data this paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density we show that the auto encoder captures the score derivative of the log density with respect to the input it contradicts previous interpretations of reconstruction error as an energy function unlike previous results the theorems provided here are completely generic and do not depend on the parameterization of the auto encoder they show what the auto encoder would tend to if given enough capacity and examples these results are for a contractive training criterion we show to be similar to the denoising auto encoder training criterion with small corruption noise but with contraction applied on the whole reconstruction function rather than just encoder similarly to score matching one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function finally we show how an approximate metropolis hastings mcmc can be setup to recover samples from the estimated distribution and this is confirmed in sampling experiments

a novel parameter estimation method for boltzmann machines we propose a novel estimator for a specific class of probabilistic models on discrete spaces such as the boltzmann machine the proposed estimator is derived from minimization of a convex risk function and can be constructed without calculating the normalization constant whose computational cost is exponential order we investigate statistical properties of the proposed estimator such as consistency and asymptotic normality in the framework of the estimating function small experiments show that the proposed estimator can attain comparable performance to the maximum likelihood expectation at a much lower computational cost and is applicable to high dimensional data

kernel independent component analysis we present a class of algorithms for independent component analysis ica which use contrast functions based on canonical correlations in a reproducing kernel hilbert space on the one hand we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence on the other hand building on recent developments in kernel methods we show that these criteria and their derivatives can be computed efficiently minimizing these criteria leads to flexible and robust algorithms for ica we illustrate with simulations involving a wide variety of source distributions showing that our algorithms outperform many of the presently known algorithms

fast and robust fixed point algorithms for independent component analysis independent component analysis ica is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible we use a combination of two different approaches for linear ica comon s information theoretic approach and the projection pursuit approach using maximum entropy approximations of differential entropy we introduce a family of new contrast functions for ica these contrast functions enable both the estimation of the whole decomposition by minimizing mutual information and estimation of individual independent components as projection pursuit directions the statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model and it is shown how to choose contrast functions that are robust and or of minimum variance finally we introduce simple fixed point algorithms for practical optimization of the contrast functions

a minimum range approach to blind extraction of bounded sources in spite of the numerous approaches that have been derived for solving the independent component analysis ica problem it is still interesting to develop new methods when among other reasons specific a priori knowledge may help to further improve the separation performances in this paper the minimum range approach to blind extraction of bounded source is investigated the relationship with other existing well known criteria is established it is proved that the minimum range approach is a contrast and that the criterion is discriminant in the sense that it is free of spurious maxima the practical issues are also discussed and a range measure estimation is proposed based on the order statistics an algorithm for contrast maximization over the group of special orthogonal matrices is proposed simulation results illustrate the performances of the algorithm when using the proposed range estimation criterion

balanced minimal realization via singular value decomposition of sarason operator abstract in this paper a method is proposed to obtain a balanced minimal realization of a stable rational transfer function matrix the allison young algorithm for the interpolation problems is directly extended to matrix value cases and is applied in the realization procedure this method uses the finite matrix of a sarason operator instead of the infinite block hankel matrix and the coefficients of the transfer function matrix instead of any initial realization the advantage of this approach in connection with the solution of l optimization problems and the balanced realization is also discussed

a variational method for learning sparse and overcomplete representations an expectation maximization algorithm for learning sparse and overcomplete data representations is presented the proposed algorithm exploits a variational approximation to a range of heavy tailed distributions whose limit is the laplacian a rigorous lower bound on the sparse prior distribution is derived which enables the analytic marginalization of a lower bound on the data likelihood this lower bound enables the development of an expectation maximization algorithm for learning the overcomplete basis vectors and inferring the most probable basis coefficients

linear estimating equations for exponential families with application to gaussian linear concentration models abstract in many families of distributions maximum likelihood estimation is intractable because the normalization constant for the density which enters into the likelihood function is not easily available the score matching estimator 35 provides an alternative where this normalization constant is not required for an exponential family e g a gaussian linear concentration model the corresponding estimating equations become linear 2 36 and the score matching estimator is shown to be consistent and asymptotically normally distributed as the number of observations increase to infinity although not necessarily efficient for linear concentration models that are also linear in the covariance 37 we show that the score matching estimator is identical to the maximum likelihood estimator hence in such cases it is also efficient gaussian graphical models and graphical models with symmetries 32 form particularly interesting subclasses of linear concentration models and we investigate the potential use of the score matching estimator for this case

fields of experts we develop a framework for learning generic expressive image priors that capture the statistics of natural scenes and can be used for a variety of machine vision tasks the approach provides a practical method for learning high order markov random field mrf models with potential functions that extend over large pixel neighborhoods these clique potentials are modeled using the product of experts framework that uses non linear functions of many linear filter responses in contrast to previous mrf approaches all parameters including the linear filters themselves are learned from training data we demonstrate the capabilities of this field of experts model with two example applications image denoising and image inpainting which are implemented using a simple approximate inference scheme while the model is trained on a generic image database and is not tuned toward a specific application we obtain results that compete with specialized techniques

phase coupling estimation from multivariate phase statistics coupled oscillators are prevalent throughout the physical world dynamical system formulations of weakly coupled oscillator systems have proven effective at capturing the properties of real world systems and are compelling models of neural systems however these formulations usually deal with the forward problem simulating a system from known coupling parameters here we provide a solution to the inverse problem determining the coupling parameters from measurements starting from the dynamic equations of a system of symmetrically coupled phase oscillators given by a nonlinear langevin equation we derive the corresponding equilibrium distribution this formulation leads us to the maximum entropy distribution that captures pairwise phase relationships to solve the inverse problem for this distribution we derive a closed form solution for estimating the phase coupling parameters from observed phase statistics through simulations we show that the algorithm performs well in high dimensions d 100 and in cases with limited data as few as 100 samples per dimension in addition we derive a regularized solution to the estimation and show that the resulting procedure improves performance when only a limited amount of data is available because the distribution serves as the unique maximum entropy solution for pairwise phase statistics phase coupling estimation can be broadly applied in any situation where phase measurements are made under the physical interpretation the model may be used for inferring coupling relationships within cortical networks

a simple algorithm on minimal balanced realization for transfer function matrices given a transfer function matrix it is shown that its minimal balanced realization can be obtained directly from the singular values and the singular vectors of a constant matrix constructed from the coefficients of the least common denominator polynomial the algorithm is based on the close relationship of controllability and the observability gramians to the singular value decomposition of the associated infinite block hankel matrix

approximation of delay systems a case study abstract the approximation of delay systems is studied a numerically well behaved method for computing hankel optima rational approximations for delay systems is presented based on certain properties of hankel matrices and the theory of laguerre polynomials the cf method of trefethen and certain pade approximations of delay systems are also considered the importance of a certain wiener algebra property in the analysis of the rational approximation of an important class of delay systems in the l norm and the hankel norm is shown completing certain results presented in the literature for systems satisfying certain nuclearity or absolute continuity conditions a case study and numerical comparison is presented for the approximation of the parametric family of first order stable delay systems g s exp s ts 1 numerical experience indicates that l optimized cf approximations based on short truncated maclaurin series give in general somewhat smaller l approximation errors than hankel optim

training products of experts by minimizing contrastive divergence it is possible to combine multiple latent variable models of the same data by multiplying their probability distributions together and then renormalizing this way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data a product of experts poe is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary training a poe by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule fortunately a poe can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently examples are presented of contrastive divergence learning using several types of expert on several types of data

blind separation of mixture of independent sources through a quasi maximum likelihood approach we propose two methods for separating mixture of independent sources without any precise knowledge of their probability distribution they are obtained by considering a maximum likelihood ml solution corresponding to some given distributions of the sources and relaxing this assumption afterward the first method is specially adapted to temporally independent non gaussian sources and is based on the use of nonlinear separating functions the second method is specially adapted to correlated sources with distinct spectra and is based on the use of linear separating filters a theoretical analysis of the performance of the methods has been made a simple procedure for optimally choosing the separating functions is proposed further in the second method a simple implementation based on the simultaneous diagonalization of two symmetric matrices is provided finally some numerical and simulation results are given illustrating the performance of the method and the good agreement between the experiments and the theory

convergence of contrastive divergence algorithm in exponential family the contrastive divergence cd algorithm has achieved notable success in training energy based models including restricted boltzmann machines and played a key role in the emergence of deep learning the idea of this algorithm is to approximate the intractable term in the exact gradient of the log likelihood function by using short markov chain monte carlo mcmc runs the approximate gradient is computationally cheap but biased whether and why the cd algorithm provides an asymptotically consistent estimate are still open questions this paper studies the asymptotic properties of the cd algorithm in canonical exponential families which are special cases of the energy based model suppose the cd algorithm runs m mcmc transition steps at each iteration t and iteratively generates a sequence of parameter estimates theta t t ge 0 given an i i d data sample x i i 1 n sim p theta star under conditions which are commonly obeyed by the cd algorithm in practice we prove the existence of some bounded m such that any limit point of the time average sum s 0 t 1 theta s t as t to infty is a consistent estimate for the true parameter theta star our proof is based on the fact that theta t t ge 0 is a homogenous markov chain conditional on the data sample x i i 1 n this chain meets the foster lyapunov drift criterion and converges to a random walk around the maximum likelihood estimate the range of the random walk shrinks to zero at rate mathcal o 1 sqrt 3 n as the sample size n to infty

shiftable multiscale transforms one of the major drawbacks of orthogonal wavelet transforms is their lack of translation invariance the content of wavelet subbands is unstable under translations of the input signal wavelet transforms are also unstable with respect to dilations of the input signal and in two dimensions rotations of the input signal the authors formalize these problems by defining a type of translation invariance called shiftability in the spatial domain shiftability corresponds to a lack of aliasing thus the conditions under which the property holds are specified by the sampling theorem shiftability may also be applied in the context of other domains particularly orientation and scale jointly shiftable transforms that are simultaneously shiftable in more than one domain are explored two examples of jointly shiftable transforms are designed and implemented a 1 d transform that is jointly shiftable in position and scale and a 2 d transform that is jointly shiftable in position and orientation the usefulness of these image representations for scale space analysis stereo disparity measurement and image enhancement is demonstrated

optimal approximation of signal priors in signal restoration by bayesian inference one typically uses a parametric model of the prior distribution of the signal here we consider how the parameters of a prior model should be estimated from observations of uncorrupted signals a lot of recent work has implicitly assumed that maximum likelihood estimation is the optimal estimation method our results imply that this is not the case we first obtain an objective function that approximates the error occurred in signal restoration due to an imperfect prior model next we show that in an important special case small gaussian noise the error is the same as the score matching objective function which was previously proposed as an alternative for likelihood based on purely computational considerations our analysis thus shows that score matching combines computational simplicity with statistical optimality in signal restoration providing a viable alternative to maximum likelihood methods we also show how the method leads to a new intuitive and geometric interpretation of structure inherent in probability distributions

modeling natural images using gated mrfs this paper describes a markov random field for real valued image modeling that has two sets of latent variables one set is used to gate the interactions between all pairs of pixels while the second set determines the mean intensities of each pixel this is a powerful model with a conditional distribution over the input that is gaussian with both mean and covariance determined by the configuration of latent variables which is unlike previous models that were restricted to using gaussians with either a fixed mean or a diagonal covariance matrix thanks to the increased flexibility this gated mrf can generate more realistic samples after training on an unconstrained distribution of high resolution natural images furthermore the latent variables of the model can be inferred efficiently and can be used as very effective descriptors in recognition tasks both generation and discrimination drastically improve as layers of binary latent variables are added to the model yielding a hierarchical model called a deep belief network

existence and uniqueness of proper scoring rules to discuss the existence and uniqueness of proper scoring rules one needs to extend the associated entropy functions as sublinear functions to the conic hull of the prediction set in some natural function spaces such as the lebesgue lp spaces over rd the positive cones have empty interior entropy functions defined on such cones have directional derivatives only which typically exist on large subspaces and behave similarly to gradients certain entropies may be further extended continuously to open cones in normed spaces containing signed densities the extended densities are g teaux differentiable except on a negligible set and have everywhere continuous subgradients due to the supporting hyperplane theorem we introduce the necessary framework from analysis and algebra that allows us to give an affirmative answer to the titular question of the paper as a result of this we give a formal sense in which entropy functions have uniquely associated proper scoring rules we illustrate our framework by studying the derivatives and subgradients of the following three prototypical entropies shannon entropy hyvarinen entropy and quadratic entropy

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

simultaneous estimation of nongaussian components and their correlation structure the statistical dependencies that independent component analysis ica cannot remove often provide rich information beyond the linear independent components it would thus be very useful to estimate the dependency structure from data while such models have been proposed they have usually concentrated on higher order correlations such as energy square correlations yet linear correlations are a fundamental and informative form of dependency in many real data sets linear correlations are usually completely removed by ica and related methods so they can only be analyzed by developing new methods that explicitly allow for linearly correlated components in this article we propose a probabilistic model of linear nongaussian components that are allowed to have both linear and energy correlations the precision matrix of the linear components is assumed to be randomly generated by a higher order process and explicitly parameterized by a parameter matrix the estimation of the parameter matrix is shown to be particularly simple because using score matching hyvarinen 2005 the objective function is a quadratic form using simulations with artificial data we demonstrate that the proposed method improves the identifiability of nongaussian components by simultaneously learning their correlation structure applications on simulated complex cells with natural image input as well as spectrograms of natural audio data show that the method finds new kinds of dependencies between the components

expectation truncation and the benefits of preselection in training generative models we show how a preselection of hidden variables can be used to efficiently train generative models with binary hidden variables the approach is based on expectation maximization em and uses an efficiently computable approximation to the sufficient statistics of a given model the computational cost to compute the sufficient statistics is strongly reduced by selecting for each data point the relevant hidden causes the approximation is applicable to a wide range of generative models and provides an interpretation of the benefits of preselection in terms of a variational em approximation to empirically show that the method maximizes the data likelihood it is applied to different types of generative models including a version of non negative matrix factorization nmf a model for non linear component extraction mca and a linear generative model similar to sparse coding the derived algorithms are applied to both artificial and realistic data and are compared to other models in the literature we find that the training scheme can reduce computational costs by orders of magnitude and allows for a reliable extraction of hidden causes

a polynomial approach to hankel norm and balanced approximations abstract a new unified approach to problems of hankel norm and balanced approximations is presented which is based on a combination of polynomial algebra and the geometry of invariant subspaces contrary to state space methods where contact with external properties of systems is indirect the approach presented yields new insights into basic properties of hankel norm approximation and balanced realizations several approximation results are interpreted geometrically in terms of projections also duality results in this area are printed out

tukey g and h random fields abstractwe propose a new class of transgaussian random fields named tukey g and h tgh random fields to model non gaussian spatial data the proposed tgh random fields have extremely flexible marginal distributions possibly skewed and or heavy tailed and therefore have a wide range of applications the special formulation of the tgh random field enables an automatic search for the most suitable transformation for the dataset of interest while estimating model parameters asymptotic properties of the maximum likelihood estimator and the probabilistic properties of the tgh random fields are investigated an efficient estimation procedure based on maximum approximated likelihood is proposed and an extreme spatial outlier detection algorithm is formulated kriging and probabilistic prediction with tgh random fields are developed along with prediction confidence intervals the predictive performance of tgh random fields is demonstrated through extensive simulation studies and an application to a dataset

a connection between score matching and denoising autoencoders denoising autoencoders have been previously shown to be competitive alternatives to restricted boltzmann machines for unsupervised pretraining of each layer of a deep architecture we show that a simple denoising autoencoder training criterion is equivalent to matching the score with respect to the data of a specific energy based model to that of a nonparametric parzen density estimator of the data this yields several useful insights it defines a proper probabilistic model for the denoising autoencoder technique which makes it in principle possible to sample from them or rank examples by their energy it suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives it justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy based models