variational bayes solution of linear neural networks and its generalization performance it is well known that in unidentifiable models the bayes estimation provides much better generalization performance than the maximum likelihood ml estimation however its accurate approximation by markov chain monte carlo methods requires huge computational costs as an alternative a tractable approximation method called the variational bayes vb approach has recently been proposed and has been attracting attention its advantage over the expectation maximization em algorithm often used for realizing the ml estimation has been experimentally shown in many applications nevertheless it has not yet been theoretically shown in this letter through analysis of the simplest unidentifiable models we theoretically show some properties of the vb approach we first prove that in three layer linear neural networks the vb approach is asymptotically equivalent to a positive part james stein type shrinkage estimation then we theoretically clarify its free energy generalization error and training error comparing them with those of the ml estimation and the bayes estimation we discuss the advantage of the vb approach we also show that unlike in the bayes estimation the free energy and the generalization error are less simply related with each other and that in typical cases the vb free energy well approximates the bayes one while the vb generalization error significantly differs from the bayes one

condition for perfect dimensionality recovery by variational bayesian pca having shown its good performance in many applications variational bayesian vb learning is known to be one of the best tractable approximations to bayesian learning however its performance was not well understood theoretically in this paper we clarify the behavior of vb learning in probabilistic pca or fully observed matrix factorization more specifically we establish a necessary and sufficient condition for perfect dimensionality or rank recovery in the large scale limit when the matrix size goes to infinity our result theoretically guarantees the performance of vb pca at the same time it also reveals the conservative nature of vb learning it offers a low false positive rate at the expense of low sensitivity by contrasting with an alternative dimensionality selection method we characterize vb learning in pca in our analysis we obtain bounds of the noise variance estimator and a new and simple analytic form solution for the other parameters which themselves are useful for implementation of vb pca

a conservative discretization of the shallow water equations on triangular grids abstract a structure preserving discretization of the shallow water equations on unstructured spherical grids is introduced the unstructured grids that we consider have triangular cells with a c type staggering of variables where scalar variables are located at centres of grid cells and normal components of velocity are placed at cell boundaries the staggering necessitates reconstructions and these reconstructions are build into the algorithm such that the resulting discrete equations obey a weighted weak form this approach combined with a mimetic discretization of the differential operators of the shallow water equations provides a conservative discretization that preserves important aspects of the mathematical structure of the continuous equations most notably the simultaneous conservation of quadratic invariants such as energy and enstrophy the structure preserving nature of our discretization is confirmed through theoretical analysis and through numerical experiments on two different triangular grids a symmetrized icosahedral grid of nearly uniform resolution and a non uniform triangular grid whose resolution increases towards the poles

generalizations of arakawa s jacobian abstract a simple method yields discrete jacobians that obey analogues of the differential properties needed to conserve energy and enstrophy in 2 dimensional flow the method is actually independent of the type of discretization and thus applies to an arbitrary representation in gridpoints finite elements or spectral modes or to any mixture of the three we illustrate the method by deriving simple energy and enstrophy conserving jacobians for an irregular triangular mesh in a closed domain and for a mixed gridpoint and mode representation in a semi infinite channel

statistical relevance of vorticity conservation in the hamiltonian particle mesh method we conduct long time simulations with a hamiltonian particle mesh method for ideal fluid flow to determine the statistical mean vorticity field of the discretization lagrangian and eulerian statistical models are proposed for the discrete dynamics and these are compared against numerical experiments the observed results are in excellent agreement with the theoretical models as well as with the continuum statistical mechanical theory for ideal fluid flow developed by ellis et al 2002 10 in particular the results verify that the apparently trivial conservation of potential vorticity along particle paths within the hpm method significantly influences the mean state as a side note the numerical experiments show that a nonzero fourth moment of potential vorticity can influence the statistical mean

explicit lie poisson integration and the euler equations we give a wide class of lie poisson systems for which explicit lie poisson integrators preserving all casimirs can be constructed the integrators are extremely simple and have widespread applicability examples are the free rigid body a moment truncation and a new fast algorithm for the sine bracket truncation of the 2d euler equations

statistical mechanics of arakawa s discretizations the results of statistical analysis of simulation data obtained from long time integrations of geophysical fluid models greatly depend on the conservation properties of the numerical discretization used this is illustrated for quasi geostrophic flow with topographic forcing for which a well established statistical mechanics exists statistical mechanical theories are constructed for the discrete dynamical systems arising from three discretizations due to arakawa arakawa computational design for long term numerical integration of the equations of fluid motion two dimensional incompressible flow part i j comput phys 1 1966 119 143 which conserve energy enstrophy or both numerical experiments with conservative and projected time integrators show that the statistical theories accurately explain the differences observed in statistics derived from the discretizations

statistically relevant conserved quantities for truncated quasigeostrophic flow systematic applications of ideas from equilibrium statistical mechanics lead to promising strategies for assessing the unresolved scales of motion in many problems in science and engineering a scientific debate over more than the last 25 years involves which conserved quantities among the formally infinite list are statistically relevant for the large scale equilibrium statistical behavior here this important issue is addressed by using suitable discrete numerical approximations for geophysical flows with many conserved quantities as a numerical laboratory the results of numerical experiments are presented here for these truncated geophysical flows with topography in a suitable regime these experiments establish that the integrated third power of potential vorticity besides the familiar constraints of energy circulation and enstrophy the integrated second power is statistically relevant in this regime for the coarse grained equilibrium statistical behavior at large scales furthermore the integrated higher powers of potential vorticity larger than three are statistically irrelevant for the large scale equilibrium statistical behavior in the examples studied here

link sign prediction by variational bayesian probabilistic matrix factorization with student t prior abstract in signed social networks link sign prediction refers to using the observed link signs to infer the signs of the remaining links which is important for mining and analyzing the evolution of social networks the widely used matrix factorization based approach bayesian probabilistic matrix factorization bmf assumes that the noise between the real and predicted entry is gaussian noise and the prior of latent features is multivariate gaussian distribution however gaussian noise model is sensitive to outliers and is not robust gaussian prior model neglects the differences between latent features that is it does not distinguish between important and non important features thus gaussian assumption based models perform poorly on real world sparse datasets to address these issues a novel variational bayesian probabilistic matrix factorization with student t prior model tbmf is proposed in this paper a univariate student t distribution is used to fit the prediction noise and a multivariate student t distribution is adopted for the prior of latent features due to the high kurtosis of student t distribution tbmf can select informative latent features automatically characterize long tail cases and obtain reasonable representations on many real world datasets experimental results show that tbmf improves the prediction performance significantly compared with the state of the art algorithms especially when the observed links are few

fuzzy logic based variable step size algorithm for blind delayed source separation convergence of blind delayed source separation algorithms which use constant learning rates is known to be slow we propose a fuzzy logic based approach to adaptively select the learning rates for estimates of delays and cross weights in the blind delayed source separation algorithm we make use of the state of independence of the separated outputs we also propose a performance index to measure the convergence of the blind delayed source separation algorithm simulation results show the improved performance of the proposed algorithm over the conventional delayed source separation algorithm under stationary as well as non stationary mixing conditions

stochastic complexity and modeling on demontre un theoreme fondamental qui donne une borne inferieure pour la longueur de code et donc pour les erreurs de prediction on definit les notions d information a priori et d information utile dans les donnees

minimal atmospheric finite mode models preserving symmetry and generalized hamiltonian structures abstract a typical problem with the conventional galerkin approach for the construction of finite mode models is to keep structural properties unaffected in the process of discretization we present two examples of finite mode approximations that in some respect preserve the geometric attributes inherited from their continuous models a three component model of the barotropic vorticity equation known as lorenz maximum simplification equations e n lorenz maximum simplification of the dynamic equations tellus 12 3 1960 243 254 and a six component model of the two dimensional rayleigh benard convection problem it is reviewed that the lorenz 1960 model respects both the maximal set of admitted point symmetries and an extension of the noncanonical hamiltonian form nambu form in a similar fashion it is proved that the famous lorenz 1963 model violates the structural properties of the saltzman equations and hence cannot be considered as the maximum simplification of the rayleigh benard convection problem using a six component truncation we show that it is again possible to retain both symmetries and the nambu representation in the course of discretization the conservative part of this six component reduction is related to the lagrange top equations dissipation is incorporated using a metric tensor

characterizing the response of pet and fmri data using multivariate linear models this paper presents a new method for characterizing brain responses in both pet and fmri data the aim is to capture the correlations between the scans of an experiment and a set of external predictor variables that are thought to affect the scans such as type intensity or shape of stimulus response its main feature is a canonical variates analysis cva of the estimated effects of the predictors from a multivariate linear model mlm the advantage of this over current methods is that temporal correlations can be incorporated into the model making the mlm method suitable for fmri as well as pet data moreover tests for the presence of any correlation and inference about the number of canonical variates needed to capture that correlation can be based on standard multivariate statistics rather than simulations when applied to an fmri data set previously analyzed by another cva method the mlm method reveals a pattern of responses that is closer to that detected in an earlier non cva analysis

a new high order energy and enstrophy conserving arakawa like jacobian differential operator a new high order arakawa like method for the incompressible vorticity equation in two dimensions has been developed mimetic properties such as skew symmetry energy and enstrophy conservations for the semi discretization are proved for periodic problems using arbitrary high order summation by parts operators numerical simulations corroborate the theoretical findings

stochastic complexities of gaussian mixtures in variational bayesian approximation bayesian learning has been widely used and proved to be effective in many data modeling problems however computations involved in it require huge costs and generally cannot be performed exactly the variational bayesian approach proposed as an approximation of bayesian learning has provided computational tractability and good generalization performance in many applications the properties and capabilities of variational bayesian learning itself have not been clarified yet it is still unknown how good approximation the variational bayesian approach can achieve in this paper we discuss variational bayesian learning of gaussian mixture models and derive upper and lower bounds of variational stochastic complexities the variational stochastic complexity which corresponds to the minimum variational free energy and a lower bound of the bayesian evidence not only becomes important in addressing the model selection problem but also enables us to discuss the accuracy of the variational bayesian approach as an approximation of true bayesian learning

improving on the james stein positive part estimator the purpose of this paper is to give an explicit estimator dominating the positive part james stein rule the james stein estimator improves on the usual estimator x of a multivariate normal mean vector if the dimension p of the problem is at least 3 it has been known since at least 1964 that the positive part version of this estimator improves on the james stein estimator brown s 1971 results imply that the positive part version is itself inadmissible although this result was assumed to be true much earlier explicit improvements however have not previously been found indeed 1988 results of bock and of brown imply that no estimator dominating the positive part estimator exists whose unbiased estimator of risk is uniformly smaller than that of the positive part estimator

a variable step vs adaptive filter algorithm in recent work a new version of an lms algorithm has been developed which implements a variable feedback constant for each weight of an adaptive transversal filter this technique has been called the vs variable step algorithm and is an extension of earlier ideas in stochastic approximation for varying the step size in the method of steepest descents the method may be implemented in hardware with only modest increases in complexity approx 15 percent over the lms widrow hoff algorithm it is shown that an upper bound for the convergence time is the classical mean square error time constant and examples are given to demonstrate that for broad signal classes both narrow band and broad band the convergence time is reduced by a factor of up to 50 in noise canceller applications for the proper selection of variable step parameters finally the vs algorithm is applied to an iir filter and simulations are presented for applications of the vs fir and iir adaptive filters

bayesian group sparse modeling and variational inference in this paper we present a general class of multivariate priors for group sparse modeling within the bayesian framework we show that special cases of this class correspond to multivariate versions of several classical priors used for sparse modeling hence this general prior formulation is helpful in analyzing the properties of different modeling approaches and their connections we derive the estimation procedures with these priors using variational inference for fully bayesian estimation in addition we discuss the differences between the proposed inference and deterministic inference approaches with these priors finally we show the flexibility of this modeling by considering several extensions such as multiple measurements within group correlations and overlapping groups

theoretical analysis of bayesian matrix factorization recently variational bayesian vb techniques have been applied to probabilistic matrix factorization and shown to perform very well in experiments in this paper we theoretically elucidate properties of the vb matrix factorization vbmf method through finite sample analysis of the vbmf estimator we show that two types of shrinkage factors exist in the vbmf estimator the positive part james stein pjs shrinkage and the trace norm shrinkage both acting on each singular component separately for producing low rank solutions the trace norm shrinkage is simply induced by non flat prior information similarly to the maximum a posteriori map approach thus no trace norm shrinkage remains when priors are non informative on the other hand we show a counter intuitive fact that the pjs shrinkage factor is kept activated even with flat priors this is shown to be induced by the non identifiability of the matrix factorization model that is the mapping between the target matrix and factorized matrices is not one to one we call this model induced regularization we further extend our analysis to empirical bayes scenarios where hyperparameters are also learned based on the vb free energy throughout the paper we assume no missing entry in the observed matrix and therefore collaborative filtering is out of scope

algebraic analysis for nonidentifiable learning machines this article clarifies the relation between the learning curve and the algebraic geometrical structure of a nonidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points by using a concept in algebraic analysis we rigorously prove that the bayesian stochastic complexity or the free energy is asymptotically equal to 1 log n m1 1 log log n constant where n is the number of training samples and 1 and m1 are the rational number and the natural number which are determined as the birational invariant values of the singularities in the parameter space also we show an algorithm to calculate 1 and m1 based on the resolution of singularities in algebraic geometry in regular statistical models 2 1 is equal to the number of parameters and m1 1 whereas in nonregular models such as multilayer networks 2 1 is not larger than the number of parameters and m1 1 since the increase of the stochastic complexity is equal to the learning curve or the generalization error the nonidentifiable learning machines are better models than the regular ones if bayesian ensemble learning is applied

computational design for long term numerical integration of the equations of fluid motion the integral constraints on quadratic quantities of physical importance such as conservation of mean kinetic energy and mean square vorticity will not be maintained in finite difference analogues of the equation of motion for two dimensional incompressible flow unless the finite difference jacobian expression for the advection term is restricted to a form which properly represents the interaction between grid points as derived in this paper it is shown that the derived form of the finite difference jacobian prevents nonlinear computational instability and thereby permits long term numerical integrations

cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents abstract for random matrix models the parameter estimation based on the traditional likelihood functions is not straightforward in particular when we have only one sample matrix we introduce a new parameter optimization method for random matrix models which works even in such a case the method is based on the spectral distribution instead of the traditional likelihood in the method the cauchy noise has an essential role because the free deterministic equivalent which is a tool in free probability theory allows us to approximate the spectral distribution perturbed by cauchy noises by a smooth and accessible density function moreover we study an asymptotic property of determination gap which has a similar role as generalization gap besides we propose a new dimensionality recovery method for the signal plus noise model and experimentally demonstrate that it recovers the rank of the signal part even if the true rank is not small it is a simultaneous rank selection and parameter estimation procedure

on the statistical analysis of dirty pictures may 7th 1986 professor a f m smith in the chair summary a continuous two dimensional region is partitioned into a fine rectangular array of sites or pixels each pixel having a particular colour belonging to a prescribed finite set the true colouring of the region is unknown but associated with each pixel there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model the aim is to reconstruct the true scene with the additional knowledge that pixels close together tend to have the same or similar colours in this paper it is assumed that the local characteristics of the true scene can be represented by a nondegenerate markov random field such information can be combined with the records by bayes theorem and the true scene can be estimated according to standard criteria however the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field thus a simple iterative method of reconstruction is proposed which does not depend on these large scale characteristics the method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field some complications including parameter estimation are discussed potential applications are mentioned briefly

shrinkage estimators for covariance matrices estimation of covariance matrices in small samples has been studied by many authors standard estimators like the unstructured maximum likelihood estimator ml or restricted maximum likelihood reml estimator can be very unstable with the smallest estimated eigenvalues being too small and the largest too big a standard approach to more stably estimating the matrix in small samples is to compute the ml or reml estimator under some simple structure that involves estimation of fewer parameters such as compound symmetry or independence however these estimators will not be consistent unless the hypothesized structure is correct if interest focuses on estimation of regression coefficients with correlated or longitudinal data a sandwich estimator of the covariance matrix may be used to provide standard errors for the estimated coefficients that are robust in the sense that they remain consistent under misspecification of the covariance structure with large matrices however the inefficiency of the sandwich estimator becomes worrisome we consider here two general shrinkage approaches to estimating the covariance matrix and regression coefficients the first involves shrinking the eigenvalues of the unstructured ml or reml estimator the second involves shrinking an unstructured estimator toward a structured estimator for both cases the data determine the amount of shrinkage these estimators are consistent and give consistent and asymptotically efficient estimates for regression coefficients simulations show the improved operating characteristics of the shrinkage estimators of the covariance matrix and the regression coefficients in finite samples the final estimator chosen includes a combination of both shrinkage approaches i e shrinking the eigenvalues and then shrinking toward structure we illustrate our approach on a sleep eeg study that requires estimation of a 24 x 24 covariance matrix and for which inferences on mean parameters critically depend on the covariance estimator chosen we recommend making inference using a particular shrinkage estimator that provides a reasonable compromise between structured and unstructured estimators

finite mode analogs of 2d ideal hydrodynamics coadjoint orbits and local canonical structure abstract the algebraic finite mode hydrodynamic type systems that have o n integrals of motion for o n n modes and are intrinsically connected with two dimensional ideal fluid flows are studied special emphasis is given on the structure of their phase space and the problem of finding proper dynamical variables for its description in connection with appropriate problems for the original hydrodynamic equations various possibilities for embedding in the space of canonical variables are demonstrated on the simplest example of the su 3 model giving the finite dimensional analogs of clebsch variables

explicit parallel poisson integration of point vortices on the sphere point vortex models are frequently encountered in conceptual studies in geophysical fluid dynamics but also in practical applications for instance in aeronautics in spherical geometry the motion of vortex centres is governed by a dynamical system with a known poisson structure we construct poisson integration methods for these dynamics by splitting the hamiltonian into its constituent vortex pair terms from backward error analysis the method is formally known to provide solutions to a modified poisson system with the correct bracket but with a modified hamiltonian function different orderings of the pairwise interactions are considered and also used for the construction of higher order methods the energy and momentum conservation of the splitting schemes is demonstrated for several test cases for particular orderings of the pairwise interactions the schemes allow scalable parallelization this results in a linear as opposed to quadratic scaling of computation time with system size when scaling the number of processors accordingly

global analytic solution of fully observed variational bayesian matrix factorization the variational bayesian vb approximation is known to be a promising approach to bayesian estimation when the rigorous calculation of the bayes posterior is intractable the vb approximation has been successfully applied to matrix factorization mf offering automatic dimensionality selection for principal component analysis generally finding the vb solution is a nonconvex problem and most methods rely on a local search algorithm derived through a standard procedure for the vb approximation in this paper we show that a better option is available for fully observed vbmf the global solution can be analytically computed more specifically the global solution is a reweighted svd of the observed matrix and each weight can be obtained by solving a quartic equation with its coefficients being functions of the observed singular value we further show that the global optimal solution of empirical vbmf where hyperparameters are also learned from data can also be analytically computed we illustrate the usefulness of our results through experiments in multi variate analysis

symplectic integration of hamiltonian wave equations the numerical integration of a wide class of hamiltonian partial differential equations by standard symplectic schemes is discussed with a consistent hamiltonian approach we discretize the hamiltonian and the poisson structure separately then form the the resulting ode s the stability accuracy and dispersion of different explicit splitting methods are analyzed and we give the circumstances under which the best results can be obtained in particular when the hamiltonian can be split into linear and nonlinear terms many different treatments and examples are compared

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

parameterization and bayesian modeling progress in statistical computation often leads to advances in statistical modeling for example it is surprisingly common that an existing model is reparameterized solely for computational purposes but then this new configuration motivates a new family of models that is useful in applied statistics one reason why this phenomenon may not have been noticed in statistics is that reparameterizations do not change the likelihood in a bayesian framework however a transformation of parameters typically suggests a new family of prior distributions we discuss examples in censored and truncated data mixture modeling multivariate imputation stochastic processes and multilevel models