measuring time frequency information content using the renyi entropies the generalized entropies of renyi inspire new measures for estimating signal information and complexity in the time frequency plane when applied to a time frequency representation tfr from cohen s class or the affine class the renyi entropies conform closely to the notion of complexity that we use when visually inspecting time frequency images these measures possess several additional interesting and useful properties such as accounting and cross component and transformation invariances that make them natural for time frequency analysis this paper comprises a detailed study of the properties and several potential applications of the renyi entropies with emphasis on the mathematical foundations for quadratic tfrs in particular for the wigner distribution we establish that there exist signals for which the measures are not well defined

maximum entropy and bayesian data analysis entropic prior distributions the problem of assigning probability distributions which reflect the prior information available about experiments is one of the major stumbling blocks in the use of bayesian methods of data analysis in this paper the method of maximum relative entropy me is used to translate the information contained in the known form of the likelihood into a prior distribution for bayesian inference the argument is inspired and guided by intuition gained from the successful use of me methods in statistical mechanics for experiments that cannot be repeated the resulting entropic prior is formally identical with the einstein fluctuation formula for repeatable experiments however the expected value of the entropy of the likelihood turns out to be relevant information that must be included in the analysis the important case of a gaussian likelihood is treated in detail

gradient descent learning in and out of equilibrium relations between the off thermal equilibrium dynamical process of on line learning and the thermally equilibrated off line learning are studied for potential gradient descent learning the approach of opper to study on line bayesian algorithms is used for potential based or maximum likelihood learning we look at the on line learning algorithm that best approximates the off line algorithm in the sense of least kullback leibler information loss the closest on line algorithm works by updating the weights along the gradient of an effective potential which is different from the parent off line potential a few examples are analyzed and the origin of the potential annealing is discussed

the distance of an eigenvector to a krylov subspace and the convergence of the arnoldi method for eigenvalue problems abstract we study the distance of an eigenvector of a diagonalizable matrix a to the krylov subspace generated from a and a given starting vector v this distance is involved in studies of the convergence of the arnoldi method for computing eigenvalues contrary to the previous studies on this problem we provide closed form expressions for this distance in terms of the eigenvalues and eigenvectors of a as well as the components of v in the eigenvector basis the formulas simplify when the matrix a is normal for a non normal we derive upper and lower bounds that are simpler than the exact expressions we also show how to generate starting vectors such that the distance to the krylov subspace is equal to the worst possible case

performance of the bayesian online algorithm for the perceptron in this letter we derive continuum equations for the generalization error of the bayesian online algorithm bona for the one layer perceptron with a spherical covariance matrix using the rosenblatt potential and show by numerical calculations that the asymptotic performance of the algorithm is the same as the one for the optimal algorithm found by means of variational methods with the added advantage that the bona does not use any inaccessible information during learning

banach lie poisson spaces and reduction the category of banach lie poisson spaces is introduced and studied it is shown that the category of w algebras can be considered as one of its subcategories examples and applications of banach lie poisson spaces to quantization and integration of hamiltonian systems are given the relationship between classical and quantum reduction is discussed

an analysis of the rayleigh ritz method for approximating eigenspaces this paper concerns the rayleigh ritz method for computing an approximation to an eigenspace x of a general matrix a from a subspace w that contains an approximation to x the method produces a pair n x that purports to approximate a pair l x where x is a basis for x and ax xl in this paper we consider the convergence of n x as the sine e of the angle between x and w approaches zero it is shown that under a natural hypothesis called the uniform separation condition the ritz pairs n x converge to the eigenpair l x when one is concerned with eigenvalues and eigenvectors one can compute certain refined ritz vectors whose convergence is guaranteed even when the uniform separation condition is not satisfied an attractive feature of the analysis is that it does not assume that a has distinct eigenvalues or is diagonalizable

generalizations of the field of values useful in the study of polynomial functions of a matrix for a given square matrix a and positive integer k we consider sets in the complex plane satisfying p a maxz p z for all polynomials p of degree k or less the largest such set referred to as the polynomial numerical hull of degree k was introduced by o nevanlinna convergence of iterations for linear equations birkhauser basel 1993 and a number of properties of this set were derived for both matrices and linear operators we give several equivalent characterizations of the polynomial numerical hull of degree k and we actually compute these sets for several matrices for k 1 this set is just the field of values of a and for k m where m is the degree of the minimal polynomial of a it is the spectrum of a for 1 k m these sets are intermediate between the field of values and the spectrum and sometimes resemble pseudospectra

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

estimating the entropy of a signal with applications we present a new estimator of the entropy of continuous signals we model the unknown probability density of data in the form of an ar spectrum density and use regularized long ar models to identify the ar parameters we then derive both an analytical expression and a practical procedure for estimating the entropy from sample data we indicate how to incorporate recursive and adaptive features in the procedure we evaluate and compare the new estimator with other estimators based on histograms kernel density models and order statistics finally we give several examples of applications an adaptive version of our entropy estimator is applied to detection of law changes blind deconvolution and source separation

information content of signals using correlation function expansions of the entropy formally exact series expressions are derived for the entropy information content of a time series or signal by making systematic expansions for the higher order correlation functions using generalized kirkwood and markov superpositions termination of the series after two or three terms provides tractable and accurate approximations for calculating the entropy signals generated by a gaussian random process are simulated using lorentzian and gaussian spectral densities exponential and gaussian covariance functions and the entropy is calculated as a function of the correlation length the validity of the truncated kirkwood expansion is restricted to weakly correlated signals whereas the truncated markov expansion is uniformly accurate the leading two terms yield the entropy exactly in the limits of both weak and strong correlations the concept of entropy for a continuous signal is explored in detail and it is shown that it depends upon the level of digitization and the frequency of sampling the limiting forms are analyzed for a continuous signal with exponentially decaying covariance for which explicit results can be obtained explicit results are also obtained for the binary discrete case that is isomorphic to the ising spin lattice model

bracket formulation of diffusion convection equations abstract an abstract diffusion convection equation is introduced it is shown that many well known and experimentally tested time evolution equations of physical systems e g the boltzmann and the enskog kinetic equations and the navier stokes fourier fluid mechanics equations are identical to or well approximated by realizations of the abstract diffusion convection equation the physical content of the abstract diffusion convection equation is compatibility with equilibrium thermodynamics it means that properties of solutions of the abstract diffusion convection equation are mathematical formulations of the experience on which equilibrium thermodynamics is based the requirement that all admissible time evolution equations are realizations of the abstract diffusion convection equation is used to find admissible closures in the bbgky and the maxwell grad hierarchies and admissible constitutive relations in generalized hydrodynamics

waveform information from quantum mechanical entropy although the entropy of a given signal type waveform is technically zero it is nonetheless desirable to use entropic measures to quantify the associated information several such prescriptions have been advanced in the literature but none are generally successful here we report that the fourier conjugated total entropy associated with quantum mechanical probabilistic amplitude functions pafs is a meaningful measure of information in non probabilistic real waveforms with either the waveform itself or its normalized analytic representation acting in the role of the paf detailed numerical calculations are presented for both adaptations showing the expected informatic behaviours in a variety of rudimentary scenarios particularly noteworthy are the sensitivity to the degree of randomness in a sequence of pulses and potential for detection of weak signals

computing the discrete time analytic signal via fft starting with a real valued n point discrete time signal frequency domain algorithms are provided for computing 1 the complex valued standard n point discrete time analytic signal of the same sample rate 2 the complex valued decimated n 2 point discrete time analytic signal of half the original sample rate and 3 the complex valued interpolated nm point discrete time analytic signal of m times the original sample rate special adjustment of the transform end points are shown to be necessary in order to generate proper discrete time analytic signals

essential equivalence of the general equation for the nonequilibrium reversible irreversible coupling generic and steepest entropy ascent models of dissipation for nonequilibrium thermodynamics by reformulating the steepest entropy ascent sea dynamical model for nonequilibrium thermodynamics in the mathematical language of differential geometry we compare it with the primitive formulation of the general equation for the nonequilibrium reversible irreversible coupling generic model and discuss the main technical differences of the two approaches in both dynamical models the description of dissipation is of the entropy gradient type sea focuses only on the dissipative i e entropy generating component of the time evolution chooses a sub riemannian metric tensor as dissipative structure and uses the local entropy density field as potential generic emphasizes the coupling between the dissipative and nondissipative components of the time evolution chooses two compatible degenerate structures poisson and degenerate co riemannian and uses the global energy and entropy functionals as potentials as an illustration we rewrite the known generic formulation of the boltzmann equation in terms of the square root of the distribution function adopted by the sea formulation we then provide a formal proof that in more general frameworks whenever all degeneracies in the generic framework are related to conservation laws the sea and generic models of the dissipative component of the dynamics are essentially interchangeable provided of course they assume the same kinematics as part of the discussion we note that equipping the dissipative structure of generic with the leibniz identity makes it automatically sea on metric leaves

self averaging and on line learning conditions are given under which one may prove that the stochastic dynamics of on line learning can be described by the deterministic evolution of a finite set of order parameters in the thermodynamic limit a global constraint on the average magnitude of the increments in the stochastic process is necessary to ensure self averaging in the absence of such a constraint convergence may only be in probability

further analysis of the arnoldi process for eigenvalue problems this paper takes another look at the convergence analysis of the arnoldi procedure for solving non hermitian eigenvalue problems two main viewpoints are put in contrast the first exploits the eigenbasis when there is one and relies on classical min max approximation theory results the second approach relies on the schur factorization its aim is to link the convergence analysis of the arnoldi process for eigenvalue problems to that of the generalized minimal residual iterations gmres for which much is known

the rosenblatt bayesian algorithm learning in a nonstationary environment in this letter we study online learning in neural networks nns obtained by approximating bayesian learning the approach is applied to gibbs learning with the rosenblatt potential in a nonstationary environment the online scheme is obtained by the minimization maximization of the kullback leibler divergence cross entropy between the true posterior distribution and the parameterized one the complexity of the learning algorithm is further decreased by projecting the posterior onto a gaussian distribution and imposing a spherical covariance matrix we study in detail the particular case of learning linearly separable rules in the case of a fixed rule we observe an asymptotic generalization error egpropalpha 1 for both the spherical and the full covariance matrix approximations however in the case of drifting rule only the full covariance matrix algorithm shows a good performance this good performance is indeed a surprise since the algorithm is obtained by projecting without the benefit of the extra information on drifting

iterative methods for transmission eigenvalues transmission eigenvalues have important applications in inverse scattering theory they can be used to obtain useful information of the physical properties such as the index of refraction of the scattering target despite considerable effort devoted to the existence and estimation for the transmission eigenvalues the numerical treatment is limited since the problem is nonstandard classical finite element methods result in non hermitian matrix eigenvalue problems in this paper we focus on the computation of a few lowest transmission eigenvalues which are of practical importance instead of a non hermitian problem we work on a series of generalized hermitian problems we first use a fourth order reformulation of the transmission eigenproblem to construct functions involving an associated generalized eigenvalue problem the roots of these functions are the transmission eigenvalues then we apply iterative methods to compute the transmission eigenvalues we show the convergence of the numerical schemes the effectiveness of the methods is demonstrated using various numerical examples

optimal generalization in perceptions a new learning algorithm for the one layer perceptron is presented it aims to maximize the generalization gain per example analytical results are obtained for the case of single presentation of each example the weight attached to a hebbian term is a function of the expected stability of the example in the teacher perceptron this leads to the obtention of upper bounds for the generalization ability this scheme can be iterated and the results of numerical simulations show that it converges within errors to the theoretical optimal generalization ability of the bayes algorithm analytical and numerical results for an algorithm with maximized generalization in the learning strategy with selection of examples are obtained and it is proved that as expected orthogonal selection is optimal exponential decay of the generalization error is obtained for the single presentation of selected examples

convergence of polynomial restart krylov methods for eigenvalue computations krylov subspace methods have led to reliable and effective tools for resolving large scale non hermitian eigenvalue problems since practical considerations often limit the dimension of the approximating krylov subspace modern algorithms attempt to identify and condense significant components from the current subspace encode them into a polynomial filter and then restart the krylov process with a suitably refined starting vector in effect polynomial filters dynamically steer low dimensional krylov spaces toward a desired invariant subspace through their action on the starting vector the spectral complexity of nonnormal matrices makes convergence of these methods difficult to analyze and these effects are further complicated by the polynomial filter process the principal object of study in this paper is the angle an approximating krylov subspace forms with a desired invariant subspace convergence analysis is posed in a geometric framework that is robust to eigenvalue ill conditioning yet remains relatively uncluttered the bounds described here suggest that the sensitivity of desired eigenvalues exerts little influence on convergence provided the associated invariant subspace is well conditioned ill conditioning of unwanted eigenvalues plays an essential role this framework also gives insight into the design of effective polynomial filters numerical examples illustrate the subtleties that arise when restarting non hermitian iterations

generalization of the dynamical lack of fit reduction from generic to generic the lack of fit statistical reduction developed and formulated first by bruce turkington is a general method taking liouville equation for probability density detailed level and transforming it to reduced dynamics of projected quantities less detailed level in this paper the method is generalized the hamiltonian liouville equation is replaced by an arbitrary hamiltonian evolution combined with gradient dynamics generic the boltzmann entropy is replaced by an arbitrary entropy and the kinetic energy by an arbitrary energy the gradient part is a generalized gradient dynamics generated by a dissipation potential the reduced evolution of the projected state variables is shown to preserve the generic structure of the original detailed level evolution the dissipation potential is obtained by solving a hamilton jacobi equation in summary the lack of fit reduction can start with generic and obtain generic for the reduced state variables

fields of values and iterative methods abstract the performance of an iterative scheme to solve x t x c t c n n c c n is often judged by spectral properties of t if t is not normal it is however well known that only conclusions about the assymptotic behavior of an iterative method can be drawn from spectral information to anticipate the progress of the iteration after a finite number of steps the knowledge of the eigenvalues alone is often useless in addition the spectrum of t may be highly sensitive to perturbations if t is not normal an iterative method which on the basis of some spectral information is predicted to converge rapidly for t may well diverge if t is slightly perturbed in practice the convergence of the iteration x m t x m 1 c is therefore frequently measured by some norm verbar t verbar rather than by the spectral radius t but apart from the fact that norms lead to error estimates which are often too pessimistic they cannot be used to analyze more general schemes such as e g the chebyshev iterative methods here we discuss another tool to analyze the behavior of an iterative method namely the field of values w t the collection of all rayleigh quotients of t w t contains the eigenvalues of t and the numerical radius t max z w t z defines a norm on c n n the field of values represents therefore an intermediate concept to judge an iterative scheme by it is related to the spectral approach but has also certain norm properties