coercive inequalities for gibbs measures abstract we prove the generalized nash and logarithmic nash inequalities for gibbs measures with dirichlet form associated to the kawasaki dynamics

ergodic primal convergence in dual subgradient schemes for convex programming lagrangean dualization and subgradient optimization techniques are frequently used within the field of computational optimization for finding approximate solutions to large structured optimization prob lems the dual subgradient scheme does not automatically produce primal feasible solutions there is an abundance of techniques for computing such solutions via penalty functions tangential approximation schemes or the solution of auxiliary primal programs all of which require a fair amount of computational effort we consider a subgradient optimization scheme applied to a lagrangean dual formulation of a convex program and construct at minor cost an ergodic sequence of subproblem solutions which converges to the primal solution set numerical experiments performed on a traffic equilibrium assignment problem under road pricing show that the computation of the ergodic sequence results in a considerable improvement in the quality of the primal solutions obtained compared to those generated in the basic subgradient scheme

minimum expected length of fixed to variable lossless compression without prefix constraints the minimum expected length for fixed to variable length encoding of an n block memoryless source with entropy h grows as nh o 1 where the term o 1 lies between 0 and 1 however this well known performance is obtained under the implicit constraint that the code assigned to the whole n block is a prefix code dropping the prefix constraint which is rarely necessary at the block level we show that the minimum expected length for a finite alphabet memoryless source with known distribution grows as nh 1 2 log n o 1 unless the source is equiprobable we also refine this result up to o 1 for those memoryless sources whose log probabilities do not reside on a lattice

minimum cost distributed source coding over a network this paper considers the problem of transmitting multiple compressible sources over a network at minimum cost the aim is to find the optimal rates at which the sources should be compressed and the network flows using which they should be transmitted so that the cost of the transmission is minimal we consider networks with capacity constraints and linear cost functions the problem is complicated by the fact that the description of the feasible rate region of distributed source coding problems typically has a number of constraints that is exponential in the number of sources this renders general purpose solvers inefficient we present a framework in which these problems can be solved efficiently by exploiting the structure of the feasible rate regions coupled with dual decomposition and optimization techniques such as the subgradient method and the proximal bundle method

reversed version of a generalized sharp holder s inequality and its applications in this paper we give a reversed version of a generalized sharp holder s inequality which is due to wu the results are then used to improve beckenbach s inequality and minkowski s inequality moreover as an application in information theory we present a refinement of singh s inequality which is one generalized shannon s inequality

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

survival exponential entropies the multivariate survival function of a random vector x is used to define a broad class of entropy measures several properties of the proposed class are studied and explicit expressions of the measures derived for specific probabilistic models the cumulative residual entropy introduced by rao et al and wang et al is a particular case of the proposed class of measures

a unified characterization of generalized information and certainty measures abstract in this paper we consider the axiomatic characterization of information and certainty measures in a unified way we present the general axiomatic system which captures the common properties of a large number of the measures previously considered by numerous authors we provide the corresponding characterization theorems and define a new generalized measure called the inforcer which is the quasi linear mean of the function associated with the event probability following the general composition law in particular we pay attention to the polynomial composition and the corresponding polynomially composable inforcer measure the most common measures appearing in literature can be obtained by specific choice of parameters appearing in our generic measure and they are listed in tables

maximum a posteriori estimation of time varying arma processes from noisy observations the estimation of the parameters of discrete time autoregressive moving average arms processes observed in white noise is considered a class of time varying arma processes in which the parameter process is the output of a known linear system driven by white gaussian noise is examined the maximum a posteriori map estimator is defined for the trajectory of the parameter s random process a two step estimate and maximize em based e step and m step iterative algorithm is derived the posterior probability of the parameters is increased in each iteration and convergence to stationary points of the posterior probability is guaranteed each iteration involves two linear systems and is easily implemented it is shown that similar results can be obtained for a wide class of parameter estimation problems

on quantizer distortion and the upper bound for exponential entropy a sharp upper bound is derived for the exponential entropy in the class of absolutely continuous distributions with specific standard deviation and an exact description of the extremal distributions this result is interpreted as determining the least favorable cases for certain methods of quantization of analog sources it is known that for a large class of quantizers both zero memory and vector the rth power distortion as well as some other distortion criteria are bounded below by a constant depending on r multiplied by a certain integral of the source s probability density it is pointed out that this bound can be rewritten in terms of the exponential entropy the exponential entropy measures the quantitative extent or range of the source distribution this fact gives a physical interpretation of the indicated limits of quantizer performance further elucidated by the main result

application of holder s inequality in information theory two coding theorems for two useful measures have been proved by using holder s inequality the measures considered for decipherable codes are h p u 1 1 log i 1 n p i u i i 1 n p i u i for power distribution and useful r norm information measure h r p u r r 1 1 i 1 n u i p i r i 1 n u i p i 1 r some particular cases such as gurdial and pessoa jciss 2 4 1977 150 longo siam j appl math 30 4 1976 239 guiasu and picard c r acad sci paris 273 1971 248 campbell inf control 8 1965 423 aczel inst nazionale di alta matematica symposia mathematica xv 1975 153 boekee and lubbe inf control 45 1980 136 have been presented

new bounds on the expected length of optimal one to one codes in this correspondence we consider one to one encodings for a discrete memoryless source which are one shot encodings associating a distinct codeword with each source symbol such encodings could be employed when only a single source symbol rather than a sequence of source symbols needs to be transmitted for example such a situation can arise when the last message must be acknowledged before the next message can be transmitted we consider two slightly different types of one to one encodings depending on whether the empty codeword is used or not and obtain lower and upper bounds on the expected length of optimal one to one codes we first give an extension of a known tight lower bound on the expected length of optimal one to one codes for the case that the the size of the source alphabet is finite and partial information about the source symbol probabilities is available as expected our lower bound is no less than the previously known lower bound obtained without side information about the source symbol probabilities we then consider the case that the source entropy is available and derive arbitrarily tight lower bounds on the expected length of optimal one to one codes we also derive arbitrarily tight lower bounds for the case that the source entropy and the probability of the most likely source symbol are available finally given that the probability of the most likely source symbol is available we obtain an upper bound on the expected length of optimal one to one codes our upper bound is tighter than the best upper bound known in the literature

tight lower bounds for optimum code length corresp a new lower bound for the mean code length of all one to one codes for a random variable with n outcomes is derived the bound which is tight improves an earlier one due to leung yan cheong and cover another bound for one to one codes for binary information sources is derived

stochastic resonance for nonlinear sensors with saturation we analyze the transmission of a noisy signal by sensor devices which are linear for small inputs and saturate at large inputs large information carrying signals are thus distorted in their transmission we demonstrate conditions where addition of noise to such large input signals can reduce the distortion that they undergo in the transmission this is established for periodic as well as aperiodic and random information carrying signals various measures characterizing the transmission such as signal to noise ratio input output cross correlation and mutual information are shown improvable by addition of noise these results constitute another instance of the nonlinear phenomenon of stochastic resonance where addition of noise enhances the signal

entropy dissipation estimates in a zero range dynamics we study the exponential decay of relative entropy functionals for zero range processes on the complete graph for the standard model with rates increasing at infinity we prove entropy dissipation estimates uniformly over the number of particles and the number of vertices

the relative information generating function abstract the aim of the paper is to introduce the relative information generating function whose derivatives give well known statistical indices such as the kullback leibler divergence between two probability distributions and watanabe s measure of interdependence it contains golomb s generating function as a particular case and takes simple forms for the binomial and the poisson distributions neither of which fits into golomb s model besides its unifying role the relative information generating function suggests new information indices as for instance the standard deviation of the variation of information

spectral gap and logarithmic sobolev inequality for kawasaki and glauber dynamics we prove that the spectral gap of the kawasaki dynamics shrink at the rate of 1 l2 for cubes of sizel provided that some mixing conditions are satisfied we also prove that the logarithmic sobolev inequality for the glauber dynamics in standard cubes holds uniformly in the size of the cube if the dobrushin shlosman mixing condition holds for standard cubes

stochastic resonance in the mutual information between input and output spike trains of noisy central neurons abstract within the framework of information theory we investigate the phenomenon of stochastic resonance in a realistic stochastic model of central neurons the essential point is that for central neurons the input is not a continuous signal but a train of spikes which has in general a non periodic distribution the behavior of an individual neuron is modeled by a combination of an ornstein uhlenbeck diffusion process and a jump process corresponding to a discrete input spike train the input spike train is assumed to be given by a poisson distribution if the parameters are such that the input spike train provokes a membrane potential that hovers under the threshold the noise given by the diffusive part of the neuron model induces a maximum in the mutual information between the input and output spike trains thus the stochastic resonance effect results in a strategy for efficient information transmission of subthreshold signals

noise enhanced parameter estimation this paper investigates the phenomenon of noise enhanced systems for a general parameter estimation problem when the estimator is fixed and known the estimation performance before and after the addition of noise are evaluated performance comparisons are made between the original estimators and noise enhanced estimators based on different criteria the form of the optimal noise probability density function pdf is determined the results are further extended to the general case where the noise is introduced to the system via a transformation for the case where the estimator is fixed and unknown approaches are also proposed to find the optimum noise finally two illustrative examples are presented where the performance comparison is made between the optimal noise modified estimator and gaussian noise modified estimator

spectral gap for zero range dynamics we give a lower bound on the spectral gap for symmetric zero range processes under some conditions on the rate function we show that the gap shrinks as n 2 independent of the density for the dynamics localized on a cube of size n d we follow the method outlined by lu and yau where a similar spectral gap is proved for kawasaki dynamics

on the parameters estimation of continuous time arma processes from noisy observations recently an iterative algorithm has been presented for estimating the parameters of partially observed continuous time processes 1 in this note we concentrate on continuous time arma processes observed in white noise a maximum a posteriori map estimator is defined for the trajectory of the parameters random process this approach enables the map estimation of randomly slowly varying parameters and extends the conventional treatment of time invariant parameters the iterative algorithm derived for the map estimation increases the posterior probability of the parameters in each iteration and converges to a stationary point of the posterior probability functional each iteration involves a standard linear smoother followed by a finite dimensional linear system and thus is easily implemented

bulk diffusion in a system with site disorder we consider a system of random walks in a random environment interacting via exclusion the model is reversible with respect to a family of disordered bernoulli measures assuming some weak mixing conditions it is shown that under diffusive scaling the system has a deterministic hydrodynamic limit which holds for almost every realization of the environment the limit is a nonlinear diffusion equation with diffusion coefficient given by a variational formula the model is nongradient and the method used is the long jump variation of the standard nongradient method which is a type of renormalization the proof is valid in all dimensions

identification and restoration of noisy blurred images using the expectation maximization algorithm a maximum likelihood approach to the blur identification problem is presented the expectation maximization algorithm is proposed to optimize the nonlinear likelihood function in an efficient way in order to improve the performance of the identification algorithm low order parametric image and blur models are incorporated into the identification method the resulting iterative technique simultaneously identifies and restores noisy blurred images

arma spectral estimation of time series with missing observations the problem of estimating the power spectral density of stationary time series when the measurements are not contiguous is considered a new autoregressive moving average arma method is proposed for this problem based on nonlinear optimization of a weighted squared error criterion the method can handle either regularly or randomly missing observations as a special case the method can handle the problem of missing sample covariances the computational complexity is modest compared to exact maximum likelihood estimation of the same parameters the performance of the algorithm is illustrated by some numerical examples and is shown to be statistically efficient in these cases

enhancement by noise in parallel arrays of sensors with power law characteristics an optimally tuned power law sensor is shown capable of amplifying the signal to noise ratio of a sine wave in gaussian white noise when associated in parallel arrays further improvement can be obtained with independent noises injected on these sensors this form of stochastic resonance in arrays obtained here with smooth threshold free nonlinearities yields signal to noise ratio gains above unity in a true regime of added noise for a sine wave in gaussian white noise along with a class of nonlinear devices with useful potentialities for noise aided information processing

hidden markov processes an overview of statistical and information theoretic aspects of hidden markov processes hmps is presented an hmp is a discrete time finite state homogeneous markov chain observed through a discrete time memoryless invariant channel in recent years the work of baum and petrie 1966 on finite state finite alphabet hmps was expanded to hmps with finite as well as continuous state spaces and a general alphabet in particular statistical properties and ergodic theorems for relative entropy densities of hmps were developed consistency and asymptotic normality of the maximum likelihood ml parameter estimator were proved under some mild conditions similar results were established for switching autoregressive processes these processes generalize hmps new algorithms were developed for estimating the state parameter and order of an hmp for universal coding and classification of hmps and for universal decoding of hidden markov channels these and other related topics are reviewed

evaluation of stochastic resonance based detectors of weak harmonic signals in additive white gaussian noise a thorough evaluation of stochastic resonance in the framework of statistical detection theory is presented both as a nonlinear signal preprocessor and as a detector the pertinent receiver operating characteristics are compared with those of the known statistically optimum detector using extensive monte carlo simulations parameter optimization and computational budget aspects are discussed

convergence to equilibrium of conservative particle systems on mathbb z bm d we consider the ginzburg landau process on the lattice zd whose potential is a bounded perturbation of the gaussian potential we prove that the decay rate to equilibrium in the variance sense is t d 2 up to a logarithmic correction for any function u with finite triple norm that is u x zd xu

optimal prefix codes for infinite alphabets with nonlinear costs let p p i be a measure of strictly positive probabilities on the set of nonnegative integers although the countable number of inputs prevents usage of the huffman algorithm there are nontrivial p for which known methods find a source code that is optimal in the sense of minimizing expected codeword length for some applications however a source code should instead minimize one of a family of nonlinear objective functions beta exponential means those of the form loga sigmaip i an i where n i is the length of the ith codeword and a is a positive constant applications of such minimizations include a novel problem of maximizing the chance of message receipt in single shot communications a 1 this paper introduces methods for finding codes optimal for such exponential means one method applies to geometric distributions while another applies to distributions with lighter tails the latter algorithm is applied to poisson distributions and both are extended to alphabetic codes as well as to minimizing maximum pointwise redundancy the aforementioned application of minimizing the chance of buffer overflow is also considered

nonlinear diffusion limit for a system with nearest neighbor interactions we consider a system of interacting diffusions the variables are to be thought of as charges at sites indexed by a periodic one dimensional lattice the diffusion preserves the total charge and the interaction is of nearest neighbor type with the appropriate scaling of lattice spacing and time a nonlinear diffusion equation is derived for the time evolution of the macroscopic charge density

logarithmic sobolev inequality for lattice gases with mixing conditions let mu lambda l lambda gc denote the grand canonical gibbs measure of a lattice gas in a cube of sizel with the chemical potential and a fixed boundary condition let mu lambda l n c be the corresponding canonical measure defined by conditioning mu lambda l lambda gc on sigma x in lambda eta x n consider the lattice gas dynamics for which each particle performs random walk with rates depending on near by particles the rates are chosen such that for everyn andl fixed mu lambda l n c is a reversible measure suppose that the dobrushin shlosman mixing conditions holds for mu l lambda gc forall chemical potentials we prove that smallint f log fd mu lambda l n c leqq const l 2 d sqrt f for any probability densityf with respect to mu lambda l n c here the constant is independent ofn orl andd denotes the dirichlet form of the dynamics the dependence onl is optimal

logarithmic sobolev inequality for generalized simple exclusion processes let be a probability measure on the set 0 1 r for some r and l a cube of width l in d denote by gc l the grand canonical product measure on the configuration space on l with as the marginal measure here the superscript indicates the grand canonical ensemble the canonical ensemble denoted by c l n is defined by conditioning gc l given the total number of particles to be n consider the exclusion dynamics where each particle performs random walk with rates depending only on the number of particles at the same site the rates are chosen such that for every n and l fixed the measure c l n is reversible we prove the logarithmic sobolev inequality in the sense that flogfd c l n for any probability density f with respect to c l n here the constant is independent of n or l and d denotes the dirichlet form of the dynamics the dependence on l is optimal

on the spectral gap of kawasaki dynamics under a mixing condition revisited we consider a conservative stochastic spin exchange dynamics which is reversible with respect to the canonical gibbs measure of a lattice gas model we assume that the corresponding grand canonical measure satisfies a suitable strong mixing condition we give an alternative and quite natural from the physical point of view proof of the famous lu yau result which states that the relaxation time in a box of side l scales like l2 we then show how to use such an estimate to prove a decay to equilibrium for local functions of the form 1 t e where e is positive and arbitrarily small and 12 for d 1 1 for d 2

minimum entropy of error principle in estimation abstract the principle of minimum error entropy estimation as found in the work of weidemann and stear is reformulated as a problem of finding optimum locations of probability densities in a given mixture such that the resulting differential entropy is minimized new results concerning the entropy lower bound are derived continuity of the entropy and attaining the minimum entropy are proved in the case where the mixture is finite some other examples and situations in particular that of symmetric unimodal densities are studied in more detail

diffusive scaling of the spectral gap for the dilute ising lattice gas dynamics below the percolation threshold we consider a conservative stochastic lattice gas dynamics reversible with respect to the canonical gibbs measure of the bond dilute ising model on d at inverse temperature when the bond dilution density p is below the percolation threshold we prove that for any particle density and any with probability one the spectral gap of the generator of the dyamics in a box of side l centered at the origin scales like l 2 such an estimate is then used to prove a decay to equilibrium for local functions of the form where e is positive and arbitrarily small and for d 1 1 for d 2 in particular our result shows that contrary to what happes for the glauber dynamics there is no dynamical phase transition when crosses the critical value c of the pure system

the renyi redundancy of generalized huffman codes huffman s algorithm gives optimal codes as measured by average codeword length and the redundancy can be measured as the difference between the average codeword length and shannon s entropy if the objective function is replaced by an exponentially weighted average then a simple modification of huffman s algorithm gives optimal codes the redundancy can now be measured as the difference between this new average and a renyi s 1961 generalization of shannon s entropy by decreasing some of the codeword lengths in a shannon code the upper bound on the redundancy given in the standard proof of the noiseless source coding theorem is improved the lower bound is improved by randomizing between codeword lengths allowing linear programming techniques to be used on an integer programming problem these bounds are shown to be asymptotically equal the results are generalized to the renyi case and are related to r g gallager s 1978 bound on the redundancy of huffman codes

new lower bounds on the cost of binary search trees abstract in this paper we provide new lower bounds on the cost c of binary search trees the bounds are expressed in terms of the entropy h of the probability distribution the number of elements and the probability that a search is successful most of our lower bounds are derived by means of a new technique which exploits the relation between trees and codes our lower bounds compare favorably with known limitations we also provide an achievable upper bound on the kraft sum generalized to the internal nodes of a tree this improves on a previous result

on jessen s inequality for convex functions ii on etablit quelques resultats lies a l integralite de b jessen pour les fonctions convexes

universal modeling and coding the problems arising in the modeling and coding of strings for compression purposes are discussed the notion of an information source that simplifies and sharpens the traditional one is axiomatized and adaptive and nonadaptive models are defined with a measure of complexity assigned to the models a fundamental theorem is proved which states that models that use any kind of alphabet extension are inferior to the best models using no alphabet extensions at all a general class of so called first in first out fifo arithmetic codes is described which require no alphabet extension devices and which therefore can be used in conjunction with the best models because the coding parameters are the probabilities that define the model their design is easy and the application of the code is straightforward even with adaptively changing source models

on one to one codes for memoryless cost channels a new twist to the classical problem of compressing a discrete source over noiseless channel alphabets where different code symbols may have different transmission costs is considered one to one or nonsingular codes map each letter from the source alphabet into a distinct codeword consisting of a string of channel alphabet letters they need not be uniquely decodable they have been studied for binary code alphabets with equal letter costs since the early 1970s and are extended here to memoryless cost channel alphabets the study of optimal one to one codes is linked in this paper to three other well known source coding problems the algorithm to construct the optimal one to one code is closely related to the varn code and the tunstall code these relationships and convexity arguments are used to establish some upper and lower bounds on the expected cost per source symbol of the optimal one to one code finally the average cost per symbol of the optimal one to one code is demonstrated to offer a new lower bound to the average cost per symbol of the optimal uniquely decodable code which is sometimes tighter than the classic bound from shannon s landmark 1948 paper

development of two new mean codeword lengths two new mean codeword lengths l a b and l b are defined and it is shown that these lengths satisfy desirable properties as a measure of typical codeword lengths consequently two new noiseless coding theorems subject to kraft s inequality have been proved further we have shown that the mean codeword lengths l 1 1 a b and l 1 1 b for the best one to one code not necessarily uniquely decodable are shorter than the mean codeword length l u d a b and l u d b respectively for the best uniquely decodable code by no more than log dlog dn 3 for d 2 moreover we have studied tighter bounds of l a b

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

some properties of generalized exponential entropies with applications to data compression abstract the scale of exponential entropies of order is introduced it unifies a number of notions of generalized differential entropies for continuous distributions the properties of this scale are derived from results in the theory of means we show that the scale can represent the extent of a probability distribution i e its degree of concentration with regard to a reference measure this fact has several consequences that are of interest in the theory of data or signal compression the aspects of data compression considered are the intrinsic extent of a probability distribution introduced by l l campbell and the properties of entropy series we investigate specially the exponential families of distributions in particular the miller thomas or generalized gaussian family of distributions

relaxation to equilibrium of conservative dynamics i zero range processes under mild assumptions we prove that for any local function u the decay rate to equilibrium in the variance sense of zero range dynamics on the d dimensional integer lattice is c u t d 2 o t d 2 the constant c u is computed explicitly

map symbol estimation on frequency flat rayleigh fading channels via a bayesian em algorithm in this paper an expectation maximization em technique for maximum a posteriori estimation is employed to devise novel soft in soft out algorithms for symbol detection over frequency flat rayleigh fading channels an application of these algorithms to iterative decoding of coded psk signals is proposed and some performance results are illustrated

information transmission in parallel threshold arrays suprathreshold stochastic resonance the information transmitted through a parallel summing array of noisy threshold elements with a common threshold is considered in particular using theoretical and numerical analysis a recently reported n g stocks phys rev lett 84 2310 2000 form of stochastic resonance termed suprathreshold stochastic resonance ssr is studied in detail ssr is observed to occur in arrays with two or more elements and unlike stochastic resonance sr in a single element gives rise to noise induced information gains that occur independent of the setting of the threshold or the size of the signal the transmitted information is maximized when all thresholds are set to coincide with the signal mean in this situation and for large arrays the noise can enhance performance up to approximately half the theoretical noiseless channel capacity the theory is tested against digital simulation

a new method of nonstationary time series analysis based on inhomogeneous ar equation we propose a new model for nonstationary time series analysis the model is of a noise contaminated signal of an ar system excited by a sequence of an input signal represented in terms of orthogonal functions we also propose an algorithm that enables us to estimate parameters of the ar part and the input signal simultaneously the models are finally evaluated by testing the recovery of an output signal examples of data analysis of the synthetic time series are shown for the ease in which the input signal is represented by a sequence of wavelets

noise enhanced capacity via stochastic resonance in an asymmetric binary channel a nonlinear system is considered where an aperiodic binary input signal is added to an arbitrarily distributed noise and compared to a fixed threshold to determine the binary output signal noise enhancement of the transmission of the aperiodic signal via stochastic resonance is demonstrated and studied in this nonlinear information channel the characterization developed goes up to the calculation of the information capacity of the channel defined as the maximal achievable input output transinformation occurring when the statistics of the input signal is matched to the noise it is then demonstrated that a regime exists where the information capacity of the channel can be increased by means of an increase of the noise up to an optimal noise level where the capacity resonates at a maximum value the influence on this resonance of the noise distribution is also studied

a fast improvement to the em algorithm on its own terms the em algorithm is a numerical technique for the evaluation of maximum likelihood estimates for parameters describing incomplete data models it is easy to apply in many problems and is stable but slow the algorithm fails to provide a consistent estimator of the standard errors of the maximum likelihood estimates unless the additional analysis required by the louis method is performed newton type or other gradient methods are faster and provide error estimates but tend to be unstable and require the analytical evaluation of likelihoods to derive expressions for the score function and at least approximations to the fisher information matrix the purpose of this paper is to expand on a result by fisher that permits a unification of em methodology and newton methods the evaluation of the individual observation by observation score functions of the incomplete data is a byproduct of the application of the e step of the em algorithm once these become available the fisher information matrix may be consistently estimated and the m step may be replaced by a fast newton type step

diffusion of color in the simple exclusion process we prove a diffusion scaling limit for the macroscopic densities of colored particles performing the simply excluded random walk and relate this to the limiting behavior of a test particle in equilibrium

a universal data compression system a universal data compression algorithm is described which is capable of compressing long strings generated by a finitely generated source with a near optimum per symbol length without prior knowledge of the source this class of sources may be viewed as a generalization of markov sources to random fields moreover the algorithm does not require a working storage much larger than that needed to describe the source generating parameters

renyi entropy measure of noise aided information transmission in a binary channel this paper analyzes a binary channel by means of information measures based on the renyi entropy the analysis extends and contains as a special case the classic reference model of binary information transmission based on the shannon entropy measure the extended model is used to investigate further possibilities and properties of stochastic resonance or noise aided information transmission the results demonstrate that stochastic resonance occurs in the information channel and is registered by the renyi entropy measures at any finite order including the shannon order furthermore in definite conditions when seeking the renyi information measures that best exploit stochastic resonance then nontrivial orders differing from the shannon case usually emerge in this way through binary information transmission stochastic resonance identifies optimal renyi measures of information differing from the classic shannon measure a confrontation of the quantitative information measures with visual perception is also proposed in an experiment of noise aided binary image transmission

method and apparatus for denoising and deverberation using variational inference and strong speech models a probability distribution for speech model parameters such as auto regression parameters is used to identify a distribution of denoised values from a noisy signal under one embodiment the probability distributions of the speech model parameters and the denoised values are adjusted to improve a variational inference so that the variational inference better approximates the joint probability of the speech model parameters and the denoised values given a noisy signal in some embodiments this improvement is performed during an expectation step in an expectation maximization algorithm the statistical model can also be used to identify an average spectrum for the clean signal and this average spectrum may be provided to a speech recognizer instead of the estimate of the clean signal

poincare and logarithmic sobolev inequality for ginzburg landau processes in random environment we consider reversible conservative ginzburg landau processes in a random environment whose potential are bounded perturbations of the gaussian potential evolving on a d dimensional cube of length l we prove in all dimensions that the spectral gap of the generator and the logarithmic sobolev constant are of order l 2 almost surely with respect to the environment

estimating and improving the signal to noise ratio of time series by symbolic dynamics we investigate the effect of symbolic encoding applied to times series consisting of some deterministic signal and additive noise as well as time series given by a deterministic signal with randomly distributed initial conditions as a model of event related brain potentials we introduce an estimator of the signal to noise ratio snr of the system by means of time averages of running complexity measures such as shannon and r enyi entropies and prove its asymptotical equivalence with the linear snr in the case of shannon entropies of symbol distributions a snr improvement factor is defined exhibiting a maximum for intermediate values of noise amplitude in analogy to stochastic resonance phenomena we demonstrate that the maximum of the snr improvement factor can be shifted toward smaller noise amplitudes by using higher order r enyi entropies instead of the shannon entropy for a further improvement of the snr a half wave encoding of noisy time series is introduced finally we discuss the effect of noisy phases on the linear snr as well as on the snr defined by symbolic dynamics it is shown that longer symbol sequences yield an improvement of the latter

statistical analysis of stochastic resonance in a simple setting a subthreshold signal may be detected if noise is added to the data we study a simple model consisting of a constant signal to which at uniformly spaced times independent and identically distributed noise variables with known distribution are added a detector records the times at which the noisy signal exceeds a threshold there is an optimal noise level called stochastic resonance we explore the detectability of the signal in a system with one or more detectors with different thresholds we use a statistical detectability measure the asymptotic variance of the best estimator of the signal from the thresholded data or equivalently the fisher information in the data in particular we determine optimal configurations of detectors varying the distances between the thresholds and the signal as well as the noise level the approach generalizes to nonconstant signals

source coding for quasiarithmetic penalties whereas huffman coding finds a prefix code minimizing mean codeword length for a given finite item probability distribution quasiarithmetic or quasilinear coding problems have the goal of minimizing a generalized mean of the form rho 1 sigmaipirho li where li denotes the length of the ith codeword p i denotes the corresponding probability and rho is a monotonically increasing cost function such problems proposed by campbell have a number of diverse applications several cost functions are shown here to yield quasiarithmetic problems with simple redundancy bounds in terms of a generalized entropy a related property also shown here involves the existence of optimal codes for well behaved cost functions optimal codes always exist for possibly infinite alphabet sources having finite generalized entropy an algorithm is introduced for finding binary codes optimal for convex cost functions this algorithm which can be extended to other minimization utilities can be performed using quadratic time and linear space this reduces the computational complexity of a problem involving minimum delay in a queue allows combinations of previously considered problems to be optimized and greatly expands the set of problems solvable in quadratic time and linear space

some new fuzzy entropy formulas the purpose of this paper is twofold firstly a general conclusion about fuzzy entropy induced by distance measure is presented based on the axiom definitions of fuzzy entropy and distance measure secondly some fuzzy entropy formulas which relate to the fuzzy entropy formula defined by de luca and termini inform control 20 1972 301 are given

noise enhanced fisher information in parallel arrays of sensors with saturation this paper investigates stochastic resonance in parallel arrays of uncoupled saturating devices the fisher information is used to demonstrate the possibility of noise improved parameter estimation for arbitrary parametric signals especially it is shown that improvement by noise always occurs in these arrays for any configuration of the input signal even in optimal configuration the results contribute to establish stochastic resonance in parallel uncoupled arrays as a general mechanism of enhancement by noise which can occur in wide classes of nonlinearities and for various information processing tasks it can supplement other mechanisms of stochastic resonance that take place in isolated nonlinearities but generally in restricted configurations of the input signal

new property of a generalized holder s inequality and its applications in this paper we present a new property of a generalized holder s inequality and then we obtain a new refinement of the generalized holder s inequality as applications we give a generalization and refinement of the opial beesack inequality by using the obtained result moreover as an application in information theory we give a refinement of singh s inequality which is a generalized shannon s inequality

optimal lossless data compression non asymptotics and asymptotics this paper provides an extensive study of the behavior of the best achievable rate and other related fundamental limits in variable length strictly lossless compression in the non asymptotic regime the fundamental limits of fixed to variable lossless compression with and without prefix constraints are shown to be tightly coupled several precise quantitative bounds are derived connecting the distribution of the optimal code lengths to the source information spectrum and an exact analysis of the best achievable rate for arbitrary sources is given fine asymptotic results are proved for arbitrary not necessarily prefix compressors on general mixing sources nonasymptotic explicit gaussian approximation bounds are established for the best achievable rate on markov sources the source dispersion and the source varentropy rate are defined and characterized together with the entropy rate the varentropy rate serves to tightly approximate the fundamental nonasymptotic limits of fixed to variable compression for all but very small block lengths

bounds on the entropy series upper bounds on the entropy of a countable integer valued random variable are furnished in terms of the expectation of the logarithm function in particular an upper bound is derived that is sharper than that of p elias ibid vol it 21 no 2 p 194 203 1975 for all values of e sub p log bounds that are better only for large values of e sub p than the previous known upper bounds are also provided

redundancy related bounds for generalized huffman codes this paper presents new lower and upper bounds for the compression rate of binary prefix codes optimized over memoryless sources according to various nonlinear codeword length objectives like the most well known redundancy bounds for minimum average redundancy coding huffman coding these are in terms of a form of entropy and or the probability of an input symbol often the most probable one the bounds here some of which are tight improve on known bounds of the form l h h 1 where h is some form of entropy in bits or in the case of redundancy objectives 0 and l is the length objective also in bits the objectives explored here include exponential average length maximum pointwise redundancy and exponential average pointwise redundancy also called dth exponential redundancy the first of these relates to various problems involving queueing uncertainty and lossless communications the second relates to problems involving shannon coding and universal modeling also explored here for these two objectives is the related matter of individual codeword length

an achievable bound for optimal noiseless coding of a random variable corresp for a discrete n valued random variable n possibly denumerably infinite leung yan cheong and cover have given bounds for the minimal expected length of a one to one not necessarily uniquely decodable code l 1 1 sum i 1 n p i log left frac 1 2 1 right it is shown that the best possible case occurs for certain denumerably infinite sets of nonzero probabilities this absolute bound is related to the shannon entropy h of the distribution by h cdot is the binary entropy function