weak variable length source coding given a general source x x sup n sub n 1 sup spl infin source coding is characterized by a pair spl phi sub n spl psi sub n of encoder spl phi sub n and decoder spl psi sub n together with the probability of error spl epsi sub n spl equiv pr spl psi sub n spl phi sub n x sup n spl ne x sup n if the length of the encoder output spl phi sub n x sup n is fixed then it is called fixed length source coding while if the length of the encoder output spl phi sub n x sup n is variable then it is called variable length source coding usually in the context of fixed length source coding the probability of error spl epsi sub n is required to asymptotically vanish i e lim sub n spl rarr spl infin spl epsi sub n 0 whereas in the context of variable length source coding the probability of error spl epsi sub n is required to be exactly zero i e spl epsi sub n 0 spl forall n 1 2 in contrast to these we consider the problem of variable length source coding with asymptotically vanishing probability of error i e lim sub n spl rarr spl infin spl epsi sub n 0 and establish several fundamental theorems on this new subject

kolmogorov s contributions to information theory and algorithmic complexity briefly information theory says a random object x p x has complexity entropy h xp x log p x with the attendant interpretation that h bits are sufficient to describe x on the average algorithmic complexity says an object x has a complexity k x equal to the length of the shortest binary program that describes x it is a beautiful fact that these ideas are much the same in fact it is roughly true that ek x h moreover if we let pu x pr u prints x be the probability that a given computer u prints x when given a random program it can be shown that log 1 pu x k x for all x thus establishing a vital link between the universal probability measure pu and the universal complexity k more on this later the relationship of these ideas to probability theory was summed up in kolmogorov s 1983 paper which was based on his 1970 talk in nice perhaps only

locally optimal nonshortening linear covering codes and minimal saturating sets in projective spaces a concept of locally optimal lo linear covering codes is introduced in accordance with the concept of minimal saturating sets in projective spaces over finite fields an lo code is nonshortening in the sense that one cannot remove any column from a parity check matrix without increasing the code covering radius several q sup m concatenating constructions of lo covering codes are described taking a starting lo code as a seed such constructions produce an infinite family of lo codes with the same covering radius the infinite families of lo codes are designed using minimal saturating sets as starting codes new upper bounds on the length function are given new extremal and classification problems for linear covering codes are formulated and investigated in particular the spectrum of possible lengths of lo codes including the greatest possible length the complete computer classification of the minimal saturating sets in small geometries and of the corresponding lo codes is obtained

the strong law of large numbers for sequential decisions under uncertainty combines optimization and ergodic theory to characterize the optimum long run average performance that can be asymptotically attained by nonanticipating sequential decisions let x sub t be a stationary ergodic process and suppose an action b sub t must be selected in a space spl bscr with knowledge of the t past x sub 0 spl middot spl middot spl middot x sub t 1 at the beginning of every period t spl ges 0 action b sub t will incur a loss l b sub t x sub t at the end of period t when the random variable x sub t is revealed the author proves under mild integrability conditions that the optimum strategy is to select actions that minimize the conditional expected loss given the currently available information at each step the minimum long run average loss per decision can be approached arbitrarily closely by strategies that are finite order markov and under certain continuity conditions it is equal to the minimum expected loss given the infinite past if the loss l b x is bounded and continuous and if the space spl bscr is compact then the minimum can be asymptotically attained even if the distribution of the process x sub t is unknown a priori and must be learned from experience

independent vector analysis identification conditions and performance bounds recently an extension of independent component analysis ica from one to multiple datasets termed independent vector analysis iva has been a subject of significant research interest iva has also been shown to be a generalization of hotelling s canonical correlation analysis in this paper we provide the identification conditions for a general iva formulation which accounts for linear nonlinear and sample to sample dependencies the identification conditions are a generalization of previous results for ica and for iva when samples are independently and identically distributed furthermore a principal aim of iva is identification of dependent sources between datasets thus we provide additional conditions for when the arbitrary ordering of the estimated sources can be common across datasets performance bounds in terms of the cramer rao lower bound are also provided for demixing matrices and interference to source ratio the performance of two iva algorithms are compared to the theoretical bounds

large deviations for gibbs random fields a large deviation principle for gibbs random fields on zd is proven and a corresponding large deviations proof of the gibbs variational formula is given a generalization of the lanford theory of large deviations is also obtained

self similarity of brownian motion and a large deviation principle for random fields on a binary tree using self similarity of brownian motion and its representation as a product measure on a binary tree we construct a random sequence of probability measures which converges to the distribution of the brownian bridge we establish a large deviation principle for random fields on a binary tree this leads to a class of probability measures with a certain self similarity property the same construction can be carried out forc 0 1 valued processes and we can describe for instance ac 0 1 valued ornstein uhlenbeck process as a large deviation of brownian sheet

concentration of the information in data with log concave distributions on which the distribution of xis supported in this case eh x is essentially the number of bits neededto represent x by a coding scheme that minimizes average code length sha48 in the continuous case with reference measure dx one may stillcall eh x the information content even though the coding interpretation nolonger holds in statistics one may think of the information content as thelog likelihood function the average value of the information content of xis known more com monly as the entropy indeed the entropy of xis de ned byh x zf x logf x dx elogf x

a sandwich proof of the shannon mcmillan breiman theorem let x t be a stationary ergodic process with distribution p admitting densities p x 0 ldots x n 1 relative to a reference measure m that is finite order markov with stationary transition kernel let i m p denote the relative entropy rate then n 1 log p x 0 ldots x n 1 rightarrow i m p mathrm a s p we present an elementary proof of the shannon mcmillan breiman theorem and the preceding generalization obviating the need to verify integrability conditions and also covering the case i m p infty a sandwich argument reduces the proof to direct applications of the ergodic theorem

hidden markov processes an overview of statistical and information theoretic aspects of hidden markov processes hmps is presented an hmp is a discrete time finite state homogeneous markov chain observed through a discrete time memoryless invariant channel in recent years the work of baum and petrie 1966 on finite state finite alphabet hmps was expanded to hmps with finite as well as continuous state spaces and a general alphabet in particular statistical properties and ergodic theorems for relative entropy densities of hmps were developed consistency and asymptotic normality of the maximum likelihood ml parameter estimator were proved under some mild conditions similar results were established for switching autoregressive processes these processes generalize hmps new algorithms were developed for estimating the state parameter and order of an hmp for universal coding and classification of hmps and for universal decoding of hidden markov channels these and other related topics are reviewed

the strong ergodic theorem for densities generalized shannon mcmillan breiman theorem let xi x2 be a stationary process with probability densities f x1 x2 xn with respect to lebesgue measure or with respect to a markov measure with a stationary transition measure it is shown that the sequence of relative entropy densities 1 n log f xi x2 x0 converges almost surely this long conjectured result extends the l convergence obtained by moy perez and kieffer and generalizes the shannon mcmillanbreiman theorem to nondiscrete processes the heart of the proof is a new martingale inequality which shows that logarithms of densities are l dominated

space time large deviations for interacting particle systems the asymptotic behavior of the space time empirical averages of several classes of interacting particle systems is studied here the authors prove that the empirical averages obey a process level large deviation principle whose rate function is given explicitly in the case of spin systems the authors establish some relevant properties of the rate function such as the correspondence between its minima and the invariant measures for the process finally for discrete time non ergodic systems the authors give some results on the critical behavior of the empirical averages

interval algorithms for homophonic coding it is shown that the idea of the successive refinement of interval partitions which plays the key role in the interval algorithm for random number generation proposed by han and hoshi see ibid vol 43 p 599 611 1997 is also applicable to the homophonic coding an interval algorithm for homophonic coding is introduced which produces an independent and identically distributed i i d sequence with probability p lower and upper bounds for the expected codeword length are given based on this an interval algorithm for fixed to variable homophonic coding is established the expected codeword length per source letter converges to h x h p in probability as the block length tends to infinity where h x is the entropy rate of the source x the algorithm is asymptotically optimal an algorithm for fixed to fixed homophonic coding is also established the decoding error probability tends to zero as the block length tends to infinity homophonic coding with cost is generally considered the expected cost of the codeword per source letter converges to c h x h p in probability as the block length tends to infinity where c denotes the average cost of a source letter the main contribution of this paper can be regarded as a novel application of elias coding technique to homophonic coding intrinsic relations among these algorithms the interval algorithm for random number generation and the arithmetic code are also discussed

nonparametric entropy estimation for stationary processes and random fields with applications to english text we discuss a family of estimators for the entropy rate of a stationary ergodic process and prove their pointwise and mean consistency under a doeblin type mixing condition the estimators are cesaro averages of longest match lengths and their consistency follows from a generalized ergodic theorem due to maker 1940 we provide examples of their performance on english text and we generalize our results to countable alphabet processes and to random fields

algebraic geometry over groups i algebraic sets and ideal theory abstract the object of this paper which is the first in a series of three is to lay the foundations of the theory of ideals and algebraic sets over groups

hypothesis testing with the general source the asymptotically optimal hypothesis testing problem with general sources as the null and alternative hypotheses is studied under exponential type error constraints on the first kind of error probability our fundamental philosophy is to convert all of the hypothesis testing problems to the pertinent computation problems in the large deviation probability theory this methodologically new approach enables us to establish compact general formulas of the optimal exponents of the second kind of error and correct testing probabilities for the general sources including all nonstationary and or nonergodic sources with arbitrary abstract alphabet countable or uncountable these general formulas are presented from the information spectrum point of view

consistency of the maximum likelihood estimator for general hidden markov models consider a parametrized family of general hidden markov models where both the observed and unobserved components take values in a complete separable metric space we prove that the maximum likelihood estimator mle of the parameter is strongly consistent under a rather minimal set of assumptions as special cases of our main result we obtain consistency in a large class of nonlinear state space models as well as general results on linear gaussian state space models and finite state models a novel aspect of our approach is an information theoretic technique for proving identifiability which does not require an explicit representation for the relative entropy rate our method of proof could therefore form a foundation for the investigation of mle consistency in more general dependent and non markovian time series also of independent interest is a general concentration inequality for v uniformly ergodic markov chains

determining and approaching achievable rates of binary intersymbol interference channels using multistage decoding by examining the achievable rates of a multistage decoding system on stationary ergodic channels we derive lower bounds on the mutual information rate corresponding to independent and uniformly distributed i u d inputs also referred to as the i u d information rate for binary intersymbol interference isi channels we show that these bounds become tight as the number of decoding stages increases our analysis which focuses on the marginal conditional output densities at each stage of decoding provides an information rate corresponding to each stage these rates underlie the design of multilevel coding schemes based upon low density parity check ldpc codes and message passing that in combination with multistage decoding approach the i u d information rate for binary isi channels we give example constructions for channel models that have been commonly used in magnetic recording these examples demonstrate that the technique is very effective even for a small number of decoding stages

hilberg exponents new measures of long memory in the process this paper concerns the rates of power law growth of mutual information computed for a stationary measure or for a universal code the rates are called hilberg exponents and four such quantities are defined for each measure and each code two random exponents and two expected exponents a particularly interesting case arises for the conditional algorithmic mutual information in this case the random hilberg exponents are almost surely constant on ergodic sources and are bounded by the expected hilberg exponents this property is the second order analog of the shannon mcmillan breiman theorem proved without invoking the ergodic theorem it carries over to hilberg exponents for the underlying probability measure via shannon fano coding and barron inequality moreover the expected hilberg exponents can be linked for different universal codes namely if one code dominates another the expected hilberg exponents are greater for the former than for the latter this paper is concluded by an evaluation of hilberg exponents for certain sources such as the mixture bernoulli process and the santa fe processes

compression based methods for nonparametric prediction and estimation of some characteristics of time series we address the problem of online prediction for time series we show that any universal code or a universal data compressor can be used as a basis for constructing asymptotically optimal methods for this problem for a certain class of stationary and ergodic processes

large deviations for coding markov chains and gibbs random fields fixed block coding schemes for gibbs random fields are proposed in which the empirical expectations of the local interactions of the gibbs measure is compared to its expectation with respect to all gibbs measures having those interactions the exponential decay of the error probabilities is proven and it is shown that the code rates equal the entropy of the random field in addition it is shown that any coding scheme based on regarding the field as a 1 d sequence of symbols has rate greater than the entropy of the field the theory of fixed length coding is approached from the point of view of large deviations both for the calculation of the error exponents or error probabilities and for the calculation of the encoding rates or the asymptotic combinatorics of the coding schemes this approach is also applied to fixed length coding schemes of markov sources for which estimates on the error exponents and on the rates are derived