distribution estimation consistent in total variation and in two types of information divergence the problem of the nonparametric estimation of a probability distribution is considered from three viewpoints the consistency in total variation the consistency in information divergence and consistency in reversed order information divergence these types of consistencies are relatively strong criteria of convergence and a probability distribution cannot be consistently estimated in either type of convergence without any restrictions on the class of probability distributions allowed histogram based estimators of distribution are presented which under certain conditions converge in total variation in information divergence and in reversed order information divergence to the unknown probability distribution some a priori information about the true probability distribution is assumed in each case as the concept of consistency in information divergence is stronger than that of convergence in total variation additional assumptions are imposed in the cases of informational divergences

minimax renyi redundancy the redundancy for universal lossless compression of discrete memoryless sources in campbell s setting is characterized as a minimax renyi divergence which is shown to be equal to the maximal alpha mutual information via a generalized redundancy capacity theorem special attention is placed on the analysis of the asymptotics of minimax renyi divergence which is determined up to a term vanishing in blocklength

universal coding on infinite alphabets exponentially decreasing envelopes this paper deals with the problem of universal lossless coding on a countable infinite alphabet it focuses on some classes of sources defined by an envelope condition on the marginal distribution namely exponentially decreasing envelope classes with exponent the minimax redundancy of exponentially decreasing envelope classes is proved to be equivalent to then an adaptive algorithm is proposed whose maximum redundancy is equivalent to the minimax redundancy

redundancy of the context tree weighting method on renewal and markov renewal processes the context tree weighting method ctw is shown to be almost adaptive on the classes of renewal and markov renewal processes up to logarithmic factor ctw achieves the minimax pointwise redundancy described by i csiszaacuter and p shields in ieee trans inf theory vol 42 no 6 pp 2065 2072 nov 1996 this result not only complements previous results on the adaptivity of the context tree weighting method on the relatively small class of all finite context tree sources which encompasses the class of all finite order markov sources it shows that almost minimax redundancy can be achieved on massive classes of sources classes that cannot be smoothly parameterized by subsets of finite dimensional spaces moreover it shows that almost adaptive compression can be achieved in a computationally efficient way on those massive classes while previous adaptivity results for ctw could rely on the fact that any markov source is a finite context tree source this is no longer the case for renewal sources in order to prove almost adaptivity of ctw over renewal sources it is necessary to establish that ctw carefully balances estimation error and approximation error

properties of jeffreys mixture for markov sources we discuss the properties of jeffreys mixture for a markov model first we show that a modified jeffreys mixture asymptotically achieves the minimax coding regret for universal data compression where we do not put any restriction on data sequences moreover we give an approximation formula for the prediction probability of jeffreys mixture for a markov model by this formula it is revealed that the prediction probability by jeffreys mixture for the markov model with alphabet 0 1 is not of the form nx s ns where nx s is the number of occurrences of the symbol x following the context s 0 1 and ns n0 s n1 s moreover we propose a method to compute our minimax strategy which is a combination of a monte carlo method and the approximation formula where the former is used for earlier stages in the data while the latter is used for later stages

the performance of universal encoding universal coding theory is surveyed from the viewpoint of the interplay between delay and redundancy the price for universality turns out to be acceptably small

nonlinear quantization effects in the lms and block lms adaptive algorithms a comparison digital implementations of the least mean square lms and block lms blms algorithms are compared with respect to finite word effects the algorithm stalling phenomenon is studied using gaussian data models and conditional expectation arguments it is shown that the blms algorithm requires 1 2 log sub 2 l k fewer bits for the same stalling behavior l block length and k lies between 0 2 and 1 0 depending on the precise definition of algorithm stalling on the other hand the lms algorithm requires log sub 2 l fewer bits than blms for the same level of saturation behavior transient response at algorithm initialization hence the lms algorithm requires 1 2 log sub 2 l k fewer bits than the blms algorithm for the same saturation and stalling effects

on the mdl principle for i i d sources with large alphabets average case universal compression of independent and identically distributed i i d sources is investigated where the source alphabet is large and may be sublinear in size or even larger than the compressed data sequence length n in particular the well known results including rissanen s strongest sense lower bound for fixed size alphabets are extended to the case where the alphabet size k is allowed to grow with n it is shown that as long as k o n instead of the coding cost in the fixed size alphabet case of 0 5logn extra code bits for each one of the k 1 unknown probability parameters the cost is now 0 5log n k code bits for each unknown parameter this result is shown to be the lower bound in the minimax and maximin senses as well as for almost every source in the class achievability of this bound is demonstrated with two part codes based on quantization of the maximum likelihood ml probability parameters as well as by using the well known krichevsky trofimov kt low complexity sequential probability estimates for very large alphabets kgtn it is shown that an average minimax and maximin bound on the redundancy is essentially to first order log k n bits per symbol this bound is shown to be achievable both with two part codes and with a sequential modification of the kt estimates for k theta n the redundancy is theta 1 bits per symbol finally sequential codes are designed for coding sequences in which only m min k n alphabet symbols occur

universal prediction this paper consists of an overview on universal prediction from an information theoretic perspective special attention is given to the notion of probability assignment under the self information loss function which is directly related to the theory of universal data compression both the probabilistic setting and the deterministic setting of the universal prediction problem are described with emphasis on the analogy and the differences between results in the two settings

information optimality and bayesian modelling the general approach of treating a statistical problem as one of information processing led to the bayesian method of moments reference priors minimal information likelihoods and stochastic complexity these techniques rest on quantities that have physical interpretations from information theory current work includes the role of prediction the emergence of data dependent priors the role of information measures in model selection and the use of conditional mutual information to incorporate partial information

stochastic complexity and modeling on demontre un theoreme fondamental qui donne une borne inferieure pour la longueur de code et donc pour les erreurs de prediction on definit les notions d information a priori et d information utile dans les donnees

universal coding information prediction and estimation a connection between universal codes and the problems of prediction and statistical estimation is established a known lower bound for the mean length of universal codes is sharpened and generalized and optimum universal codes constructed the bound is defined to give the information in strings relative to the considered class of processes the earlier derived minimum description length criterion for estimation of parameters including their number is given a fundamental information theoretic justification by showing that its estimators achieve the information in the strings it is also shown that one cannot do prediction in gaussian autoregressive moving average arma processes below a bound which is determined by the information in the data

asymptotics and non asymptotics for universal fixed to variable source coding universal fixed to variable lossless source coding for memoryless sources is studied in the finite blocklength and higher order asymptotic regimes optimal three term fixed error asymptotic expressions are derived for general fixed to variable codes and for prefix codes it is shown that the non prefix type size code in which codeword lengths are chosen in ascending order of type class size achieves the optimal third order term and outperforms classical two stage codes converse results are proved making use of a result on the distribution of the empirical entropy and laplace s approximation finally the fixed to variable coding problem without a prefix constraint is shown to be essentially the same as the universal guessing problem

coding on countably infinite alphabets this paper describes universal lossless coding strategies for compressing sources on countably infinite alphabets classes of memoryless sources defined by an envelope condition on the marginal distribution provide benchmarks for coding techniques originating from the theory of universal coding over finite alphabets we prove general upper bounds on minimax regret and lower bounds on minimax redundancy for such source classes the general upper bounds emphasize the role of the normalized maximum likelihood nml codes with respect to minimax regret in the infinite alphabet context lower bounds are derived by tailoring sharp bounds on the redundancy of krichevsky trofimov coders for sources over finite alphabets up to logarithmic resp constant factors the bounds are matching for source classes defined by algebraically declining resp exponentially vanishing envelopes effective and almost adaptive coding techniques are described for the collection of source classes defined by algebraically vanishing envelopes those results extend our knowledge concerning universal coding to contexts where the key tools from parametric inference are known to fail

a new code for encoding all monotone sources with a fixed large alphabet size the problem of designing a fixed code for encoding all monotone sources with a large alphabet size n is studied it is proved that if the codelengths of a code sequence are of order o log n then the redundancy for almost all monotone sources with n symbols is very close to that for a specific distribution so called average distribution noting this point for any large alphabet size n a new code with a very simple structure is proposed whose redundancy is close to that of the huffman code for almost all monotone sources with alphabet size n

a simple statistical speech recognition of mandarin monosyllables each mandarin syllable is represented by a sequence of vectors of linear predict coding cepstra lpcc since all syllables have a simple phonetic structure in our speech recognition we partition the sequence of lpcc vectors of all syllables into equal segments and average the lpcc vectors in each segment the mean vector of lpcc is used as the feature of a syllable our simple feature does not need any time consuming and complicated nonlinear contraction and expansion as adopted by the dynamic time warping we propose several probability distributions for the feature values a simplified bayes decision rule is used for classification of mandarin syllables for the speaker independent mandarin digits the recognition rate is 98 6 if a normal distribution is used for feature values and the rate is 98 1 if an exponential distribution is used for the absolute values of the features the feature proposed in this paper to represent a syllable is the simplest one much easier to be extracted than any other known features the computation for feature extraction and classification is much faster and more accurate than using the hmm method or any other known techniques

predicting a binary sequence almost as well as the optimal biased coin we apply the exponential weight algorithm introduced and littlestone and warmuth 26 and by vovk 35 to the problem of predicting a binary sequence almost as well as the best biased coin we first show that for the case of the logarithmic loss the derived algorithm is equivalent to the bayes algorithm with jeffrey s prior that was studied by xie and barron 38 under probabilistic assumptions we derive a uniform bound on the regret which holds for any sequence we also show that if the empirical distribution of the sequence is bounded away from 0 and from 1 then as the length of the sequence increases to infinity the difference between this bound and a corresponding bound on the average case regret of the same algorithm which is asymptotically optimal in that case is only 1 2 we show that this gap of 1 2 is necessary by calculating the regret of the min max optimal algorithm for this problem and showing that the asymptotic upper bound is tight we also study the application of this algorithm to the square loss and show that the algorithm that is derived in this case is different from the bayes algorithm and is better than it for prediction in the worst case

precise minimax redundancy and regret recent years have seen a resurgence of interest in redundancy of lossless coding the redundancy regret of universal fixed to variable length coding for a class of sources determines by how much the actual code length exceeds the optimal ideal over the class code length in a minimax scenario one finds the best code for the worst source either in the worst case called also maximal minimax or on average we first study the worst case minimax redundancy over a class of stationary ergodic sources and replace shtarkov s bound by an exact formula among others we prove that a generalized shannon code minimizes the worst case redundancy derive asymptotically its redundancy and establish some general properties this allows us to obtain precise redundancy for memoryless markov and renewal sources for example we present the exact constant of the redundancy for memoryless and markov sources by showing that the integer nature of coding contributes log logm m 1 logm o 1 where m is the size of the alphabet then we deal with the average minimax redundancy and regret our approach here is orthogonal to most recent research in this area since we aspire to show that asymptotically the average minimax redundancy is equivalent to the worst case minimax redundancy for some classes of sources after formulating some general bounds relating these two redundancies we prove our assertion for memoryless and markov sources nevertheless we provide evidence that maximal redundancy of renewal processes does not have the same leading term as the average minimax redundancy however our general results show that maximal and average regrets are asymptotically equivalent

spl alpha parallel prior and its properties it is known that the jeffreys prior plays an important role in statistical inference in this paper we generalize the jeffreys prior from the point of view of information geometry and introduce a one parameter family of prior distributions which we named the spl alpha parallel priors the spl alpha parallel prior is defined as the parallel volume element with respect to the spl alpha connection and coincides with the jeffreys prior when spl alpha 0 further we analyze asymptotic behavior of the various estimators such as the projected bayes estimator the estimator obtained by projecting the bayes predictive density onto the original class of distributions and the minimum description length mdl estimator when the spl alpha parallel prior is used the difference of these estimators from maximum likelihood estimator mle due to the spl alpha prior is shown to be regulated by an invariant vector field of the statistical model although the jeffreys prior always exists the existence of spl alpha parallel prior with spl alpha spl ne 0 is not always guaranteed hence we consider conditions for the existence of the spl alpha parallel prior elucidating the conjugate symmetry in a statistical model

markov types and minimax redundancy for markov sources redundancy of universal codes for a class of sources determines by how much the actual code length exceeds the optimal code length in the minimax scenario one designs the best code for the worst source within the class such minimax redundancy comes in two flavors average minimax or worst case minimax we study the worst case minimax redundancy of universal block codes for markovian sources of any order we prove that the maximal minimax redundancy for markov sources of order r is asymptotically equal to 1 2m sup r m 1 log sub 2 n log sub 2 a sub m sup r lnlnm sup 1 m 1 lnm o 1 where n is the length of a source sequence m is the size of the alphabet and a sub m sup r is an explicit constant e g we find that for a binary alphabet m 2 and markov of order r 1 the constant a sub 2 sup 1 16 spl middot g spl ap 14 655449504 where g is the catalan number unlike previous attempts we view the redundancy problem as an asymptotic evaluation of certain sums over a set of matrices representing markov types the enumeration of markov types is accomplished by reducing it to counting eulerian paths in a multigraph in particular we propose exact and asymptotic formulas for the number of strings of a given markov type all of these findings are obtained by analytic and combinatorial tools of analysis of algorithms

exchangeability characterizes optimality of sequential normalized maximum likelihood and bayesian prediction we study online learning under logarithmic loss with regular parametric models in this setting each strategy corresponds to a joint distribution on sequences the minimax optimal strategy is the normalized maximum likelihood nml strategy we show that the sequential nml s nml strategy predicts minimax optimally i e as nml if and only if the joint distribution on sequences defined by snml is exchangeable this property also characterizes the optimality of a bayesian prediction strategy in that case the optimal prior distribution is jeffreys prior for a broad class of parametric models for which the maximum likelihood estimator is asymptotically normal the optimal prediction strategy nml depends on the number n of rounds of the game in general however when a bayesian strategy is optimal nml becomes independent of n our proof uses this to exploit the asymptotics of nml the asymptotic normality of the maximum likelihood estimator is responsible for the necessity of jeffreys prior

signal acquisition via polarization modulation in single photon sources a simple model system is introduced for demonstrating how a single photon source might be used to transduce classical analog information the theoretical scheme results in measurements of analog source samples that are i quantized in the sense of analog to digital conversion and ii corrupted by random noise that is solely due to the quantum uncertainty in detecting the polarization state of each photon this noise is unavoidable if more than 1 bit per sample is to be transmitted and we show how it may be exploited in a manner inspired by suprathreshold stochastic resonance the system is analyzed information theoretically as it can be modeled as a noisy optical communication channel although unlike classical poisson channels the detector s photon statistics are binomial previous results on binomial channels are adapted to demonstrate numerically that the classical information capacity and thus the accuracy of the transduction increases logarithmically with the square root of the number of photons n although the capacity is shown to be reduced when an additional detector nonideality is present the logarithmic increase with n remains

about adaptive coding on countable alphabets this paper sheds light on adaptive coding with respect to classes of memoryless sources over a countable alphabet defined by an envelope function with finite and non decreasing hazard rate log concave envelope distributions we prove that the auto censuring ac code is adaptive with respect to the collection of such classes the analysis builds on the tight characterization of universal redundancy rate in terms of metric entropy and on a careful analysis of the performance of the ac coding algorithm the latter relies on nonasymptotic bounds for maxima of samples from discrete distributions with finite and nondecreasing hazard rate

efficient universal noiseless source codes although the existence of universal noiseless variable rate codes for the class of discrete stationary ergodic sources has previously been established very few practical universal encoding methods are available efficient implementable universal source coding techniques are discussed in this paper results are presented on source codes for which a small value of the maximum redundancy is achieved with a relatively short block length a constructive proof of the existence of universal noiseless codes for discrete stationary sources is first presented the proof is shown to provide a method for obtaining efficient universal noiseless variable rate codes for various classes of sources for memoryless sources upper and lower bounds are obtained for the minimax redundancy as a function of the block length of the code several techniques for constructing universal noiseless source codes for memoryless sources are presented and their redundancies are compared with the bounds consideration is given to possible applications to data compression for certain nonstationary sources

about adaptive coding on countable alphabets max stable envelope classes in this paper we study the problem of lossless universal source coding for stationary memoryless sources on countably infinite alphabets this task is generally not achievable without restricting the class of sources over which universality is desired building on our prior work we propose natural families of sources characterized by a common dominating envelope we particularly emphasize the notion of adaptivity which is the ability to perform as well as an oracle knowing the envelope without actually knowing it this is closely related to the notion of hierarchical universal source coding but with the important difference that families of envelope classes are not discretely indexed and not necessarily nested our contribution is to extend the classes of envelopes over which adaptive universal source coding is possible namely by including max stable heavy tailed envelopes which are excellent models in many applications such as natural language modeling we derive a minimax lower bound on the redundancy of any code on such envelope classes including an oracle that knows the envelope we then propose a constructive code that does not use knowledge of the envelope the code is computationally efficient and is structured to use an expanding threshold for auto censoring etac and we therefore dub it the etac code we prove that the etac code achieves the lower bound on the minimax redundancy within a factor logarithmic in the sequence length and can be therefore qualified as a near adaptive code over families of heavy tailed envelopes for finite and light tailed envelopes the penalty is even less and the same code follows closely previous results that explicitly made the light tailed assumption our technical results are founded on methods from regular variation theory and concentration of measure

a bound on the financial value of information it is shown that each bit of information at most doubles the resulting wealth in the general stock market setup this information bound on the growth of wealth is actually attained for certain probability distributions on the market investigated by j kelly 1956 the bound is shown to be a special case of the result that the increase in exponential growth of wealth achieved with true knowledge of the stock market distribution f over that achieved with incorrect knowledge g is bounded above by the entropy of f relative to g

a general minimax result for relative entropy suppose nature picks a probability measure p sub spl theta on a complete separable metric space x at random from a measurable set p sub spl theta p spl theta spl theta spl isin spl theta then without knowing spl theta a statistician picks a measure q on s finally the statistician suffers a loss d p sub 0 spl par q the relative entropy between p sub spl theta and q we show that the minimax and maximin values of this game are always equal and there is always a minimax strategy in the closure of the set of all bayes strategies this generalizes previous results of gallager 1979 and davisson and leon garcia 1980

minimax redundancy for the class of memoryless sources let x sup n x sub 1 x sub n be a memoryless source with unknown distribution on a finite alphabet of size k we identify the asymptotic minimax coding redundancy for this class of sources and provide a sequence of asymptotically minimax codes equivalently we determine the limiting behavior of the minimax relative entropy min sub qxn max sub pxn d p sub xn spl par q sub xn where the maximum is over all independent and identically distributed i i d source distributions and the minimum is over all joint distributions we show in this paper that the minimax redundancy minus k 1 2 log n 2 spl pi e converges to log spl int spl radic det i spl theta d spl theta log spl gamma 1 2 sup k spl gamma k 2 where i spl theta is the fisher information and the integral is over the whole probability simplex the bayes strategy using jeffreys prior is shown to be asymptotically maximin but not asymptotically minimax in our setting the boundary risk using jeffreys prior is higher than that of interior points we provide a sequence of modifications of jeffreys prior that put some prior mass near the boundaries of the probability simplex to pull down that risk to the asymptotic minimax level in the limit

minimax compression and large alphabet approximation through poissonization and tilting this paper introduces a convenient strategy for coding and predicting sequences of independent identically distributed random variables generated from a large alphabet of size m in particular the size of the sample is allowed to be variable the employment of a poisson model and tilting method simplifies the implementation and analysis through independence the resulting strategy is optimal within the class of distributions satisfying a moment condition and it is close to optimal for the class of all i i d distributions on strings of a given length the method also can be used to code and predict strings with a condition on the tail of the ordered counts and it can be applied to distributions in an envelope class moreover we show that our model permits exact computation of the minimax optimal code for all alphabet sizes when conditioning on the size of the sample

information theoretic asymptotics of bayes methods in the absence of knowledge of the true density function bayesian models take the joint density function for a sequence of n random variables to be an average of densities with respect to a prior the authors examine the relative entropy distance d sub n between the true density and the bayesian density and show that the asymptotic distance is d 2 log n c where d is the dimension of the parameter vector therefore the relative entropy rate d sub n n converges to zero at rate log n n the constant c which the authors explicitly identify depends only on the prior density function and the fisher information matrix evaluated at the true parameter value consequences are given for density estimation universal data compression composite hypothesis testing and stock market portfolio selection