analytic variations on redundancy rates of renewal processes csiszar and shields 1996 proved that the minimax redundancy for a class of stationary renewal processes is spl theta spl radic n where n is the block length this interesting result provides a nontrivial bound on redundancy for a nonparametric family of processes the present paper gives a precise estimate of the redundancy rate for such nonstationary renewal sources namely 2 log2 spl radic spl pi sup 2 6 1 n o log n this asymptotic expansion is derived by complex analytic methods that include generating function representations mellin transforms singularity analysis and saddle point estimates this work places itself within the framework of analytic information theory

the performance of universal encoding universal coding theory is surveyed from the viewpoint of the interplay between delay and redundancy the price for universality turns out to be acceptably small

polynomial wigner ville distributions and their relationship to time varying higher order spectra the wigner ville distribution wvd has optimal energy concentration for linear frequency modulated fm signals this paper presents a generalization of the wvd in order to effectively process nonlinear polynomial fm signals a class of polynomial wvd s pwvd s that give optimal concentration in the time frequency plane for fm signals with a modulation law of arbitrary polynomial form are defined a class of polynomial time frequency distributions ptfd s are also defined based on the class of pwvd s the optimal energy concentration of the pwvd enables it to be used for estimation of the instantaneous frequency if of polynomial fm signals finally a link between pwvd s and time varying higher order spectra tvhos is established just as the expected value of the wvd of a nonstationary random signal is the time varying power spectrum the expected values of the pwvd s have interpretations as reduced tvhos

caratheodory type selections and random fixed point theorems abstract we provide some new caratheodory type selection theorems i e selections for correspondences of two variables which are continuous with respect to one variable and measurable with respect to the other these results generalize simultaneously michael s 21 continuous selection theorem for lower semicontinuous correspondences as well as a caratheodory type selection theorem of fryszkowski 10 random fixed point theorems which generalize ordinary fixed point theorems e g browder s 6 follow as easy corollaries of our results

universal prediction this paper consists of an overview on universal prediction from an information theoretic perspective special attention is given to the notion of probability assignment under the self information loss function which is directly related to the theory of universal data compression both the probabilistic setting and the deterministic setting of the universal prediction problem are described with emphasis on the analogy and the differences between results in the two settings

estimation and classification of polynomial phase signals the measurement of the parameters of complex signals with constant amplitude and polynomial phase measured in additive noise is considered a novel new integral transform that is adapted for signals of this type is introduced this transform is used to derive estimation and classification algorithms that are simple to implement and that exhibit good performance the algorithms are extended to constant amplitude and continuous nonpolynomial phase signals

caratheodory selections for multivalued mappings abstract we prove the existence of a caratheodory selection for a set valued mapping satisfying standard conditions also an application to random fixed point for random operators is established

universal coding information prediction and estimation a connection between universal codes and the problems of prediction and statistical estimation is established a known lower bound for the mean length of universal codes is sharpened and generalized and optimum universal codes constructed the bound is defined to give the information in strings relative to the considered class of processes the earlier derived minimum description length criterion for estimation of parameters including their number is given a fundamental information theoretic justification by showing that its estimators achieve the information in the strings it is also shown that one cannot do prediction in gaussian autoregressive moving average arma processes below a bound which is determined by the information in the data

speaking of infinity i i d strings we study the redundancy of three approaches to compression of independent and identically distributed i i d strings over large possibly infinite alphabets standard compression of the string itself and compression of the string s shape and pattern which describe its symbols relative magnitude and precedence respectively we determine the rate at which per symbol standard redundancy increases to infinity as the alphabet size increases show that the maximum per symbol shape redundancy is between 0 027 and 1 and compare these to results showing that per symbol pattern redundancy diminishes to zero for all alphabet sizes we relate these concepts to ordered and unordered partitions of integers and sets and use this framework to explore relations between several combinatorial quantities including the bell fubini and second type stirling numbers

polynomial phase signal analysis based on the polynomial derivatives decompositions an exact decomposition of the derivatives of any order of a polynomial spl phi t is proposed in terms of spl phi t t sub 0 spl phi t t sub n this result allows us to introduce generalized time frequency distributions for studying signals having a polynomial phase and a constant amplitude in order to determine the degree and the coefficients of the corresponding phase the relationships between these distributions and the already known polynomial distributions i e the polynomial phase transform and the polynomial wigner ville distribution are discussed illustrations by example are proposed

universal source coding for monotonic and fast decaying monotonic distributions we study universal compression of sequences generated by monotonic distributions we show that for a monotonic distribution over an alphabet of size k each probability parameter costs essentially 0 5log n k3 bits where n is the coded sequence length as long as k o n1 3 otherwise for k o n the total average sequence redundancy is o n1 3 e bits overall we then show that there exists a sub class of monotonic distributions over infinite alphabets for which redundancy of o n1 3 e bits overall is still achievable this class contains fast decaying distributions including many distributions over the integers such as the family of zipf distributions and geometric distributions for some slower decays including other distributions over the integers redundancy of o n bits overall is achievable a method to compute specific redundancy rates for such distributions is derived the results are specifically true for finite entropy monotonic distributions finally we study individual sequence redundancy behavior assuming a sequence is governed by a monotonic distribution we show that for sequences whose empirical distributions are monotonic individual redundancy bounds even tighter than those in the average case can be obtained the relation of universal compression with monotonic distributions to universal compression of patterns of sequences is demonstrated

coding on countably infinite alphabets this paper describes universal lossless coding strategies for compressing sources on countably infinite alphabets classes of memoryless sources defined by an envelope condition on the marginal distribution provide benchmarks for coding techniques originating from the theory of universal coding over finite alphabets we prove general upper bounds on minimax regret and lower bounds on minimax redundancy for such source classes the general upper bounds emphasize the role of the normalized maximum likelihood nml codes with respect to minimax regret in the infinite alphabet context lower bounds are derived by tailoring sharp bounds on the redundancy of krichevsky trofimov coders for sources over finite alphabets up to logarithmic resp constant factors the bounds are matching for source classes defined by algebraically declining resp exponentially vanishing envelopes effective and almost adaptive coding techniques are described for the collection of source classes defined by algebraically vanishing envelopes those results extend our knowledge concerning universal coding to contexts where the key tools from parametric inference are known to fail

design of higher order polynomial wigner ville distributions the polynomial wigner ville distribution pwvd is a time frequency signal analysis tool for representing polynomial frequency modulated fm signals we present an optimization procedure to design pwvds and discuss the need for higher order pwvds

precise minimax redundancy and regret recent years have seen a resurgence of interest in redundancy of lossless coding the redundancy regret of universal fixed to variable length coding for a class of sources determines by how much the actual code length exceeds the optimal ideal over the class code length in a minimax scenario one finds the best code for the worst source either in the worst case called also maximal minimax or on average we first study the worst case minimax redundancy over a class of stationary ergodic sources and replace shtarkov s bound by an exact formula among others we prove that a generalized shannon code minimizes the worst case redundancy derive asymptotically its redundancy and establish some general properties this allows us to obtain precise redundancy for memoryless markov and renewal sources for example we present the exact constant of the redundancy for memoryless and markov sources by showing that the integer nature of coding contributes log logm m 1 logm o 1 where m is the size of the alphabet then we deal with the average minimax redundancy and regret our approach here is orthogonal to most recent research in this area since we aspire to show that asymptotically the average minimax redundancy is equivalent to the worst case minimax redundancy for some classes of sources after formulating some general bounds relating these two redundancies we prove our assertion for memoryless and markov sources nevertheless we provide evidence that maximal redundancy of renewal processes does not have the same leading term as the average minimax redundancy however our general results show that maximal and average regrets are asymptotically equivalent

sequential prediction and ranking in universal context modeling and data compression most state of the art lossless image compression schemes use prediction followed by some form of context modeling this might seem redundant at first as the contextual information used for prediction is also available for building the compression model and a universal coder will eventually learn the predictive patterns of the data in this correspondence we provide a format justification to the combination of these two modeling tools by showing that a combined scheme may result in faster convergence rate to the source entropy this is achieved via a reduction in the model cost of universal coding in deriving the main result we develop the concept of sequential ranking which can be seen as a generalization of sequential prediction and we study its combinatorial and probabilistic properties

on a caratheodory type selection theorem abstract we offer a new caratheodory type selection theorem this result arose naturally from the authors study of equilibria in abstract economies generalized games with a measure space of agents

characterization of polynomial functions and application to time frequency analysis we propose necessary and sufficient conditions for a function spl phi t to be a polynomial and establish a formula that allows us to compute the value of the derivative spl phi t of a polynomial at any point in terms of the values of this polynomial for this purpose we establish that any polynomial spl phi t of degree spl les q can be represented as a linear combination of spl phi t t sub 1 q t t sub q where t sub 1 t sub q denote arbitrary parameters a similar property is established for the derivative spl phi t some signal processing applications of these results are discussed

the discrete polynomial phase transform the discrete polynomial phase transform dpt is a new tool for analyzing constant amplitude polynomial phase signals the main properties of the dpt are its ability to identify the degree of the phase polynomial and to estimate its coefficients the transform is robust to deviations from the ideal signal model such as slowly varying amplitude additive noise and nonpolynomial phase the authors define the dpt derive its basic properties and use it to develop computationally efficient estimation and detection algorithms a statistical accuracy analysis of the estimated parameters is also presented

about adaptive coding on countable alphabets this paper sheds light on adaptive coding with respect to classes of memoryless sources over a countable alphabet defined by an envelope function with finite and non decreasing hazard rate log concave envelope distributions we prove that the auto censuring ac code is adaptive with respect to the collection of such classes the analysis builds on the tight characterization of universal redundancy rate in terms of metric entropy and on a careful analysis of the performance of the ac coding algorithm the latter relies on nonasymptotic bounds for maxima of samples from discrete distributions with finite and nondecreasing hazard rate

about adaptive coding on countable alphabets max stable envelope classes in this paper we study the problem of lossless universal source coding for stationary memoryless sources on countably infinite alphabets this task is generally not achievable without restricting the class of sources over which universality is desired building on our prior work we propose natural families of sources characterized by a common dominating envelope we particularly emphasize the notion of adaptivity which is the ability to perform as well as an oracle knowing the envelope without actually knowing it this is closely related to the notion of hierarchical universal source coding but with the important difference that families of envelope classes are not discretely indexed and not necessarily nested our contribution is to extend the classes of envelopes over which adaptive universal source coding is possible namely by including max stable heavy tailed envelopes which are excellent models in many applications such as natural language modeling we derive a minimax lower bound on the redundancy of any code on such envelope classes including an oracle that knows the envelope we then propose a constructive code that does not use knowledge of the envelope the code is computationally efficient and is structured to use an expanding threshold for auto censoring etac and we therefore dub it the etac code we prove that the etac code achieves the lower bound on the minimax redundancy within a factor logarithmic in the sequence length and can be therefore qualified as a near adaptive code over families of heavy tailed envelopes for finite and light tailed envelopes the penalty is even less and the same code follows closely previous results that explicitly made the light tailed assumption our technical results are founded on methods from regular variation theory and concentration of measure

minimax pointwise redundancy for memoryless models over large alphabets we study the minimax pointwise redundancy of universal coding for memoryless models over large alphabets and present two main results we first complete studies initiated in orlitsky and santhanam deriving precise asymptotics of the minimax pointwise redundancy for all ranges of the alphabet size relative to the sequence length second we consider the minimax pointwise redundancy for a family of models in which some symbol probabilities are fixed the latter problem leads to a binomial sum for functions with superpolynomial growth our findings can be used to approximate numerically the minimax pointwise redundancy for various ranges of the sequence length and the alphabet size these results are obtained by analytic techniques such as tree like generating functions and the saddle point method

universal lossless compression with unknown alphabets the average case universal compression of patterns of sequences generated by independent and identically distributed i i d sources with unknown possibly large alphabets is investigated a pattern is a sequence of indices that contains all consecutive indices in increasing order of first occurrence if the alphabet of a source that generated a sequence is unknown the inevitable cost of coding the unknown alphabet symbols can be exploited to create the pattern of the sequence this pattern can in turn be compressed by itself it is shown that if the alphabet size k is essentially small then the average minimax and maximin redundancies as well as the redundancy of every code for almost every source when compressing a pattern consist of at least 0 5log n k3 bits per each unknown probability parameter and if all alphabet letters are likely to occur there exist codes whose redundancy is at most 0 5log n k2 bits per each unknown probability parameter where n is the length of the data sequences otherwise if the alphabet is large these redundancies are essentially at least theta n 2 3 bits per symbol and there exist codes that achieve redundancy of o n 1 2 bits per symbol two suboptimal low complexity sequential algorithms for compression of patterns are presented and their description lengths analyzed also pointing out that the pattern average universal description length can decrease below the underlying i i d entropy for large enough alphabets

there is no universal source code for an infinite source alphabet shows that a discrete infinite distribution with finite entropy cannot be estimated consistently in information divergence as a corollary the authors show that there is no universal source code for an infinite source alphabet over the class of all discrete memoryless sources with finite entropy

singularity analysis and asymptotics of bernoulli sums the asymptotic analysis of a class of binomial sums that arise in information theory can be performed in a simple way by means of singularity analysis of generating functions the method developed extends the range of applicability of singularity analysis techniques to combinatorial sums involving transcendental elements like logarithms or fractional powers

some results on caratheodory selections and extensions we show that results on caratheodory selections and extensions follow from corresponding theorems on continuous selections and extensions respectively our basic tool is the marczewski function

universal compression of memoryless sources over unknown alphabets it has long been known that the compression redundancy of independent and identically distributed i i d strings increases to infinity as the alphabet size grows it is also apparent that any string can be described by separately conveying its symbols and its pattern the order in which the symbols appear concentrating on the latter we show that the patterns of i i d strings over all including infinite and even unknown alphabets can be compressed with diminishing redundancy both in block and sequentially and that the compression can be performed in linear time to establish these results we show that the number of patterns is the bell number that the number of patterns with a given number of symbols is the stirling number of the second kind and that the redundancy of patterns can be bounded using results of hardy and ramanujan on the number of integer partitions the results also imply an asymptotically optimal solution for the good turing probability estimation problem

entropy computations via analytic depoissonization we investigate the basic question of information theory namely evaluation of shannon entropy and a more general renyi 1961 entropy for some discrete distributions e g binomial negative binomial etc we aim at establishing analytic methods i e those in which complex analysis plays a pivotal role for such computations which often yield estimates of unparalleled precision the main analytic tool used here is that of analytic poissonization and depoissonization we illustrate our approach on the entropy evaluation of the binomial distribution that is we prove that for binomial n p distribution shannon s h sub n becomes h sub n spl ap 1 2 ln n 1 2 ln spl radic 2 spl pi p 1 p spl sigma sub k spl ges 1 a sub k n sup k where a sub k are explicitly computable constants moreover we argue that analytic methods e g complex asymptotics such as rice s method and singularity analysis mellin transforms poissonization and depoissonization can offer new tools for information theory especially for studying second order asymptotics e g redundancy in fact there has been a resurgence of interest and a few successful applications of analytic methods to a variety of problems of information theory therefore we propose to name such investigations as analytic information theory

minimax estimation of functionals of discrete distributions we propose a general methodology for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters and elaborate on the case of discrete distributions where the support size s is unknown and may be comparable with or even much larger than the number of observations n we treat the respective regions where the functional is nonsmooth and smooth separately in the nonsmooth regime we apply an unbiased estimator for the best polynomial approximation of the functional whereas in the smooth regime we apply a bias corrected version of the maximum likelihood estimator mle we illustrate the merit of this approach by thoroughly analyzing the performance of the resulting schemes for estimating two important information measures 1 the entropy h p sum i 1 s p i ln p i and 2 f alpha p sum i 1 s p i alpha alpha 0 we obtain the minimax l 2 rates for estimating these functionals in particular we demonstrate that our estimator achieves the optimal sample complexity n asymp s ln s for entropy estimation we also demonstrate that the sample complexity for estimating f alpha p 0 is n asymp s 1 alpha ln s which can be achieved by our estimator but not the mle for 1 we show the minimax l 2 rate for estimating f alpha p is n ln n 2 alpha 1 for infinite support size while the maximum l 2 rate for the mle is n 2 alpha 1 for all the above cases the behavior of the minimax rate optimal estimators with n samples is essentially that of the mle plug in rule with n ln n samples which we term effective sample size enlargement we highlight the practical advantages of our schemes for the estimation of entropy and mutual information we compare our performance with various existing approaches and demonstrate that our approach reduces running time and boosts the accuracy moreover we show that the minimax rate optimal mutual information estimator yielded by our framework leads to significant performance boosts over the chow liu algorithm in learning graphical models the wide use of information measure estimation suggests that the insights and estimators obtained in this paper could be broadly applicable