concentration of measure inequalities for markov chains and phi mixing processes we prove concentration inequalities for some classes of markov chains and f mixing processes with constants independent of the size of the sample that extend the inequalities for product measures of talagrand the method is based on information inequalities put forward by marton in case of contracting markov chains using a simple duality argument on entropy our results also include the family of logarithmic sobolev inequalities for convex functions applications to bounds on supremum of dependent empirical processes complete this work

renyi divergence and kullback leibler divergence renyi divergence is related to renyi entropy much like kullback leibler divergence is related to shannon s entropy and comes up in many settings it was introduced by renyi as a measure of information that satisfies almost the same axioms as kullback leibler divergence and depends on a parameter that is called its order in particular the renyi divergence of order 1 equals the kullback leibler divergence we review and extend the most important properties of renyi divergence and kullback leibler divergence including convexity continuity limits of sigma algebras and the relation of the special order 0 to the gaussian dichotomy and contiguity we also show how to generalize the pythagorean inequality to orders different from 1 and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs for all orders and present several other minimax results

size depth trade offs for monotone arithmetic circuits abstract consider the problem of computing the product a 1 a 1 a t b where a 1 a t are n n matrices a and b are vectors we show that the size s and depth d of monotone arithmetic circuits for this problem are related as s n 3 d tn 3 thus a reduction to depth d o t requires an increase from optimal size n 2 t to size n 3 t a similar trade off is shown for the evaluation of linear recurrences

renyi divergence measures for commonly used univariate continuous distributions abstract probabilistic distances also called divergences which in some sense assess how close two probability distributions are from one another have been widely employed in probability statistics information theory and related fields of particular importance due to their generality and applicability are the renyi divergence measures this paper presents closed form expressions for the renyi and kullback leibler divergences for nineteen commonly used univariate continuous distributions as well as those for multivariate gaussian and dirichlet distributions in addition a table summarizing four of the most important information measure rates for zero mean stationary gaussian processes namely renyi entropy differential shannon entropy renyi divergence and kullback leibler divergence is presented lastly a connection between the renyi divergence and the variance of the log likelihood ratio of two distributions is established thereby extending a previous result by song j stat plan infer 93 2001 on the relation between renyi entropy and the log likelihood function a table with the corresponding variance expressions for the univariate distributions considered here is also included

some exact complexity results for straight line computations over semirings the problem of computing polynomials in certain semmngs is considered precise bounds are obtained on the number of multiplications required by straight hne algorithms which compute such functions as iterated matrix multiplication iterated convolution and permanent usmg these bounds tt is shown that the use of branching can exponentially speed up computations using the min operations and that subtraction can exponentially speed up arithmetic computations these results can be interpreted as denying the existence of fast universal algorithms for computing certain polynomials k v wol os and prmases artthmeuc complexity convexity theory farkas lemma minimax algebra straight hne algorithm

on divergences and informations in statistics and information theory the paper deals with the f divergences of csiszar generalizing the discrimination information of kullback the total variation distance the hellinger divergence and the pearson divergence all basic properties of f divergences including relations to the decision errors are proved in a new manner replacing the classical jensen inequality by a new generalized taylor expansion of convex functions some new properties are proved too e g relations to the statistical sufficiency and deficiency the generalized taylor expansion also shows very easily that all f divergences are average statistical informations differences between prior and posterior bayes errors mutually differing only in the weights imposed on various prior distributions the statistical information introduced by de groot and the classical information of shannon are shown to be extremal cases corresponding to alpha 0 and alpha 1 in the class of the so called arimoto alpha informations introduced in this paper for 0 alpha 1 by means of the arimoto alpha entropies some new examples of f divergences are introduced as well namely the shannon divergences and the arimoto alpha divergences leading for alphauarr1 to the shannon divergences square roots of all these divergences are shown to be metrics satisfying the triangle inequality the last section introduces statistical tests and estimators based on the minimal f divergence with the empirical distribution achieved in the families of hypothetic distributions for the kullback divergence this leads to the classical likelihood ratio test and estimator

information divergence and risk for binary experiments we unify f divergences bregman divergences surrogate regret bounds proper scoring rules cost curves roc curves and statistical information we do this by systematically studying integral and variational representations of these objects and in so doing identify their representation primitives which all are related to cost sensitive binary classification as well as developing relationships between generative and discriminative views of learning the new machinery leads to tight and more general surrogate regret bounds and generalised pinsker inequalities relating f divergences to variational divergence the new viewpoint also illuminates existing algorithms it provides a new derivation of support vector machines in terms of divergences and relates maximum mean discrepancy to fisher linear discriminants

fast parallel computation of polynomials using few processors it is shown that any multivariate polynomial of degree d that can be computed sequentially in c steps can be computed in parallel in o log d log c log d steps using only cd o 1 processors

transforms associated to square integrable group representations i general results let g be a locally compact group which need not be unimodular let x u x x g be an irreducible unitary representation of g in a hilbert space h u assume that u is square integrable i e that there exists in h u at least one nonzero vector g such that u x g g 2 dx we give here a reasonably self contained analysis of the correspondence associating to every vector f h u the function u x g f on g discussing its isometry characterization of the range inversion and simplest interpolation properties this correspondence underlies many properties of generalized coherent states

a wreath product group approach to signal and image processing ii convolution correlation and applications for pt i see ibid vol 48 no 1 p 102 32 2000 this paper continues the investigation of the use of spectral analysis on certain noncommutative finite groups wreath product groups in digital signal processing we describe the generalization of discrete cyclic convolution in convolution over these groups and show how it reduces to multiplication in the spectral domain finite group based convolution is defined in both the spatial and spectral domains and its properties established we pay particular attention to wreath product cyclic groups and further describe convolution properties from a geometric view point in terms of operations with specific signals and filters group based correlation is defined in a natural way and its properties follow from those of convolution the detection of similarity of perceptually similar signals and an application of correlation the detection of similarity of group transformed signals several examples using images are included to demonstrate the ideas pictorially

a wreath product group approach to signal and image processing i multiresolution analysis we propose the use of spectral analysis on certain noncommutative finite groups in digital signal processing and in particular image processing we pay significant attention to groups constructed as wreath products of cyclic groups within this large class of groups our approach recovers the discrete fourier transform dft the haar wavelet transform various multichannel pyramid filter banks and other aspects of multiresolution analysis as special cases of a more general phenomenon in addition the group structure provides a rich algebraic structure that can be exploited for the analysis and manipulation of signals our approach relies on a synthesis of ideas found in the early work of holmes 1987 1990 karpovsky and trachtenberg 1985 and others on noncommutative filtering as well as diaconis s 1989 spectral analysis approach to understanding data

channel coding rate in the finite blocklength regime this paper investigates the maximal channel coding rate achievable at a given blocklength and error probability for general classes of channels new achievability and converse bounds are given which are tighter than existing bounds for wide ranges of parameters of interest and lead to tight approximations of the maximal achievable rate for blocklengths n as short as 100 it is also shown analytically that the maximal rate achievable with error probability isclosely approximated by c v n q 1 where c is the capacity v is a characteristic of the channel referred to as channel dispersion and q is the complementary gaussian cumulative distribution function

lower bounds for the minimax risk using f divergences and applications lower bounds involving f divergences between the underlying probability measures are proved for the minimax risk in estimation problems our proofs just use simple convexity facts special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as fano s inequality pinsker s inequality and inequalities based on global entropy conditions two applications are provided a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix

an algebraic approach to multiresolution analysis the notion of a weak multiresolution analysis is defined over an arbitrary field in terms of cyclic modules for a certain affine group ring in this setting the basic properties of weak multiresolution analyses are established including characterizations of their submodules and quotient modules the existence and uniqueness of reduced scaling equations and the existence of wavelet bases these results yield some sta ndard facts on classical multiresolution analyses over the rea ls as special cases but provide a different perspective by not relying on orthogonality or topology connections with other areas of algebra and possible further directions are mentioned

probability inequalities for likelihood ratios and convergence rates of sieve mles let y 1 y n be independent identically distributed with density p 0 and let f be a space of densities we show that the supremum of the likelihood ratios i 1 n p y i p 0 y i where the supremum is over p f with p l 2 p 0 1 2 2 e is exponentially small with probability exponentially close to 1 the exponent is proportional to ne 2 the only condition required for this to hold is that e exceeds a value determined by the bracketing hellinger entropy of f a similar inequality also holds if we replace f by f n and p 0 by q n where q n is an approximation to p 0 in a suitable sense these results are applied to establish rates of convergence of sieve mles furthermore weak conditions are given under which the optimal rate e n defined by h e n f ne n 2 where h f is the hellinger entropy of f is nearly achievable by sieve estimators

dissipation of information in channels with input constraints one of the basic tenets in information theory the data processing inequality states that the output divergence does not exceed the input divergence for any channel for channels without input constraints various estimates on the amount of such contraction are known dobrushin s coefficient for the total variation being perhaps the most well known this paper investigates channels with an average input cost constraint it is found that while the contraction coefficient typically equals one no contraction the information nevertheless dissipates a certain nonlinear function the dobrushin curve of the channel is proposed to quantify the amount of dissipation tools for evaluating the dobrushin curve of additive noise channels are developed based on coupling arguments some basic applications in stochastic control uniqueness of gibbs measures and fundamental limits of noisy circuits are discussed as an application it is shown that in the chain of n power constrained relays and gaussian channels the end to end mutual information and maximal squared correlation decay as theta log log n log n which is in stark contrast with the exponential decay in chains of discrete channels similarly the behavior of noisy circuits composed of gates with bounded fan in and broadcasting of information on trees of bounded degree does not experience threshold behavior in the signal to noise ratio snr namely unlike the case of discrete channels the probability of bit error stays bounded away from 1 2 regardless of the snr

extremum problems with total variation distance and their applications the aim of this paper is to investigate extremum problems with pay off being the total variation distance metric defined on the space of probability measures subject to linear functional constraints on the space of probability measures and vice versa that is with the roles of total variation metric and linear functional interchanged utilizing concepts from signed measures the extremum probability measures of such problems are obtained in closed form by identifying the partition of the support set and the mass of these extremum measures on the partition the results are derived for abstract spaces specifically complete separable metric spaces known as polish spaces while the high level ideas are also discussed for denumerable spaces endowed with the discrete topology these extremum problems often arise in many areas such as approximating a family of probability distributions by a given probability distribution maximizing or minimizing entropy subject to total variation distance metric constraints quantifying uncertainty of probability distributions by total variation distance metric stochastic minimax control and in many problems of information decision theory and minimax theory

a distribution dependent refinement of pinsker s inequality given two probability distributions q and p let spl par q p spl par sub 1 and d q spl par p respectively denote the l sub 1 distance and divergence between q and p we derive a refinement of pinsker s inequality of the form d q spl par p spl ges c p spl par q p spl par sub 1 sup 2 and characterize the best p dependent factor c p we apply the refined inequality to large deviations and measure concentration

the interplay between entropy and variational distance the relation between the shannon entropy and variational distance two fundamental and frequently used quantities in information theory is studied in this paper by means of certain bounds on the entropy difference between two probability distributions in terms of the variational distance between them and their alphabet sizes we also show how to find the distribution achieving the minimum or maximum entropy among those distributions within a given variational distance from any given distribution these results are applied to solve a number of problems that are of fundamental interest for entropy estimation we obtain an analytic formula for the confidence interval solving a problem that has been opened for more than 30 years for approximation of probability distributions we find the minimum entropy difference between two distributions in terms of their alphabet sizes and the variational distance between them in particular we show that the entropy difference between two distributions that are close in variational distance can be arbitrarily large if the alphabet sizes of the two distributions are unconstrained for random number generation we characterize the tradeoff between the amount of randomness required and the distortion in terms of variation distance new tools for non convex optimization have been developed to establish the results in this paper

mutual information metric entropy and cumulative relative entropy risk assume p theta theta epsilon theta is a set of probability distributions with a common dominating measure on a complete separable metric space y a state theta epsilon theta is chosen by nature a statistician obtains n independent observations y 1 dots y n from y distributed according to p theta for each time t between 1 and n based on the observations y 1 dots y t 1 the statistician produces an estimated distribution hat p t for p theta and suffers a loss l p theta hat p t the cumulative risk for the statistician is the average total loss up to time n of special interest in information theory data compression mathematical finance computational learning theory and statistical mechanics is the special case when the loss l p theta hat p t is the relative entropy between the true distribution p theta and the estimated distribution hat p t here the cumulative bayes risk from time 1 to n is the mutual information between the random parameter theta and the observations y 1 dots y n new bounds on this mutual information are given in terms of the laplace transform of the hellinger distance between pairs of distributions indexed by parameters in theta from these bounds on the cumulative minimax risk are given in terms of the metric entropy of theta with respect to the hellinger distance the assumptions required for these bounds are very general and do not depend on the choice of the dominating measure they apply to both finite and infinite dimensional theta they apply in some cases where y is infinite dimensional in some cases where y is not compact in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails

f divergence inequalities this paper develops systematic approaches to obtain f divergence inequalities dealing with pairs of probability measures defined on arbitrary alphabets functional domination is one such approach where special emphasis is placed on finding the best possible constant upper bounding a ratio of f divergences another approach used for the derivation of bounds among f divergences relies on moment inequalities and the logarithmic convexity property which results in tight bounds on the relative entropy and bhattacharyya distance in terms of chi 2 divergences a rich variety of bounds are shown to hold under boundedness assumptions on the relative information special attention is devoted to the total variation distance and its relation to the relative information and relative entropy including reverse pinsker inequalities as well as on the e gamma divergence which generalizes the total variation distance pinsker s inequality is extended for this type of f divergence a result which leads to an inequality linking the relative entropy and relative information spectrum integral expressions of the renyi divergence in terms of the relative information spectrum are derived leading to bounds on the renyi divergence in terms of either the variational distance or relative entropy

statistical information and discrimination in analogy with the definition of shannon information m h de groot 1962 defined statistical information as the difference between prior and posterior risk of a statistical decision problem relations are studied between the statistical information and the discrimination functions of information theory known as f divergences using previous results it is shown that every f divergence i sub f p q is an average statistical information or decision problem with dichotomic parameter 0 1 loss function and corresponding observation distributions p and q the average is taken over a distribution on the parameter s prior probability this distribution is uniquely determined by the function f the main result is that every f divergence is statistical information in some properly chosen statistical decision problem and conversely that every piece of statistical information is an f divergence this provides a new representation of discrimination functions figuring in signal detection data compression coding pattern classification cluster analysis etc

on pinsker s and vajda s type inequalities for csiszar s f divergences let d and v denote respectively information divergence and total variation distance pinsker s and vajda s inequalities are respectively d 1 2 v2 and d log 2 v 2 v 2v 2 v in this paper several generalizations and improvements of these inequalities are established for wide classes of f divergences first conditions on f are determined under which an f divergence df will satisfy df cf v2 or df c2 f v2 c4 f v4 where the constants cf c2 f and c4 f are best possible as a consequence lower bounds in terms of v are obtained for many well known distance and divergence measures including the 2 and hellinger s discrimination and the families of tsallis and renyi s divergences for instance if d p q 1 1 p q1 d 1 and p q 1 1 log p q1 d are respectively the relative information of type and the renyi s information gain of order it is shown that d 1 2 v2 1 72 1 2 v4 whenever 1 2 0 1 and that 2 v2 1 36 1 5 5 2 v4 for 0 1 in a somewhat different direction and motivated by the fact that these pinsker s type lower bounds are accurate only for small variation v close to zero lower bounds for df which are accurate for both small and large variation v close to two are also obtained in the special case of the information divergence they imply that d log 2 2 v 2 v 2 log 2 v 2 which uniformly improves vajda s inequality

divergence measures based on the shannon entropy a novel class of information theoretic divergence measures based on the shannon entropy is introduced unlike the well known kullback divergences the new measures do not require the condition of absolute continuity to be satisfied by the probability distributions involved more importantly their close relationship with the variational distance and the probability of misclassification error are established in terms of bounds these bounds are crucial in many applications of divergence measures the measures are also well characterized by the properties of nonnegativity finiteness semiboundedness and boundedness

context tree estimation for not necessarily finite memory processes via bic and mdl the concept of context tree usually defined for finite memory processes is extended to arbitrary stationary ergodic processes with finite alphabet these context trees are not necessarily complete and may be of infinite depth the familiar bayesian information criterion bic and minimum description length mdl principles are shown to provide strongly consistent estimators of the context tree via optimization of a criterion for hypothetical context trees of finite depth allowed to grow with the sample size n as o logn algorithms are provided to compute these estimators in o n time and to compute them on line for all i les n in o nlogn time

minimum kl divergence on complements of l_ 1 balls pinsker s widely used inequality upper bounds the total variation distance left vert p q right vert 1 in terms of the kullback leibler divergence d p vert q although in general a bound in the reverse direction is impossible in many applications the quantity of interest is actually d ast v q defined for an arbitrary fixed q as the infimum of d p vert q over all distributions p that are at least v far away from q in total variation we show that d ast v q leq c v 2 o v 3 where c c q 1 2 for balanced distributions thereby providing a kind of reverse pinsker inequality some of the structural results obtained in the course of the proof may be of independent interest an application to large deviations is given