Minimization Problems Based on a Parametric Family of Relative Entropies I: Forward Projection

Minimization problems with respect to a one-parameter fami ly of generalized relative entropies are studied. These rel ativ entropies, which we term relative α-entropies (denoted Iα), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected co mpressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler di vergence). Just like relative entropy, these relative αntropies behave like squared Euclidean distance and satisfy the Pythagorean pro perty. Minimizers of these relative α-entropies on closed and convex sets are shown to exist. Such minimizations generalize the m aximum Rényi or Tsallis entropy principle. The minimizing probability distribution (termed forwardIα-projection) for a linear family is shown to have a power-law . Other results in connection with statistical inference, namely subspace transitivity and i terated projections, are also established. In a companion p aper, a related minimization problem of interest in robust statistics that leads to a reverse Iα-projection is studied. Index Terms Best approximant; exponential family; information geomet ry; Kullback-Leibler divergence; linear family; power-la w family; projection; Pythagorean property; relative entropy; Rén yi entropy; Tsallis entropy. I. I NTRODUCTION Relative entropy 1 or Kullback-Leibler divergenceI (P‖Q) between two probability measures is a fundamental quantity that arises in a variety of situations in probability theory , statistics, and information theory. In probability theor y, it arises as the rate function for estimating the probability of a large d viation for the empirical measure of independent sampling s. In statistics, for example, it arises as the best error exponen t i deciding between two hypothetical distributions for ob served data. In Shannon theory, it is the penalty in expected compressed l ength, namely the gap from Shannon entropy H(P ), when the compressor assumes (for a finite-alphabet source) a mismatc hed probability measure Q instead of the true probability measure P . Relative entropy also brings statistics and probability th eory together to provide a foundation for the well-known max imum entropy principle for decision making under uncertainty. T his is an idea that goes back to L. Boltzmann, was popularized by E. T. Jaynes [3], and has its foundation in the theory of large deviation. Suppose that an ensemble average measurement (s ay sample mean, sample second moment, or any other similar line ar statistic) is made on the realization of a sequence of iid random variables. The realization must then have an empiric al measure that obeys the constraint placed by the measureme nt – the empirical measure must belong to an appropriate convex s et, sayE. Large deviation theory tells us that a special member of E, denotedP , is overwhelmingly more likely than the others. If the alpha bet X is finite (with cardinality|X|), and the prior probability (before measurement) is the uniform meas ureU on X, thenP ∗ is the one that minimizes the relative entropy I (P‖U) = log |X| −H(P ), which is the same as the one that maximizes (Shannon) entropy 2, subject toP ∈ E. In Jaynes’ words,“. it is maximally noncommittal to the missing information” [3]. M. Ashok Kumar was supported by a Council for Scientific and In ustrial Research (CSIR) fellowship and by the Department o f Science and Technology. R. Sundaresan was supported in part by the University Grants Commission by Grant Part (2B) UGC-CAS-(Ph.IV) and in part by the Department of Science and Technology. A part of the material in this paper was prese nt d at the IEEE International Symposium on Information The ory (ISIT 2011), St. Petersburg, Russia, August 2011 [1]. A part of the material in the Introdu ction has overlap with a conference submission [2] under con sideration for presentation at the National Conference on Communication (NCC 2015), Mumbai, I ndia, to be held during February 2015. M. Ashok Kumar and R. Sundaresan are with the ECE Department, Indian Institute of Science, Bangalore 560012, India. 1The relative entropy of P with respect toQ is defined as I (P‖Q) := ∑ x∈X P (x) log P (x) Q(x) and the Shannon entropy of P is defined as H(P ) := − ∑