informatics mathematics
Inria

Proofs for Folklore
Theorems on the
Radon-Nikodym
Derivative

Yaiza Bermudez, Gaetan Bisson, Iñaki Esnaola, and Samir M. Perlaza

RESEARCH
REPORT

Nº 9591

July 2025

Project-Team NEO
<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# Proofs for Folklore Theorems on the Radon-Nikodym Derivative

Yaiza Bermudez, Gaetan Bisson, Iñaki Esnaola, and
Samir M. Perlaza

Project-Team NEO

Research Report n° 9591 — July 2025 — 20 pages

**Abstract:** In this research report, rigorous statements and formal proofs are presented for both foundational and advanced folklore theorems on the Radon-Nikodym derivative. The cases of conditional and marginal probability measures are carefully considered, which leads to an identity involving the sum of mutual and lautum information suggesting a new interpretation for such a sum.

**Key-words:** Radon Nikodym Derivative.

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex

<footnote>Yaiza Bermudez and Samir M. Perlaza are with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis 06902, France. Gaetan Bisson and Samir M. Perlaza are with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia. Iñaki Esnaola is with the School of Electrical and Electronic Engineering at The University of Sheffield, Sheffield S1 3JD, UK. Samir M. Perlaza and Iñaki Esnaola are also with the Electrical and Computer Engineering Department at Princeton University, Princeton N.J. 08544, USA.</footnote>
# Démonstrations des théorèmes folkloriques sur la
dérivée de Radon-Nikodym

**Résumé :** Dans ce rapport de recherche, des énoncés rigoureux et des preuves formelles sont présentés pour des théorèmes folkloriques, tant fondamentaux qu’avancés, portant sur la dérivée de Radon-Nikodym. Les cas des mesures de probabilité conditionnelles et marginales sont examinés avec soin, ce qui conduit à une identité impliquant la somme de l’information mutuelle et de l’information lautum, suggérant une nouvelle interprétation de celle-ci.

**Mots-clés :** Dérivé de Radon Nikodym
# Contents

1 Introduction 4

2 Preliminaries 4

3 Basic Folklore Theorems 5

4 Advanced Folklore Theorems 10

References 17
# 1 Introduction

In mathematics, folklore theorems refer to results that are widely accepted and frequently utilized by experts but are often not formally proven or explicitly documented. In game theory for example, the original Folk theorem earned its name because, although it was widely recognized among game theorists during the 1950s, it remains unpublished and without attribution to particular authors. See for instance [1] and [2]. Folklore theorems populate all areas in mathematics. In information theory, a large set of folklore theorems involve the Radon-Nikodym derivative (RND), first introduced by Radon [3]; and later generalized by Nikodym [4]. The existence of such folklore theorems in this area arises in part from the fact that all Shannon's information measures can be defined in terms of the RND. Interestingly, most properties of the RND are often presented without proof in most textbooks on measure theory and probability theory, c.f. [5-11], with the exception of some properties presented in [12].

Claude Shannon did not use the RND in his foundational papers [13, 14] to define entropy and mutual information. Instead, Shannon opted for restricting his work to the case in which measures either possess a probability mass function or a probability density function, which are both instances of RNDs. This choice significantly influenced the presentation of most subsequent results in information theory and established the style in which classical textbooks were written [15-31]. Nonetheless, the RND has been increasingly adopted in modern textbooks [32] and in the definition of new information measures, e.g., lautum information [33], to privilege a unified presentation. That is, independently of the measure used as a reference, e.g., the counting measure, the Lesbegue measure, etc. Some recent results whose presentation relies on the RND are for instance [34-45].

At the light of this observation, this technical report presents rigorous statements and alternative proofs for the most common folklore theorems on the RND. These include the change of measure theorem, the proportional-measures theorem, the chain rule, the multiplicative inverse, linearity and continuity theorems, as well as the product-measure theorem and nonnegativity and finiteness theorem. Less common theorems such as the unit measure and two Bayes-like rules of the RND are also formally proved. Finally, these novel theorems are used to prove a key identity involving the sum of mutual and lautum information, which is often observed in statistical learning [42, 46]. Hopefully, these contributions would provide a valuable reference for researchers and students in information theory.

# 2 Preliminaries

This section introduces relevant notational conventions alongside the Radon-Nikodym theorem. In particular, some equalities presented in this paper are valid almost surely with respect to a given measure. For clarity, given a measure space $(\Omega, \mathcal{F}, P)$, the notation $\frac{a.s.}{P}$ is introduced and shall be read as “equal for all $x \in \Omega$ except on a negligible set with respect to $P$”; or equivalently as “equal almost surely with respect to $P$”. Moreover, given two measures $P$
and $Q$ on the same measurable space, the notation $P \ll Q$ stands for “the measure $P$ is absolutely continuous with respect to $Q$”. Using this notation, the Radon-Nikodym derivative is introduced by the following theorem.

**Theorem 1** (Radon-Nikodym theorem, [5, Theorem 2.2.1]). Let $P$ be a signed measure and $Q$ be a $\sigma$-finite measure both on a given measurable space $(\Omega, \mathcal{F})$, with $P \ll Q$. Then, there exists a Borel measurable function $g : \Omega \to \mathbb{R}$ such that for all $\mathcal{A} \in \mathcal{F}$,

$$P(\mathcal{A}) = \int_{\mathcal{A}} g(x)dQ(x). \qquad (1)$$

Moreover, if another function $h$ satisfies for all $\mathcal{A} \in \mathcal{F}$ that $P(\mathcal{A}) = \int_{\mathcal{A}} h(x)dQ(x)$, then $g(x) \stackrel{a.s.}{=} h(x)$.

The function $g$ in (1) is often referred to as the Radom-Nikodym derivative of $P$ with respect to $Q$; and is also written as $\frac{dP}{dQ}$, such that $g(x) = \frac{dP}{dQ}(x)$.

The Radon-Nikodym derivative exhibits a number of properties from which many foundational results originate. Some of these properties are certainly folklore theorems and thus, the purpose of the following section is to provide formal proofs. Other properties are formally proved in literature and thus, for the sake of completeness, an alternative proof is presented using the described notation.

## 3 Basic Folklore Theorems

This section focuses on basic folklore theorems, where “basic” denotes their well-established nature. One of the most common folklore theorems is often referred to as the “change of measure” theorem.

**Theorem 2** (Change of Measure). Let $P$ be a signed measure and $Q$ be a $\sigma$-finite measure both on a given measurable space $(\Omega, \mathcal{F})$, with $P \ll Q$. Let $f : \Omega \to \mathbb{R}$ be a Borel measurable function such that the integral $\int_{\Omega} f(x)dP(x)$ exists. Then, for all $\mathcal{A} \in \mathcal{F}$,

$$\int_{\mathcal{A}} f(x)dP(x) = \int_{\mathcal{A}} f(x)\frac{dP}{dQ}(x)dQ(x). \qquad (2)$$

A proof of this theorem is presented in [12, Proposition 3.9]. For the sake of completeness, an alternative proof is presented hereunder, using the previously described notation.

*Proof:* Assume that $f$ is a simple function, i.e. for all $x \in \Omega$, $f(x) = \sum_{i=1}^{m} a_i 1_{\mathcal{A}_i}(x)$, for some finite $m \in \mathbb{N}$, disjoint sets $\mathcal{A}_1, \dots, \mathcal{A}_m$ in $\mathcal{F}$, and reals $a_1, \dots, a_m$. For all $\mathcal{A} \in \mathcal{F}$ and $i \in \{1, \dots, m\}$, let $\mathcal{B}_i = \mathcal{A} \cap \mathcal{A}_i$. Then,

$$\int_{\mathcal{A}} f(x) \frac{dP}{dQ}(x) dQ(x) = \int_{\mathcal{A}} \frac{dP}{dQ}(x) \sum_{i=1}^{n} a_i 1_{\mathcal{A}_i}(x) dQ(x) \qquad (3)$$

$$= \sum_{i=1}^{n} a_i \int_{\mathcal{B}_i} \frac{dP}{dQ}(x) dQ(x) = \sum_{i=1}^{n} a_i P(\mathcal{B}_i), \qquad (4)$$
where the equality in (4) follows Theorem 1 and the linearity of the integral [5, Theorem 1.6.3]. On the other hand, for all $\mathcal{A} \in \mathcal{F}$,

$$ \int_{\mathcal{A}} f(x) dP(x) = \sum_{i=1}^{m} a_i \int_{\mathcal{B}_i} dP(x) = \sum_{i=1}^{m} a_i P(\mathcal{B}_i), \qquad (5) $$

where the equality in (5) follows from the linearity of the integral [5, Theorem 1.6.3]. Hence, from Theorem 1 when $f$ is simple, the desired equality in (2) holds immediately.

The proof proceeds by observing that: (a) simple functions are dense in the space of Borel measurable functions [5, Theorem 1.5.5(b)], and (b) the integral is a continuous map on that space [5, Theorem 1.6.2]. From (a) and (b), it follows that (2) also holds for any Borel measurable function $f$. This completes the proof. ■

Another reputed folklore theorem, which is often referred to as the “proportional measures” theorem, establishes the explicit form of the Radon-Nikodym derivative between two measures, in which one is proportional to the other.

**Theorem 3 (Proportional Measures).** Let $P$ and $Q$ be two $\sigma$-finite measures on the measurable space $(\Omega, \mathcal{F})$, such that for all $\mathcal{A} \in \mathcal{F}$

$$ Q(\mathcal{A}) = cP(\mathcal{A}), \qquad (6) $$

with $c > 0$. Then,

$$ \frac{dP}{dQ}(x) \stackrel{a.s.}{=} \frac{1}{c}, \quad \text{and} \quad \frac{dQ}{dP}(x) \stackrel{a.s.}{=} c. \qquad (7) $$

*Proof:* Note that from (6), $P$ and $Q$ are mutually absolutely continuous. Moreover, from Theorem 1, for all $\mathcal{A} \in \mathcal{F}$,

$$ P(\mathcal{A}) = \int_{\mathcal{A}} dP(x) = \int_{\mathcal{A}} \frac{dP}{dQ}(x) dQ(x), \qquad (8) $$

and from (6), it follows that

$$ P(\mathcal{A}) = \frac{1}{c} Q(\mathcal{A}) = \int_{\mathcal{A}} \frac{1}{c} dQ(x). \qquad (9) $$

Hence, $\frac{dP}{dQ}(x) \stackrel{a.s.}{=} \frac{1}{c}$ follows directly from Theorem 1. A similar argument shows that $\frac{dQ}{dP}(x) \stackrel{a.s.}{=} c$. ■

In the special case $c = 1$ in (7), $P$ and $Q$ are identical, yielding $\frac{dP}{dQ}(x) \stackrel{a.s.}{=} \frac{dQ}{dP}(x) \stackrel{a.s.}{=} 1$.

The following folklore theorem is often referred to as the “chain rule” and a proof appears in [12, Proposition 3.9]. For the sake of completeness, an alternative proof is presented hereunder, using the previously described notation.
**Theorem 4 (Chain Rule).** Let P be a signed measure and let Q and R be two σ-finite measures, all on the same measurable space $(\Omega, \mathcal{F})$, with $P \ll Q$ and $Q \ll R$. Then,

$$ \frac{dP}{dR}(x) \stackrel{a.s.}{=} \frac{dP}{dQ}(x) \frac{dQ}{dR}(x). \qquad (10) $$

*Proof:* From the assumptions of the theorem and Theorem 2, it follows that for all $\mathcal{A} \in \mathcal{F}$,

$$ P(\mathcal{A}) = \int_{\mathcal{A}} dP(x) = \int_{\mathcal{A}} \frac{dP}{dQ}(x) dQ(x) \qquad (11) $$

$$ = \int_{\mathcal{A}} \frac{dP}{dQ}(x) \frac{dQ}{dR}(x) dR(x) = \int_{\mathcal{A}} \frac{dP}{dR}(x) dR(x), \qquad (12) $$

where the second equality in (12) holds given that $P \ll R$ and Theorem 1 which yields (10). ■

The following folklore theorem is a consequence of Theorem 4 and shows the connection between the Radon-Nikodym derivative and its multiplicative inverse.

**Theorem 5 (Multiplicative Inverse).** Let P and Q be two mutually absolutely continuous σ-finite measures on the measurable space $(\Omega, \mathcal{F})$; and assume that for all $x \in \Omega$, $\frac{dQ}{dP}(x) > 0$. Then,

$$ \frac{dP}{dQ}(x) \stackrel{a.s.}{=} \left( \frac{dQ}{dP}(x) \right)^{-1}. \qquad (13) $$

This result also appears in [12, Corollary 3.10].

*Proof:* From Theorem 4, it follows that

$$ \frac{dP}{dQ}(x) \frac{dQ}{dP}(x) \stackrel{a.s.}{=} \frac{dQ}{dQ}(x) \stackrel{a.s.}{=} 1, \qquad (14) $$

where the last equality follows from Theorem 3, with $c = 1$. ■

The subsequent folklore theorem establishes the linearity of the Radon-Nikodym derivative. A similar statement is presented in [12, Proposition 3.11] but left without proof.

**Theorem 6 (Linearity).** Let P be a σ-finite measure on $(\Omega, \mathcal{F})$ and let also $Q_1, Q_2, \dots, Q_n$ be finite measures on $(\Omega, \mathcal{F})$ absolutely continuous with respect to P. Let $c_1, c_2, \dots, c_n$ be positive reals; and let S be a finite measure on $(\Omega, \mathcal{F})$ such that for all $\mathcal{A} \in \mathcal{F}$, $S(\mathcal{A}) = \sum_{t=1}^{n} c_t Q_t(\mathcal{A})$. Then,

$$ \frac{dS}{dP}(x) \stackrel{a.s.}{=} \sum_{t=1}^{n} c_t \frac{dQ_t}{dP}(x). \qquad (15) $$

*Proof:* From the assumptions of the theorem, it holds that $S \ll P$, thus for all $\mathcal{A} \in \mathcal{F}$,

$$ \int_{\mathcal{A}} \frac{dS}{dP}(x) dP(x) = \int_{\mathcal{A}} dS(x) = \sum_{t=1}^{n} c_t \int_{\mathcal{A}} dQ_t(x) \qquad (16) $$
$$ = \sum_{t=1}^{n} \int_{\mathcal{A}} c_t \frac{dQ_t}{dP}(x) dP(x) = \int_{\mathcal{A}} \sum_{t=1}^{n} c_t \frac{dQ_t}{dP}(x) dP(x), \quad (17) $$

where the first equality in (16) and the first equality in (17) follow from Theorem 2; and the last equality in (17) follows from the additivity property of the integral [5, Corollary 1.6.4]. The proof ends by using Theorem 1 yielding (15). ■

The following folklore theorem establishes the continuity of the Radon-Nikodym derivative.

**Theorem 7 (Continuity).** Let $P$ be a $\sigma$-finite measure on $(\Omega, \mathcal{F})$, and let $Q_1, Q_2, \dots$ be an infinite sequence of finite measures on $(\Omega, \mathcal{F})$, converging to a signed measure $Q$. Suppose that for all $n \in \mathbb{N}$, $Q_n \ll P$. Then, $Q \ll P$ and

$$ \lim_{n \to \infty} \frac{dQ_n}{dP}(x) \stackrel{a.s.}{=} \frac{dQ}{dP}(x). \quad (18) $$

*Proof:* From the assumptions of the theorem and Theorem 2, for all $\mathcal{A} \in \mathcal{F}$, it holds that

$$ Q(\mathcal{A}) = \lim_{n \to \infty} Q_n(\mathcal{A}) = \int_{\mathcal{A}} \lim_{n \to \infty} \frac{dQ_n}{dP}(x) dP(x), \quad (19) $$

where the last equality in (19) follows from Theorem 2 and [5, Theorem 1.6.2]. Now, from the assumption that $Q \ll P$, it follows that for all $\mathcal{A} \in \mathcal{F}$,

$$ Q(\mathcal{A}) = \int_{\mathcal{A}} \frac{dQ}{dP}(x) dP(x). \quad (20) $$

Therefore, the equality in (20) and Theorem 1 yields (18). ■

<figure><img src="image_3.png" /></figure>

The ensuing folklore theorem relates the Radon-Nikodym derivative of a product measure to those of its component measures. This result also appears in [12, Exercise 3.12].

**Theorem 8 (Product of Measures).** For all $i \in \{1, 2\}$, let $P_i$ and $Q_i$ be a signed and a $\sigma$-finite measure on $(\Omega_i, \mathcal{F}_i)$, respectively; with $P_i \ll Q_i$. Let also $P_1 P_2$ and $Q_1 Q_2$ be the product measures on $(\Omega_1 \times \Omega_2, \mathcal{F}_1 \times \mathcal{F}_2)$ formed by $P_1$ and $P_2$; and $Q_1$ and $Q_2$, respectively. Then,

$$ \frac{dP_1 P_2}{dQ_1 Q_2}(x_1, x_2) \stackrel{a.s.}{=} \frac{dP_1}{dQ_1}(x_1) \frac{dP_2}{dQ_2}(x_2). \quad (21) $$

*Proof:* By definition of product measures, for all $\mathcal{A} \subseteq \Omega_1 \times \Omega_2$,

$$ P_1 P_2(\mathcal{A}) = \int_{\mathcal{A}} dP_1 P_2(x_1, x_2) = \iint_{\mathcal{A}_{x_2}} dP_1(x_1) dP_2(x_2) \quad (22) $$

$$ = \iint_{\mathcal{A}_{x_2}} \frac{dP_1(x_1)}{dQ_1} dQ_1(x_1) dP_2(x_2) \quad (23) $$

$$ = \iint_{\mathcal{A}_{x_2}} \frac{dP_1(x_1) dP_2(x_2)}{dQ_1 dQ_2} dQ_1(x_1) dQ_2(x_2) \quad (24) $$
$$= \int_{\mathcal{A}} \frac{dP_1}{dQ_1}(x_1) \frac{dP_2}{dQ_2}(x_2) dQ_1 Q_2(x_1, x_2), \qquad (25)$$

where $\mathcal{A}_{x_2}$ is the section of the set $\mathcal{A}$ determined by $x_2$, namely, $\mathcal{A}_{x_2} = \{x_1 : (x_1, x_2) \in \mathcal{A}\}$; the equality in (22) arises from the definition of $P_1 P_2$ as the product of $P_1$ and $P_2$; the equalities in (23) and (24) are direct consequences of Theorem 2; and finally, the equality in (25) is due to the construction of $Q_1 Q_2$ as the product measure of $Q_1$ and $Q_2$. From the equality in (25), it holds that $P_1 P_2 \ll Q_1 Q_2$. Thus, for all $\mathcal{A} \in \mathcal{F}_1 \times \mathcal{F}_2$,

$$P_1 P_2 (\mathcal{A}) = \int_{\mathcal{A}} \frac{dP_1 P_2}{dQ_1 Q_2} (x_1, x_2) dQ_1 Q_2 (x_1, x_2). \qquad (26)$$

From Theorem 1, the equality in (21) follows, completing the proof. ■

The following theorem shows a nonnegativity and finiteness property of the RND.

**Theorem 9 (Nonnegativity and Finiteness).** Consider a probability measure $P$ and a $\sigma$-finite measure $Q$ on $(\Omega, \mathcal{F})$, with $P \ll Q$. Then

$$P\left(\left\{\omega \in \Omega : 0 \le \frac{dP}{dQ}(\omega) < +\infty\right\}\right) = 1. \qquad (27)$$

*Proof:* Assume that

$$P\left(\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) < 0\right\}\right) > 0. \qquad (28)$$

Hence,

$$P\left(\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) < 0\right\}\right) = \int_{\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) < 0\right\}} dP(x) \qquad (29)$$

$$= \int_{\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) < 0\right\}} \frac{dP}{dQ}(x) dQ(x) \qquad (30)$$

$$< 0, \qquad (31)$$

which contradicts that $P$ is a probability measure, which is nonnegative. Alternative, assume that

$$P\left(\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) = +\infty\right\}\right) > 0. \qquad (32)$$

Hence,

$$P\left(\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) = +\infty\right\}\right) = \int_{\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) = +\infty\right\}} dP(x) \qquad (33)$$

$$= \int_{\left\{\omega \in \Omega : \frac{dP}{dQ}(\omega) = +\infty\right\}} \frac{dP}{dQ}(x) dQ(x) \qquad (34)$$

$$= +\infty, \qquad (35)$$

which contradicts that $P$ is a probability measure, which is a finite measure. ■
# 4 Advanced Folklore Theorems

This section focuses exclusively on probability measures with the aim of establishing closer ties to information measures, which requires some additional notation. In particular, denote by $\Delta(\mathcal{X}, \mathcal{F}_\mathcal{X})$, or simply $\Delta(\mathcal{X})$, the set of all probability measures on the measurable space $(\mathcal{X}, \mathcal{F}_\mathcal{X})$, where $\mathcal{F}_\mathcal{X}$ is a $\sigma$-algebra on $\mathcal{X}$. Using this notation, conditional probability measures can be defined as follows.

**Definition 1 (Conditional Probability).** A family $P_{Y|X} \triangleq (P_{Y|X=x})_{x \in \mathcal{X}}$ of elements of $\Delta(\mathcal{Y}, \mathcal{F}_\mathcal{Y})$ indexed by $\mathcal{X}$ is said to be a conditional probability measure if, for all sets $\mathcal{B} \in \mathcal{F}_\mathcal{Y}$, the map

$$ \begin{cases} \mathcal{X} \to [0, 1] \\ x \mapsto P_{Y|X=x}(\mathcal{B}) \end{cases} \qquad (36) $$

is Borel measurable. The set of such conditional probability measures is denoted by $\Delta(\mathcal{Y}|\mathcal{X})$.

A conditional probability $P_{Y|X} \in \Delta(\mathcal{Y} | \mathcal{X})$ and a probability measure $P_X \in \Delta(\mathcal{X})$ jointly determine two unique probability measures in $\Delta(\mathcal{X} \times \mathcal{Y})$ and $\Delta(\mathcal{Y} \times \mathcal{X})$, respectively. These probability measures are denoted by $P_{XY}$ and $P_{YX}$, respectively, and for all sets $\mathcal{A} \in \mathcal{F}_\mathcal{X} \times \mathcal{F}_\mathcal{Y}$, it follows that

$$ P_{XY}(\mathcal{A}) = \int P_{Y|X=x}(\mathcal{A}_x) dP_X(x), \qquad (37) $$

where $\mathcal{A}_x$ is the section of the set $\mathcal{A}$ determined by $x$, namely,

$$ \mathcal{A}_x \triangleq \{y \in \mathcal{Y} : (x, y) \in \mathcal{A}\}. \qquad (38) $$

Alternatively, for all sets $\mathcal{B} \in \mathcal{F}_\mathcal{Y} \times \mathcal{F}_\mathcal{X}$, it follows that

$$ P_{YX}(\mathcal{B}) = \int P_{Y|X=x}(\mathcal{B}_x) dP_X(x), \qquad (39) $$

where $\mathcal{B}_x$ is the section of the set $\mathcal{B}$ determined by $x$. Note that $P_{XY}$ is a measure in $\Delta(\mathcal{X} \times \mathcal{Y})$ whereas $P_{YX}$ is a measure in $\Delta(\mathcal{Y} \times \mathcal{X})$. Consequently, obtaining $P_{XY}$ from $P_{YX}$ requires more than simply renaming $X$ and $Y$; one must also properly redefine the corresponding measurable spaces. In particular, given a set $\mathcal{A} \in \mathcal{F}_\mathcal{X} \times \mathcal{F}_\mathcal{Y}$, let the set $\hat{\mathcal{A}} \in \mathcal{F}_\mathcal{Y} \times \mathcal{F}_\mathcal{X}$ be such that

$$ \hat{\mathcal{A}} = \{(y, x) \in \mathcal{Y} \times \mathcal{X} : (x, y) \in \mathcal{A}\}. \qquad (40) $$

Then, from (37) and (39), it holds that

$$ P_{XY}(\mathcal{A}) = P_{YX}(\hat{\mathcal{A}}). \qquad (41) $$

Using this notation, a marginal probability measure is defined as follows.
**Definition 2 (Marginal).** Given two joint probability measures $P_{XY} \in \Delta(\mathcal{X} \times \mathcal{Y})$ and $P_{YX} \in \Delta(\mathcal{Y} \times \mathcal{X})$, satisfying (41), the marginal probability measures in $\Delta(\mathcal{X})$ and $\Delta(\mathcal{Y})$, denoted by $P_X$ and $P_Y$, respectively, satisfy for all sets $\mathcal{A} \in \mathcal{F}_X$

$$P_X(\mathcal{A}) \triangleq P_{XY}(\mathcal{A} \times \mathcal{Y}) = P_{YX}(\mathcal{Y} \times \mathcal{A}); \quad (42)$$

and for all sets $\mathcal{B} \in \mathcal{F}_Y$,

$$P_Y(\mathcal{B}) \triangleq P_{YX}(\mathcal{B} \times \mathcal{X}) = P_{XY}(\mathcal{X} \times \mathcal{B}). \quad (43)$$

From the total probability theorem [5, Theorem 4.5.2], it follows that for all $\mathcal{B} \in \mathcal{F}_Y$,

$$P_Y(\mathcal{B}) = \iint_{\mathcal{B}} dP_{Y|X=x}(y)dP_X(x); \quad (44)$$

and for all $\mathcal{A} \in \mathcal{F}_X$,

$$P_X(\mathcal{A}) = \iint_{\mathcal{A}} dP_{X|Y=y}(x)dP_Y(y). \quad (45)$$

In a nutshell, the joint probability measures $P_{XY}$ and $P_{YX}$ can be described via the conditional probability measure $P_{Y|X}$ and the probability measure $P_X$ as in (37) and in (39); or via the conditional probability measure $P_{X|Y} \in \Delta(\mathcal{X}|\mathcal{Y})$ and the marginal probability measure $P_Y \in \Delta(\mathcal{Y})$. More specifically, for all sets $\mathcal{A} \in \mathcal{F}_X \times \mathcal{F}_Y$, it follows that

$$P_{XY}(\mathcal{A}) = \int P_{X|Y=y}(\mathcal{A}_y) dP_Y(y), \quad (46)$$

where $\mathcal{A}_y$ is the section of the set $\mathcal{A}$ determined by $y$, namely,

$$\mathcal{A}_y \triangleq \{x \in \mathcal{X} : (x, y) \in \mathcal{A}\}. \quad (47)$$

Alternatively, for all sets $\mathcal{B} \in \mathcal{F}_Y \times \mathcal{F}_X$, it follows that

$$P_{YX}(\mathcal{B}) = \int P_{X|Y=y}(\mathcal{B}_y) dP_Y(y), \quad (48)$$

where $\mathcal{B}_y$ is the section set of $\mathcal{B}$ determined by $y$.

Within this context, the following folklore theorem highlights a property of conditional measures, which is reminiscent of the unit measure axiom in probability theory.

**Theorem 10 (Unit Measure).** Consider the conditional probability measures $P_{Y|X} \in \Delta(\mathcal{Y}|\mathcal{X})$ and $P_{X|Y} \in \Delta(\mathcal{X}|\mathcal{Y})$; the probability measures $P_Y \in \Delta(\mathcal{Y})$ and $P_X \in \Delta(\mathcal{X})$ that satisfy (44) and (45). Assume that for all $x \in \mathcal{X}$, the probability measure $P_{Y|X=x} \ll P_Y$. Then,

$$\int \frac{dP_{Y|X=x}}{dP_Y}(y)dP_X(x) \stackrel{\text{a.s.}}{=} 1. \quad (49)$$
*Proof:* For all $\mathcal{A} \in \mathcal{F}_Y$, from (44), it holds that

$$P_Y(\mathcal{A}) = \iint_{\mathcal{A}} dP_{Y|X=x}(y)dP_X(x) \qquad (50)$$

$$= \iint_{\mathcal{A}} \frac{dP_{Y|X=x}}{dP_Y}(y)dP_Y(y)dP_X(x) \qquad (51)$$

$$= \int_{\mathcal{A}} \int \frac{dP_{Y|X=x}}{dP_Y}(y)dP_X(x)dP_Y(y), \qquad (52)$$

where the equality in (51) follows from a change of measure (Theorem 2). Moreover, (52) is obtained using Fubini's theorem [5, Theorem 2.6.6]. The proof proceeds by noticing that $P_Y(\mathcal{A}) = \int_{\mathcal{A}} dP_Y(y)$, and thus from Theorem 1 and the equality in (52), the statement in (49) holds. ■

The following folklore theorems are reminiscent of the Bayes rule.

**Theorem 11** (Bayes-like rule). Consider the conditional probability measures $P_{Y|X}$ and $P_{X|Y}$; the probability measures $P_Y$ and $P_X$ that satisfy (44) and (45); and the joint probability measures $P_{XY}$ and $P_{XY}$ in (39) and (46) respectively. Let also $P_X P_Y \in \Delta(\mathcal{X} \times \mathcal{Y}, \mathcal{F}_X \times \mathcal{F}_Y)$ and $P_Y P_X \in \Delta(\mathcal{Y} \times \mathcal{X}, \mathcal{F}_Y \times \mathcal{F}_X)$ be the measures formed by the product of the marginals $P_X$ and $P_Y$. Assume that: (a) For all $x \in \mathcal{X}$, $P_{Y|\mathcal{X}=x} \ll P_Y$; and (b) For all $y \in \mathcal{Y}$, $P_{X|\mathcal{Y}=y} \ll P_X$. Then,

$$\frac{dP_{XY}}{dP_X P_Y}(x, y) \stackrel{\text{a.s.}}{=} \frac{dP_{X|Y=y}}{dP_X}(x) \stackrel{\text{a.s.}}{=} \frac{dP_{Y|X=x}}{dP_Y}(y) \stackrel{\text{a.s.}}{=} \frac{dP_{YX}}{dP_Y P_X}(y, x). \quad (53)$$

*Proof:* Note that assumptions (a) and (b) are sufficient for the Radon-Nikodym derivatives of $P_{XY}$ with respect to $P_X P_Y$; and $P_{XY}$ with respect to $P_Y P_X$ to exist. Hence, it follows that for all sets $\mathcal{A} \in \mathcal{F}_X \times \mathcal{F}_Y$,

$$P_{XY}(\mathcal{A}) = \int_{\mathcal{A}} \frac{dP_{XY}}{dP_X P_Y}(x, y)dP_X P_Y(x, y), \qquad (54)$$

which follows from Theorem 2. Note also that from (46), it follows that

$$P_{XY}(\mathcal{A}) = \iint_{\mathcal{A}_Y} dP_{X|Y=y}(x) dP_Y(y) \qquad (55)$$

$$= \iint_{\mathcal{A}_Y} \frac{dP_{X|Y=y}}{dP_X}(x) dP_X(x) dP_Y(y) \qquad (56)$$

$$= \iint 1_{\mathcal{A}_Y}(x) \frac{dP_{X|Y=y}}{dP_X}(x) dP_X(x) dP_Y(y) \qquad (57)$$

$$= \int 1_{\mathcal{A}}(x, y) \frac{dP_{X|Y=y}}{dP_X}(x) dP_X P_Y(x, y) \qquad (58)$$

$$= \int_{\mathcal{A}} \frac{dP_{X|Y=y}}{dP_X}(x) dP_X P_Y(x, y), \qquad (59)$$

where, the set $\mathcal{A}_Y$ is defined in (47). Moreover, the equality in (56) follows from Assumption (b) and Theorem 1. Similarly, from (37), it follows that

$$P_{XY}(\mathcal{A}) = \iint_{\mathcal{A}_X} dP_{Y|X=x}(y) dP_X(x) \qquad (60)$$
$$ = \iint_{\mathcal{A}_x} \frac{dP_{Y|X=x}}{dP_Y} (y) dP_Y (y) dP_X (x) \quad (61) $$

$$ = \iint 1_{\mathcal{A}_x}(y) \frac{dP_{Y|X=x}}{dP_Y} (y) dP_Y (y) dP_X (x) \quad (62) $$

$$ = \iint 1_{\mathcal{A}_y}(x) \frac{dP_{Y|X=x}}{dP_Y} (y) dP_X (x) dP_Y (y) \quad (63) $$

$$ = \int 1_{\mathcal{A}}(x, y) \frac{dP_{Y|X=x}}{dP_Y} (y) dP_X P_Y (x, y) \quad (64) $$

$$ = \int_{\mathcal{A}} \frac{dP_{Y|X=x}}{dP_Y} (y) dP_X P_Y (x, y), \quad (65) $$

where, the set $\mathcal{A}_x$ is defined in (38). Moreover, the equality in (61) follows from Assumption (a) and Theorem 1; and the equality in (63) follows by exchanging the order of integration [5, Theorem 2.6.6]. Finally, from (41), it follows that

$$ P_{XY}(\mathcal{A}) = \int_{\hat{\mathcal{A}}} dP_{YX}(y,x) = \int_{\hat{\mathcal{A}}} \frac{dP_{YX}}{dP_Y P_X}(y,x) dP_Y P_X(y,x) \quad (66) $$

$$ = \int 1_{\hat{\mathcal{A}}}(y,x) \frac{dP_{YX}}{dP_Y P_X}(y,x) dP_Y P_X(y,x) \quad (67) $$

$$ = \iint 1_{\hat{\mathcal{A}}_x}(y) \frac{dP_{YX}}{dP_Y P_X}(y,x) dP_Y(y) dP_X(x) \quad (68) $$

$$ = \iint 1_{\hat{\mathcal{A}}_y}(x) \frac{dP_{YX}}{dP_Y P_X}(y,x) dP_X(x) dP_Y(y) \quad (69) $$

$$ = \int 1_{\mathcal{A}}(x, y) \frac{dP_{YX}}{dP_Y P_X}(y, x) dP_X P_Y(x, y) \quad (70) $$

$$ = \int_{\mathcal{A}} \frac{dP_{YX}}{dP_Y P_X}(y, x) dP_X P_Y(x, y), \quad (71) $$

where the set $\hat{\mathcal{A}}$ is defined in (40). The equality in (66) is obtained by performing a change of measure using Theorem 2 under assumption (a); and the equality in (69) follows by exchanging the order of integration [5, Theorem 2.6.6].

The proof is completed from Theorem 1 and by combining equations (54), (59), (65) and (71), which establish (53). ■

**Theorem 12** (Inverse Bayes-like Rule). Consider the conditional probability measures $P_{Y|X}$ and $P_{X|Y}$; and the probability measures $P_Y$ and $P_X$ that satisfy (44) and (45); and the joint probability measures $P_{YX}$ and $P_{XY}$ in (39) and (46) respectively. Assume that: (a) For all $x \in \mathcal{X}$, $P_Y \ll P_{Y|\mathcal{X}=x}$; and (b) For all $y \in \mathcal{Y}$, $P_X \ll P_{X|\mathcal{Y}=y}$. Then,

$$ \frac{dP_X P_Y}{dP_{XY}} (x, y) \stackrel{a.s.}{=} \frac{dP_X}{dP_{X|Y=y}} (x) \stackrel{a.s.}{=} \frac{dP_Y}{dP_{Y|\mathcal{X}=x}} (y) \stackrel{a.s.}{=} \frac{dP_Y P_X}{dP_{YX}} (y, x). \quad (72) $$

*Proof:* The proof follows along the same lines as the proof of Theorem 11.
Note that assumptions (a) and (b) are sufficient for the Radon-Nikodym derivatives of $P_X P_Y$ with respect to $P_{XY}$; and $P_Y P_X$ with respect to $P_{YX}$ to exist. Hence, it follows that for all sets $\mathcal{A} \in \mathcal{F}_X \times \mathcal{F}_Y$,

$$P_X P_Y (\mathcal{A}) = \int_{\mathcal{A}} \frac{dP_X P_Y}{dP_{XY}}(x, y)dP_{XY}(x, y), \quad (73)$$

which follows from Theorem 2.

The proof follows by observing that for all measurable sets $\mathcal{A} \in \mathcal{X} \times \mathcal{Y}$, the product measure $P_X P_Y \in \Delta (\mathcal{X} \times \mathcal{Y})$ satisfies

$$P_X P_Y (\mathcal{A}) = \int_{\mathcal{A}} dP_X P_Y (x, y) \quad (74)$$

$$= \int \int_{\mathcal{A}_y} dP_X (x) dP_Y (y) \quad (74)$$

$$= \int \int_{\mathcal{A}_y} \frac{dP_X}{dP_{X|Y=y}} (x) dP_{X|Y=y} (x) dP_Y (y) \quad (75)$$

$$= \iint \mathbb{1}_{\mathcal{A}_y} (x) \frac{dP_X}{dP_{X|Y=y}} (x) dP_{X|Y=y} (x) dP_Y (y) \quad (76)$$

$$= \int \mathbb{1}_{\mathcal{A}_y} (x) \frac{dP_X}{dP_{X|Y=y}} (x) dP_{XY} (x, y) \quad (77)$$

$$= \int_{\mathcal{A}} \frac{dP_X}{dP_{X|Y=y}} (x) dP_{XY} (x, y), \quad (78)$$

where the set $\mathcal{A}_y$ is defined in (47); the equality in (75) follows from Assumption (b) and [11, Theorem 2.2.3]; and the measure $P_{XY}$ is defined in (46).

The proof proceeds by noticing that for all measurable sets $\mathcal{A} \in \mathcal{X} \times \mathcal{Y}$, the product measure $P_X P_Y \in \Delta (\mathcal{X} \times \mathcal{Y})$ also satisfies

$$P_X P_Y (\mathcal{A}) = \int_{\mathcal{A}} dP_X P_Y (x, y) \quad (79)$$

$$= \int \int_{\mathcal{A}_y} dP_X (x) dP_Y (y) \quad (79)$$

$$= \int \int_{\mathcal{A}_x} dP_Y (y) dP_X (x) \quad (80)$$

$$= \int \int_{\mathcal{A}_x} \frac{dP_Y}{dP_{Y|X=x}} (y) dP_{Y|X=x} (y) dP_X (x) \quad (81)$$

$$= \iint \mathbb{1}_{\mathcal{A}_x} (y) \frac{dP_Y}{dP_{Y|X=x}} (y) dP_{Y|X=x} (y) dP_X (x) \quad (82)$$

$$= \int \mathbb{1}_{\mathcal{A}_x} (y) \frac{dP_Y}{dP_{Y|X=x}} (y) dP_{YX} (y, x) \quad (83)$$

$$= \int_{\mathcal{A}} \frac{dP_Y}{dP_{Y|X=x}} (y) dP_{YX} (y, x) \quad (84)$$

$$= \int_{\mathcal{A}} \frac{dP_Y}{dP_{Y|X=x}} (y) dP_{XY} (x, y), \quad (85)$$
where the sets $\mathcal{A}_x$, $\mathcal{A}_y$, and $\tilde{\mathcal{A}}$ are defined in (38), (47), and (40); and the measure $P_{YX}$ is defined in (39). The equality in (80) follows by exchanging the order of the integrals [5, Theorem 2.6.6]; the equality in (81) follows from Assumption (a) and [11, Theorem 2.2.3].

Finally, for all measurable sets $\mathcal{A} \in \mathcal{X} \times \mathcal{Y}$ and $\tilde{\mathcal{A}}$ in (40),

$$P_X P_Y(\mathcal{A}) = \int_{\tilde{\mathcal{A}}} dP_Y P_X(y, x) \qquad (86)$$

$$= \int_{\tilde{\mathcal{A}}} \frac{dP_Y P_X}{dP_{YX}}(y, x)dP_{YX}(y, x) \qquad (87)$$

$$= \int_{\mathcal{A}} \frac{dP_Y P_X}{dP_{YX}}(y, x)dP_{XY}(x, y). \qquad (88)$$

The proof is completed from Theorem 1 and by combining equations (73), (78), (85) and (88), which establish (72). ■

**Remark 1.** An alternative proof for Theorem 10 can be written as follows, by combining Theorems 5 and 12.

$$\int \frac{dP_{Y|X=x}}{dP_Y}(y)dP_X(x) = \int \frac{dP_{Y|X=x}}{dP_Y}(y) \frac{dP_X}{dP_{X|Y=y}}(x)dP_{X|Y=y}(x) \quad (89)$$

$$= \int \frac{dP_{Y|X=x}}{dP_Y}(y) \frac{dP_Y}{dP_{Y|X=x}}(y) dP_{X|Y=y}(x) \quad (90)$$

$$= \int dP_{X|Y=y}(x) = 1, \qquad (91)$$

where the equality in (89) follows from Theorem 2; the equality in (90) follows from Theorem 12 and the equality in (91) follows from Theorem 5.

The following theorem introduces an identity involving the sum of a mutual information [13, 14] and a lautum information [33]. Such a result appeared first in [42, Corollary 13] as a byproduct of the properties of the generalization error of machine learning algorithms. In this work, a novel proof is presented in a general context exclusively using the folklore theorems on the RND described above.

**Theorem 13.** Consider the conditional probability measure $P_{Y|X}$; the probability measures $P_Y$ and $P_X$ that satisfy (44) and (45); and the joint probability measure $P_{YX}$ in (39). Let $Q$ be some $\sigma$-finite measure in $\triangle (\mathcal{Y})$ and assume that for all $x \in \mathcal{X}$, $P_Y \ll P_{Y|X=x} \ll Q \ll P_Y$. Then,

$$I(P_{Y|X}; P_X) + L(P_{Y|X}; P_X) = \iint \log \left( \frac{dP_{Y|X=x}}{dQ}(y) \right) dP_{Y|X=x}(y)dP_X(x) \\ - \iint \log \left( \frac{dP_{Y|X=x}}{dQ}(y) \right) dP_Y(y)dP_X(x), \quad (92)$$

where $I(P_{Y|X}; P_X)$ and $L(P_{Y|X}; P_X)$ are respectively the mutual information and lautum information induced by the joint probability measure $P_{YX}$ in (39).
*Proof:* From the definition of mutual information, it follows that

$$I(P_{Y|X}; P_X) = \iint \log\left(\frac{dP_{Y|X=x}}{dP_Y}(y)\right) dP_{Y|X=x}(y)dP_X(x) \quad (93)$$

$$= \iint \log\left(\frac{dP_{Y|X=x}}{dQ}(y)\frac{dQ}{dP_Y}(y)\right) dP_{Y|X=x}(y)dP_X(x) \quad (94)$$

$$= \iint \log\left(\frac{dP_{Y|X=x}}{dQ}(y)\right) dP_{Y|X=x}(y)dP_X(x)$$

$$+ \iint \log\left(\frac{dQ}{dP_Y}(y)\right) dP_{Y|X=x}(y)dP_X(x), \quad (95)$$

where the equality in (94) follows from Theorem 4. The proof follows by re-writing the second term in (95). That is,

$$\iint \log\left(\frac{dQ}{dP_Y}(y)\right) dP_{Y|X=x}(y)dP_X(x)$$

$$= \iint \left(\log\left(\frac{dQ}{dP_Y}(y)\right) \frac{dP_{Y|X=x}}{dP_Y}(y)\right) dP_Y(y)dP_X(x) \quad (96)$$

$$= \int \log\left(\frac{dQ}{dP_Y}(y)\right) \left(\int \frac{dP_{Y|X=x}}{dP_Y}(y)dP_X(x)\right) dP_Y(y) \quad (97)$$

$$= \int \log\left(\frac{dQ}{dP_Y}(y)\right) dP_Y(y), \quad (98)$$

where the equality in (96) follows from Theorem 2; the equality in (97) is a reorganization of terms in the integrals; and the equality in (98) follows from Theorem 10. From the definition of lautum information, it follows that

$$L(P_{Y|X}; P_X) = \iint \log\left(\frac{dP_Y}{dP_{Y|X=x}}(y)\right) dP_Y(y)dP_X(x)$$

$$= \iint \log\left(\frac{dP_Y}{dQ}(y)\frac{dQ}{dP_{Y|X=x}}(y)\right) dP_Y(y)dP_X(x) \quad (99)$$

$$= \int \log\left(\frac{dP_Y}{dQ}(y)\right) dP_Y(y)$$

$$- \iint \log\left(\frac{dP_{Y|X=x}}{dQ}(y)\right) dP_Y(y)dP_X(x), \quad (100)$$

where (99) follows from Theorem 4 and (100) is a direct consequence of the properties of the logarithmic function and Theorem 5. The proof is completed by plugging (98) in (95); and adding the resulting (95) to (100). ■

The relevance of expressing the sum of a mutual information and a lautum information in terms of a free parameter (i.e., the measure $Q$ in (92)) lies in a hypothesis-testing interpretation for the term $\log\left(\frac{dP_{Y|X=x}}{dQ}(y)\right)$, which can be viewed as a log-likelihood ratio for distinguishing the hypothesis $Y \sim P_{Y|X=x}$ from $Y \sim Q$, given some data point $(x, y)$. Consequently, such a sum captures
the variation of the expectation of such a log-likelihood ratio when the ground
truth probability measure shifts from $P_Y P_X$ to $P_{XY}$. This interpretation has
played a central role for establishing bridges between the generalization theory
and hypothesis testing [42].

References

[1] D. Fudenberg and E. Maskin, “The Folk theorem in repeated games with discounting or with incomplete information,” *Econometrica*, vol. 54, no. 3, pp. 533–554, 1986.

[2] D. Fudenberg, *Game Theory*, 1st ed. Cambridge, MA, USA: MIT Press, 1991.

[3] J. Radon, *Theorie und Anwendungen der absolut additiven Mengenfunktionen*, 1st ed. Vienna, Austria: Hölder, 1913.

[4] O. Nikodym, “Sur une généralisation des intégrales de M. J. Radon,” *Fundamenta Mathematicae*, vol. 15, no. 1, pp. 131–179, 1930.

[5] R. B. Ash and C. A. Doleans-Dade, *Probability and Measure Theory*, 2nd ed. Burlington, MA, USA: Academic Press, 2000.

[6] P. R. Halmos, *Measure Theory*, 1st ed. Princeton, NJ, USA: Van Nostrand, 1950.

[7] P. Billingsley, *Probability and Measure*, 3rd ed. Hoboken, NJ, USA: John Wiley & Sons, 2012.

[8] H. L. Royden and P. M. Fitzpatrick, *Real Analysis*, 4th ed. Beijing, PRC: China Machine Press, 2010.

[9] S. Axler, *Measure, Integration & Real Analysis*, 1st ed. New York, NY, USA: Springer, 2020.

[10] R. Durrett, *Probability: Theory and Examples*, 5th ed. Cambridge, UK: Cambridge University Press, 2019.

[11] E. L. Lehmann, J. P. Romano, and G. Casella, *Testing Statistical Hypotheses*, 3rd ed. New York, NY, USA: Springer, 2005.

[12] G. B. Folland, *Real Analysis: Modern Techniques and Their Applications*, 2nd ed. New York, NY, USA: John Wiley & Sons, 1999.

[13] C. E. Shannon, “A mathematical theory of communication,” *The Bell System Technical Journal*, vol. 27, pp. 379–423, Jul. 1948.

[14] —, “A mathematical theory of communication,” *The Bell System Technical Journal*, vol. 27, pp. 623–656, Oct. 1948.
[15] S. Kullback, *Information Theory and Statistics*, 1st ed. New York, NY, USA: John Wiley & Sons, Inc., 1959.

[16] T. M. Cover and J. A. Thomas, *Elements of Information Theory*, 2nd ed. Hoboken, NJ: Wiley-Interscience, 2006.

[17] R. B. Ash, *Information Theory*, 1st ed. Mineola, NY, USA: Dover Publications, 1990.

[18] M. Bloch and J. Barros, *Physical-Layer Security: From Information Theory to Security Engineering*, 1st ed. Cambridge, UK: Cambridge University Press, 2011.

[19] E. Çinlar, *Probability and Stochastics*, 1st ed. New York, NY, USA: Springer, 2011.

[20] I. Csiszár and J. Körner, *Information Theory: Coding Theorems for Discrete Memoryless Systems*, 1st ed. Cambridge, UK: Cambridge University Press, 2011.

[21] A. El Gamal and Y.-H. Kim, *Network Information Theory*, 1st ed. Cambridge, UK: Cambridge University Press, 2011.

[22] R. M. Fano, *Transmission of Information: A Statistical Theory of Communications*, 1st ed. Cambridge, MA, USA: MIT Press, 1961.

[23] A. Feinstein, *Foundations of Information Theory*, 1st ed. New York, NY, USA: McGraw-Hill, 1958.

[24] R. G. Gallager, *Information Theory and Reliable Communication*, 1st ed. New York, NY, USA: Wiley, 1968.

[25] T. S. Han, *Information-Spectrum Methods in Information Theory*, 1st ed. Berlin, Germany: Springer, 2003.

[26] J. Kapur, *Maximum-Entropy Models in Science and Engineering*, 1st ed. New York, NY, USA: John Wiley & Sons, 1989.

[27] R. J. McEliece, *The Theory of Information and Coding*, 1st ed. Cambridge, UK: Cambridge University Press, 2002.

[28] M. Mezard and A. Montanari, *Information, Physics, and Computation*, 1st ed. Oxford, UK: Oxford University Press, 2009.

[29] M. Pinsker, *Information and Information Stability of Random Variables and Processes*, 1st ed. San Francisco, CA, USA: Holden-Day, 1964.

[30] J. Wolfowitz, *Coding Theorems of Information Theory*, 1st ed. Berlin, Germany: Springer, 1964.

[31] R. W. Yeung, *Information Theory and Network Coding*, 1st ed. New York, NY, USA: Springer, 2008.
[32] Y. Polyanskiy and Y. Wu, *Information Theory: From Coding to Learning*, 1st ed. Cambridge, UK: Cambridge University Press, 2024.

[33] D. P. Palomar and S. Verdú, “Lautum information,” *IEEE Transactions on Information Theory*, vol. 54, no. 3, pp. 964–975, Mar. 2008.

[34] I. Sason and S. Verdú, “f-divergence inequalities,” *IEEE Transactions on Information Theory*, vol. 62, no. 11, pp. 5973–6006, Jun. 2016.

[35] F. Daunas, I. Esnaola, S. M. Perlaza, and H. V. Poor, “Equivalence of empirical risk minimization to regularization on the family of f-divergences,” in *2024 IEEE International Symposium on Information Theory (ISIT)*. IEEE, 2024, pp. 759–764.

[36] ——, “Asymmetry of the relative entropy in the regularization of empirical risk minimization,” *arXiv preprint arXiv:2410.02833*, 2024.

[37] R. Agrawal and T. Horel, “Optimal bounds between f-divergences and integral probability metrics,” *The Journal of Machine Learning Research*, vol. 22, no. 1, pp. 5662–5720, Jan. 2021.

[38] A. R. Asadi and E. Abbe, “Chaining meets chain rule: Multilevel entropic regularization and training of neural networks.” *J. Mach. Learn. Res.*, vol. 21, pp. 139–1, Jun. 2020.

[39] S. Masiha, A. Gohari, and M. H. Yassaee, “f-divergences and their applications in lossy compression and bounding generalization error,” *IEEE Transactions on Information Theory*, vol. 1, no. 1, pp. 1–24, Apr. 2023.

[40] B. Rodríguez-Gálvez, “An information-theoretic approach to generalization theory,” Ph.D. dissertation, KTH Royal Institute of Technology, 2024.

[41] R. C. Yavas, V. Kostina, and M. Effros, “Variable-length sparse feedback codes for point-to-point, multiple access, and random access channels,” *IEEE Transactions on Information Theory*, 2023.

[42] S. M. Perlaza and X. Zou, “The generalization error of machine learning algorithms,” *arXiv preprint arXiv:2411.12030*, 2024.

[43] S. Verdú, “Relative information spectra with applications to statistical inference,” *AIMS Mathematics*, vol. 9, no. 12, pp. 35 038–35 090, 2024.

[44] X. Zou, S. M. Perlaza, I. Esnaola, E. Altman, and H. V. Poor, “The worst-case data-generating probability measure in statistical learning,” *IEEE Journal on Selected Areas in Information Theory*, vol. 5, p. 175–189, Apr. 2024.

[45] S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, “Empirical risk minimization with relative entropy regularization,” *IEEE Transactions on Information Theory*, vol. 70, no. 7, pp. 5122–5161, Jul. 2024.
[46] G. Aminian, Y. Bu, L. Toni, M. R. D. Rodrigues, and G. W. Wornell, "Information-theoretic characterizations of generalization error for the Gibbs algorithm," *IEEE Transactions on Information Theory*, vol. 70, no. 1, pp. 632-655, 2024.
informatics mathematics
Inria

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex

Publisher
Inria
Domaine de Voluceau - Rocquencourt
BP 105 - 78153 Le Chesnay Cedex
inria.fr

ISSN 0249-6399