informatics mathematics
Inria

Variations on the
Expectation due to
Changes in the
Probability Measure

Samir M. Perlaza and Gaetan Bisson

RESEARCH
REPORT

Nº 9592

July 23, 2025

Project-Team NEO
<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# Variations on the Expectation due to Changes in the Probability Measure

Samir M. Perlaza and Gaetan Bisson

Project-Team NEO

Research Report n° 9592 — version 2 — initial version July 23, 2025 —
revised version August 15, 2025 — 28 pages

**Abstract:** In this research report, closed-form expressions for the variation of the expectation of a given function due to changes in the probability measure (probability distribution drifts) are presented. These expressions unveil interesting connections with Gibbs probability measures, information projections, Pythagorean identities for relative entropy, mutual information, and lautum information.

**Key-words:** Gibbs probability measures; Pythagorean identities; sensitivity; probability distribution drifts; variations of the expectation

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex

<footnote>This research was supported in part by the European Commission through the H2020-MSCA-RISE-2019 project 872172; the French National Agency for Research (ANR) through the Project ANR-21-CE25-0013 and the project ANR-22-PEFT-0010 of the France 2030 program PEPR Réseaux du Futur; and in part by the Agence de l'innovation de défense (AID) through the project UK-FR 2024352.</footnote>

<footnote>Samir M. Perlaza is with the Centre Inria d'Université Côte d'Azur at INRIA, Sophia Antipolis 06902, France. He is also with the ECE Dept. at Princeton University, Princeton N.J. 08544, USA. Gaetan Bisson and Samir M. Perlaza are with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia.</footnote>

<footnote>This work was submitted for publication to Entropy [1].</footnote>
# Variations de l'Espérance dues aux Changements de la Mesure de Probabilité

**Résumé** : Dans ce rapport de recherche, nous présentons des expressions en forme fermée pour la variation de l'espérance d'une fonction donnée due à des changements de mesure de probabilité. Ces expressions révèlent des liens intéressants avec les mesures de probabilité de Gibbs, les projections informationnelles, les identités pythagoriciennes pour l'entropie relative, l'information mutuelle et l'information lautum.

**Mots-clés** : Mesures de probabilité de Gibbs; identités pythagoriciennes; sensibilité; dérives de distributions de probabilité; variations de l'espérance.
# Contents

<table><tr><td>1</td><td>Introduction</td><td>4</td></tr><tr><td>1.1</td><td>Novelty and Contributions</td><td>5</td></tr><tr><td>1.2</td><td>Applications</td><td>6</td></tr><tr><td>2</td><td>Preliminaries</td><td>6</td></tr><tr><td>3</td><td>Characterization of G<sub>h</sub> (x, P<sub>1</sub>, P<sub>2</sub>) in (2)</td><td>12</td></tr><tr><td>4</td><td>Characterizations of G&#773;<sub>h</sub> (P<sub>Y|X</sub><sup>(1)</sup>, P<sub>Y|X</sub><sup>(2)</sup>, P<sub>X</sub>) in (4)</td><td>14</td></tr><tr><td>5</td><td>Characterizations of G&#773;<sub>h</sub> (P<sub>Y</sub>, P<sub>Y|X</sub>, P<sub>X</sub>) in (10)</td><td>16</td></tr><tr><td>6</td><td>Examples</td><td>18</td></tr><tr><td>7</td><td>Final Remarks</td><td>21</td></tr><tr><td colspan="2">Appendices</td><td>22</td></tr><tr><td>A</td><td>Proof of Theorem 11</td><td>22</td></tr><tr><td>B</td><td>Proof of Theorem 12</td><td>23</td></tr><tr><td colspan="2">References</td><td>28</td></tr></table>
# 1 Introduction

Let $m$ be a positive integer and denote by $\Delta(\mathbb{R}^m)$ the set of all probability measures on the measurable space $(\mathbb{R}^m, \mathcal{B}(\mathbb{R}^m))$, with $\mathcal{B}(\mathbb{R}^m)$ being the Borel $\sigma$-algebra on $\mathbb{R}^m$. Given a Borel measurable function $h: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}$, consider the functional

$$ G_h : \begin{cases} \mathbb{R}^n \times \Delta(\mathbb{R}^m) \times \Delta(\mathbb{R}^m) \to \mathbb{R} \\ (x, P_1, P_2) \mapsto \int h(x, y)dP_1(y) - \int h(x, y)dP_2(y). \end{cases} \quad (1) $$

The functional $G_h$ in (1) is defined when both integrals exist and are finite. Hence,

$$ G_h(x, P_1, P_2) = \int h(x, y)dP_1(y) - \int h(x, y)dP_2(y), \quad (2) $$

is the variation of the expectation of the measurable function $h$ due to a change in the probability measure from $P_2$ to $P_1$. These changes in the probability measure are often referred to as probability distribution drifts in some application areas. See for instance [2–4] and references therein.

In order to define the expectation of $G_h(x, P_1, P_2)$ in (2) when $x$ is obtained by sampling a probability measure in $\Delta(\mathbb{R}^n)$, the structure formalized below is required.

**Definition 1.** A family $P_{Y|X} \triangleq (P_{Y|X=x})_{x \in \mathbb{R}^n}$ of elements of $\Delta(\mathbb{R}^m)$ indexed by $\mathbb{R}^n$ is said to be a conditional probability measure if, for all sets $A \in \mathcal{B}(\mathbb{R}^m)$, the map

$$ \begin{cases} \mathbb{R}^n \to [0, 1] \\ x \mapsto P_{Y|X=x}(A) \end{cases} $$

is Borel measurable. The set of all such conditional probability measures is denoted by $\Delta(\mathbb{R}^m|\mathbb{R}^n)$.

In this setting, consider the functional

$$ \bar{G}_h : \begin{cases} \Delta(\mathbb{R}^m|\mathbb{R}^n) \times \Delta(\mathbb{R}^m|\mathbb{R}^n) \times \Delta(\mathbb{R}^n) \to \mathbb{R} \\ (P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X) \mapsto \int G_h(x, P_{Y|X=x}^{(1)}, P_{Y|X=x}^{(2)}) dP_X(x) \end{cases} \quad (3) $$

Hence,

$$ \bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X) = \int G_h(x, P_{Y|X=x}^{(1)}, P_{Y|X=x}^{(2)}) dP_X(x) \quad (4) $$

$$ = \int \left( \int h(x, y)dP_{Y|X=x}^{(1)}(y) - \int h(x, y)dP_{Y|X=x}^{(2)}(y) \right) dP_X(x) \quad (5) $$

$$ = \int h(x, y)dP_{Y|X}^{(1)} P_X(y, x) - \int h(x, y)dP_{Y|X}^{(2)} P_X(y, x), \quad (6) $$
where the functional $\mathcal{G}_h$ is defined in (1), is the variation of the integral (expectation) of the function $h$ when the probability measure changes from the joint probability measure $P_{Y|X}^{(1)}P_X$ to another joint probability measure $P_{Y|X}^{(2)}P_X$, both in $\Delta(\mathbb{R}^m \times \mathbb{R}^n)$.

Special attention is given to the quantity $\bar{\mathcal{G}}_h(P_Y, P_{Y|X}, P_X)$, for some $P_{Y|X} \in \Delta(\mathbb{R}^m|\mathbb{R}^n)$, with $P_Y$ being the marginal on $\Delta(\mathbb{R}^m)$ of the joint probability measure $P_{Y|X}P_X$ on $\Delta(\mathbb{R}^m \times \mathbb{R}^n)$. That is, for all sets $\mathcal{A} \in \mathcal{B}(\mathbb{R}^m)$,

$$P_Y(\mathcal{A}) = \int P_{Y|X=x}(\mathcal{A}) dP_X(x). \qquad (7)$$

The relevance of the quantity $\bar{\mathcal{G}}_h(P_Y, P_{Y|X}, P_X)$ stems from the fact that it captures the variation of the expectation of the function $h$ when the probability measure changes from the joint probability measure $P_{Y|X}P_X$ to the product of its marginals $P_Y P_X$. That is,

$$\bar{\mathcal{G}}_h(P_Y, P_{Y|X}, P_X) = \int \mathcal{G}_h(x, P_Y, P_{Y|X=x}) dP_X(x) \qquad (8)$$

$$= \int \left( \int h(x, y)dP_Y(y) - \int h(x, y)dP_{Y|X=x}(y) \right) dP_X(x) \qquad (9)$$

$$= \int h(x, y)dP_Y P_X(y, x) - \int h(x, y)dP_{Y|X} P_X(y, x), \qquad (10)$$

where the functional $\mathcal{G}_h$ is defined in (1).

## 1.1 Novelty and Contributions

This work makes two key contributions: First, it provides a closed-form expression for the variation $\mathcal{G}_h(x, P_1, P_2)$ in (2) for a fixed $x \in \mathbb{R}^n$ and two arbitrary probability measures $P_1$ and $P_2$, formulated explicitly in terms of relative entropies. Second, it derives a closed-form expression for the expected variation $\bar{\mathcal{G}}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4), again in terms of information measures, for arbitrary conditional probability measures $P_{Y|X}^{(1)}, P_{Y|X}^{(2)}$, and an arbitrary probability measure $P_X$.

A further contribution of this work is the derivation of specific closed-form expressions for $\bar{\mathcal{G}}_h(P_Y, P_{Y|X}, P_X)$ in (10), which reveal deep connections with both mutual information [5] and lautum information [6]. Notably, when $P_{Y|X}$ is a Gibbs conditional probability measure, this variation simplifies (up to a constant factor) to the sum of the mutual and lautum information induced by the joint distribution $P_{Y|X}P_X$.

These results were originally discovered in the analysis of generalization error of machine-learning algorithms, see for instance [7–11]. Therein, the function $h$ in (2) was assumed to represent an empirical risk. This paper presents such results in a comprehensive and general setting that is no longer tied to such assumptions. Also, strong connections with information projections and Pythagorean identities [12, 13] are discussed. This new general presentation not
only unifies previously scattered insights but also makes the results applicable across a broad range of domains in which probability distribution shifts are relevant.

## 1.2 Applications

The study of the variation of the integral (expectation) of $h$ (for some fixed $x \in \mathbb{R}^n$) due to a measure change from $P_2$ to $P_1$, i.e., the value $G_h(x, P_1, P_2)$ in (2), plays a central role in the definition of integral probability metrics (IPMs) [14,15]. Using the notation in (2), an IPM results from the optimization problem

$$ \sup_{h \in \mathcal{H}} |G_h(x, P_1, P_2)|, \qquad (11) $$

for some fixed $x \in \mathbb{R}^n$ and a particular class of functions $\mathcal{H}$. Note, for instance, that the maximum mean discrepancy [16] and the Wasserstein distance of order one [17–20] are both IPMs.

Other areas of mathematics in which the variation $G_h(x, P_1, P_2)$ in (2) plays a key role are distributionally robust optimization (DRO) [21,22] and optimization with relative entropy regularization [8, 9]. In these areas, the variation $G_h(x, P_1, P_2)$ is a central tool. See for instance, [7,23] and references therein.

Variations of the form $G_h(x, P_1, P_2)$ in (2) have also been studied in [10, 11] in the particular case of statistical machine learning for the analysis of generalization error. The central observation is that the generalization error of machine-learning algorithms can be written in the form $\tilde{G}_h(P_Y, P_{Y|X}, P_X)$ in (10). This observation is the main building block of the method of gaps introduced in [11], which leads to a number of closed-form expressions for the generalization error involving mutual information, lautum information, among other information measures.

## 2 Preliminaries

The main results presented in this work involve Gibbs conditional probability measures. Such measures are parametrized by a Borel measurable function $h: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}$; a $\sigma$-finite measure $Q$ on $\mathbb{R}^m$; and a real $\lambda \in \mathbb{R}$. Often, the measure $Q$ is called the reference measure, and the real parameter $\lambda$ is called the *temperature* parameter. The use of a $\sigma$-finite reference measure was first introduced in [8]. This class of measures includes the Lebesgue measure and the counting measure, enabling a unified treatment of random variables with either probability density functions or probability mass functions. The standard case in which the reference measure is a probability measure representing a prior is also covered. Finally, note that the variable $x$ will remain inactive until Section 4. However, it is introduced now for consistency.

Consider the following function:

$$ K_{h,Q,x}: \begin{cases} \mathbb{R} \to \mathbb{R} \\ t \mapsto \log \left( \int \exp(t h(x, y)) dQ(y) \right). \end{cases} \qquad (12) $$

Inria
Under the assumption that $Q$ is a probability measure, the function $K_{h,Q,x}$ in (12) is the cumulant generating function of the random variable $h(x, Y)$, for some fixed $x \in \mathbb{R}^n$ and $Y \sim Q$. Using this notation, the definition of the Gibbs conditional probability measure is presented hereunder.

**Definition 2** (Gibbs Conditional Probability Measure). Given a Borel measurable function $h: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}$; a $\sigma$-finite measure $Q$ on $\mathbb{R}^m$; and a $\lambda \in \mathbb{R}$, the probability measure $P_{Y|X}^{(h,Q,\lambda)} \in \Delta(\mathbb{R}^m|\mathbb{R}^n)$ is said to be an $(h,Q,\lambda)$-Gibbs conditional probability measure if

$$ \forall x \in \mathcal{X}, K_{h,Q,x}(-\lambda) < +\infty; \quad (13) $$

for some set $\mathcal{X} \subseteq \mathbb{R}^n$; and for all $(x, y) \in \mathcal{X} \times \text{supp } Q$,

$$ \frac{\mathrm{d}P_{Y|X=x}^{(h,Q,\lambda)}}{\mathrm{d}Q}(y) = \exp(-\lambda h(x, y) - K_{h,Q,x}(-\lambda)), \quad (14) $$

where the function $K_{h,Q,x}$ is defined in (12); the set $\text{supp } Q$ is the support of the $\sigma$-finite measure $Q$; and the function $\frac{\mathrm{d}P_{Y|X=x}^{(h,Q,\lambda)}}{\mathrm{d}Q}$ is the Radon-Nikodym derivative $[24, 25]$ of the probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ with respect to $Q$.

Note that, while $P_{Y|X}^{(h,Q,\lambda)}$ is an $(h,Q,\lambda)$-Gibbs conditional probability measure, the measure $P_{Y|X=x}^{(h,Q,\lambda)}$, obtained by conditioning it upon a given vector $x \in \mathcal{X}$, is referred to as an $(h,Q,\lambda)$-Gibbs probability measure.

The condition in (13) is easily met under certain assumptions. For instance, if $h$ is a nonnegative function and $Q$ is a finite measure, then it holds for all $\lambda \in (0, +\infty)$. Let $\Delta_Q(\mathbb{R}^m) \triangleq \{P \in \Delta(\mathbb{R}^m) : P \ll Q\}$, with $P \ll Q$ standing for “$P$ absolutely continuous with respect to $Q$”. The relevance of $(h,Q,\lambda)$-Gibbs probability measures relies on the fact that under some conditions, they are the unique solutions to problems of the form,

$$ \min_{P \in \Delta_Q(\mathbb{R}^m)} \int h(x, y) \mathrm{d}P(y) + \frac{1}{\lambda} D(P \| Q), \quad \text{and} \quad (15) $$

$$ \max_{P \in \Delta_Q(\mathbb{R}^m)} \int h(x, y) \mathrm{d}P(y) + \frac{1}{\lambda} D(P \| Q), \quad (16) $$

where $\lambda \in \mathbb{R} \setminus \{0\}$, $x \in \mathbb{R}$, and $D(P \| Q)$ denotes the relative entropy (or KL divergence) of $P$ with respect to $Q$.

**Definition 3** (Relative Entropy). Given two $\sigma$-finite measures $P$ and $Q$ on the same measurable space, such that $P$ is absolutely continuous with respect to $Q$, the relative entropy of $P$ with respect to $Q$ is

$$ D(P \| Q) = \int \frac{\mathrm{d}P}{\mathrm{d}Q}(x) \log \left( \frac{\mathrm{d}P}{\mathrm{d}Q}(x) \right) \mathrm{d}Q(x), \quad (17) $$

where the function $\frac{\mathrm{d}P}{\mathrm{d}Q}$ is the Radon-Nikodym derivative of $P$ with respect to $Q$.
The key observation is that when $\lambda > 0$, the objective function in (15) is convex with $P$. Alternative, when $\lambda < 0$, the objective function in (16) is concave. The connection between the optimization problems (15) and (16) and the Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ in (14) has been pointed out by several authors. See for instance, Theorem 3 in [8] and [7, 26–34] for the former; and Theorem 1 in [10], together with [35–37] for the latter. In these references, a variety of assumptions and proof techniques have been used to highlight such connections. A general and unified statement of these observations is presented hereunder.

**Lemma 1.** Assume that the optimization problem in (15) (respectively, in (16)) admits a solution. Then, if $\lambda > 0$ (respectively, if $\lambda < 0$), the probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ in (14) is the unique solution.

*Proof.* For the case in which $\lambda > 0$, the proof follows the same approach as the proof of Theorem 3 in [8]. Alternatively, for the case in which $\lambda < 0$, the proof follows along the lines of the proof of Theorem 1 in [10]. □

The following lemma highlights a key property of $(h, Q, \lambda)$-Gibbs probability measures.

**Lemma 2.** Given an $(h, Q, \lambda)$-Gibbs probability measure, denoted by $P_{Y|X=x}^{(h,Q,\lambda)}$, with $x \in \mathbb{R}^n$,

$$ -\frac{1}{\lambda}K_{h,Q,x}(-\lambda) = \int h(x,y)dQ(y) - \frac{1}{\lambda}D(Q\|P_{Y|X=x}^{(h,Q,\lambda)}) \quad (18) $$

$$ = \int h(x,y)dP_{Y|X=x}^{(h,Q,\lambda)}(y) + \frac{1}{\lambda}D(P_{Y|X=x}^{(h,Q,\lambda)}\|Q); \quad (19) $$

moreover, if $\lambda > 0$,

$$ -\frac{1}{\lambda}K_{h,Q,x}(-\lambda) = \min_{P \in \Delta_Q(\mathbb{R}^m)} \int h(x,y)dP(y) + \frac{1}{\lambda}D(P\|Q); \quad (20) $$

alternatively, if $\lambda < 0$,

$$ -\frac{1}{\lambda}K_{h,Q,x}(-\lambda) = \max_{P \in \Delta_Q(\mathbb{R}^m)} \int h(x,y)dP(y) + \frac{1}{\lambda}D(P\|Q), \quad (21) $$

where the function $K_{h,Q,x}$ is defined in (12).

*Proof.* The proof of (19) follows from taking the logarithm of both sides of (14) and integrating with respect to $P_{Y|X=x}^{(h,Q,\lambda)}$. As for the proof of (18), it follows by noticing that for all $(x, y) \in \mathbb{R}^n \times \text{supp } Q$, the Radon-Nikodym derivative $\frac{dP_{Y|X=x}^{(h,Q,\lambda)}}{dQ}(y)$ in (14) is strictly positive. Thus, from Theorem 5 in [38], it holds that $\frac{dQ}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) = \left(\frac{dP_{Y|X=x}^{(h,Q,\lambda)}}{dQ}(y)\right)^{-1}$. Hence, taking the negative logarithm on both sides of (14) and integrating with respect to $Q$ leads to (18). Finally, the equalities in (20) and (21) follow from Lemma 1 and (19). □
The equalities (19), (20), and (21) in Lemma 2 can be seen as an immediate restatement of Donsker-Varadhan variational representation of the relative entropy [39]. Alternative interesting proofs for (18) have been presented by several authors including [10] and [34]. A proof for (19) appears in [7] (Lemma 3), in the specific case of $\lambda > 0$.

The following lemma introduces the main building block of this work, which is a characterization of the variation of the expectation of the function $h(x, \cdot) : \mathbb{R}^m \to \mathbb{R}$ when the probability measure changes from the probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ in (14) to an arbitrary measure $P \in \Delta_Q(\mathbb{R}^m)$, i.e., $G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)})$, for some fixed $x \in \mathbb{R}^n$. Such a result appeared for the first time in [7] (Theorem 1) for the case in which $\lambda > 0$; and in [10] (Theorem 6) for the case in which $\lambda < 0$, in different contexts of statistical machine learning. A general and unified statement of such results is presented hereunder.

**Lemma 3.** Consider an $(h, Q, \lambda)$-Gibbs probability measure, denoted by $P_{Y|X=x}^{(h,Q,\lambda)} \in \Delta(\mathbb{R}^m)$, with $\lambda \neq 0$ and $x \in \mathbb{R}$. For all $P \in \Delta_Q(\mathbb{R}^m)$,

$$G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)}) = \frac{1}{\lambda} \left( D(P\|P_{Y|X=x}^{(h,Q,\lambda)}) + D(P_{Y|X=x}^{(h,Q,\lambda)}\|Q) - D(P\|Q) \right). \quad (22)$$

*Proof.* The proof follows along the lines of the proofs of Theorem 1 in [7] and Theorem 6 in [10] for the cases in which $\lambda > 0$ and $\lambda < 0$, respectively. A unified proof is presented hereunder by noticing that for all $P \in \Delta_Q(\mathbb{R}^m)$,

$$D(P\|P_{Y|X=x}^{(h,Q,\lambda)}) = \int \log \left( \frac{dP}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP(y) \quad (23)$$

$$= \int \log \left( \frac{dQ}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \frac{dP}{dQ}(y) \right) dP(y) \quad (24)$$

$$= \int \log \left( \frac{dQ}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP(y) + D(P\|Q) \quad (25)$$

$$= \lambda \int h(x, y)dP(y) + K_{h,Q,x}(-\lambda) + D(P\|Q) \quad (26)$$

$$= \lambda G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)}) - D(P_{Y|X=x}^{(h,Q,\lambda)}\|Q) + D(P\|Q), \quad (27)$$

where (24) follows from Theorem 4 in [38]; (26) follows from Theorem 5 in [38]; and (14); and (27) follows from (19). $\square$

It is interesting to highlight that $G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)})$ in (22) characterizes the variation of the expectation of the function $h(x, \cdot) : \mathbb{R}^m \to \mathbb{R}$, when $\lambda > 0$ (respectively, when $\lambda < 0$) and the probability measure changes from the solution to the optimization problem in (15) (respectively, in (16)) to an alternative measure $P$. This result takes another perspective if it is seen in the context of
information projections [13]. Let $Q$ be a probability measure and $S \subseteq \Delta_Q(\mathbb{R}^m)$ be a convex set. From Theorem 1 in [13], it holds that for all measures $P \in S$,

$$D(P\|Q) \ge D(P\|P^*) + D(P^*\|Q), \qquad (28)$$

where $P^*$ satisfies

$$P^* \in \arg\min_{P \in S} D(P\|Q). \qquad (29)$$

In the particular case in which the set $S$ in (29) satisfies

$$S \triangleq \{P \in \Delta_Q(\mathbb{R}^m) : \int h(x, y)dP(y) = c\}, \qquad (30)$$

for some real $c$, with the vector $x$ and the function $h$ defined in Lemma 3, the optimal measure $P^*$ in (29) is the Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ in (14), with $\lambda > 0$ chosen to satisfy

$$\int h(x, y)dP_{Y|X=x}^{(h,Q,\lambda)}(y) = c. \qquad (31)$$

The case in which the measure $Q$ in (29) is a $\sigma$-finite measure, for instance, either the Lebesgue measure or the counting measure, respectively, leads to the classical framework of differential entropy or discrete entropy maximization, which have been studied under particular assumptions on the set $S$ in [35–37].

When the reference measure $Q$ is a probability measure, under the assumption that (31) holds, it follows from Theorem 3 in [13] that for all $P \in S$, with $S$ in (30),

$$D(P\|Q)=D\left(P\left\|P_{Y|X=x}^{(h,Q,\lambda)}\right\right) + D\left(P_{Y|X=x}^{(h,Q,\lambda)}\|Q\right), \qquad (32)$$

which is known as the Pythagorean theorem for relative entropy. Such a geometric interpretation follows from admitting relative entropy as an analog of squared Euclidean distance. The first appearance of such a “Pythagorean theorem” was in [12] and was later revisited in [13]. Interestingly, the same result can be obtained from Lemma 3 by noticing that for all $P \in S$, with $S$ in (30),

$$G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)})=0. \qquad (33)$$

The converse of the Pythagorean theorem, e.g., Proposition 48 in [40], together with Lemma 3, lead to the geometric construction shown in Figure 1. A similar interpretation was also presented in [11] in the context of the generalization error of machine-learning algorithms. Nonetheless, the interpretation in Figure 1 is general and independent of such an application.

The relevance of Lemma 3 in the context of information projections follows from the fact that $Q$ might be a $\sigma$-finite measure. The class of $\sigma$-finite measures includes the class of probability measures, and thus, unifies the results separately obtained in the realm of maximum entropy methods and information-projection methods.

The following lemma highlights that $(h, Q, \lambda)$-Gibbs conditional probability measures are related to another class of optimization problems.
<figure><img src="image_3.png" /><figcaption>Figure 1: Geometric interpretation of Lemma 3, with $Q$ a probability measure.</figcaption></figure>

**Lemma 4.** Assume that the following optimization problems possess at least one solution for some $x \in \mathbb{R}^n$,

$$ \begin{array}{ll} \min_{P \in \Delta_Q(\mathbb{R}^m)} & \displaystyle \int h(x, y)dP(y) \qquad (34) \\ \text{s.t.} & D(P\|Q) \le \rho. \end{array} $$

and

$$ \begin{array}{ll} \max_{P \in \Delta_Q(\mathbb{R}^m)} & \displaystyle \int h(x, y)dP(y) \qquad (35) \\ \text{s.t.} & D(P\|Q) \le \rho. \end{array} $$

Consider the $(h, Q, \lambda)$-Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ in (14), with $\lambda \in \mathbb{R}\setminus\{0\}$ such that $\rho = D(P_{Y|X=x}^{(h,Q,\lambda)}\|Q)$. Then, the $(h, Q, \lambda)$-Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$ is a solution to (34) if $\lambda > 0$; or to (35) if $\lambda < 0$.

*Proof.* Note that if $\lambda > 0$, then, $\frac{1}{\lambda}D(P\|P_{Y|X=x}^{(h,Q,\lambda)}\|) \ge 0$. Hence, from Lemma 3, it holds that for all probability measures $P$ such that $D(P\|Q) \le \rho$,

$$ G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)}) \ge \frac{1}{\lambda} (D(P_{Y|X=x}^{(h,Q,\lambda)}\|Q) - D(P\|Q)) \qquad (36) $$

$$ = \frac{1}{\lambda} (\rho - D(P\|Q)) \qquad (37) $$

$$ \ge 0, \qquad (38) $$

with equality if $D(P\|P_{Y|X=x}^{(h,Q,\lambda)}\|) = 0$. This implies that $P_{Y|X=x}^{(h,Q,\lambda)}$ is a solution to (34). Note also that if $\lambda < 0$, from Lemma 3, it holds that for all probability
measures P such that $D(P||Q) \le \rho$,

$$G_h(x, P, P_{Y|X=x}^{(h,Q,\lambda)}) \le \frac{1}{\lambda} (D(P_{Y|X=x}^{(h,Q,\lambda)}||Q) - D(P||Q)) \quad (39)$$

$$= \frac{1}{\lambda} (\rho - D(P||Q)) \quad (40)$$

$$\le 0, \quad (41)$$

with equality if $D(P||P_{Y|X=x}^{(h,Q,\lambda)}) = 0$. This implies that $P_{Y|X=x}^{(h,Q,\lambda)}$ is a solution to (35). $\square$

### 3 Characterization of $G_h(x, P_1, P_2)$ in (2)

The main result of this section is the following theorem.

**Theorem 5.** For all probability measures $P_1$ and $P_2$, both absolutely continuous with respect to a given $\sigma$-finite measure $Q$ on $\mathbb{R}^m$, the variation $G_h(x, P_1, P_2)$ in (2) satisfies,

$$G_h(x, P_1, P_2) = \frac{1}{\lambda} \left( D(P_1||P_{Y|X=x}^{(h,Q,\lambda)}) - D(P_2||P_{Y|X=x}^{(h,Q,\lambda)}) + D(P_2||Q) - D(P_1||Q) \right), \quad (42)$$

where the probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \ne 0$, is an $(h, Q, \lambda)$-Gibbs probability measure.

*Proof.* The proof follows from Lemma 3 and by observing that

$$G_h(x, P_1, P_2) = G_h(x, P_1, P_{Y|X=x}^{(h,Q,\lambda)}) - G_h(x, P_2, P_{Y|X=x}^{(h,Q,\lambda)}),$$

which completes the proof. $\square$

Theorem 5 might be particularly simplified in the case in which the reference measure $Q$ is a probability measure. Consider for instance the case in which $P_1 \ll P_2$ (or $P_2 \ll P_1$). In such a case, the reference measure might be chosen as $P_2$ (or $P_1$), as shown hereunder.

**Corollary 6.** Consider the variation $G_h(x, P_1, P_2)$ in (2). If the probability measure $P_1$ is absolutely continuous with respect to $P_2$, then,

$$G_h(x, P_1, P_2) = \frac{1}{\lambda} \left( D(P_1||P_{Y|X=x}^{(h,P_2,\lambda)}) - D(P_2||P_{Y|X=x}^{(h,P_2,\lambda)}) - D(P_1||P_2) \right). \quad (43)$$

Alternatively, if the probability measure $P_2$ is absolutely continuous with respect to $P_1$, then,

$$G_h(x, P_1, P_2) = \frac{1}{\lambda} \left( D(P_1||P_{Y|X=x}^{(h,P_1,\lambda)}) - D(P_2||P_{Y|X=x}^{(h,P_1,\lambda)}) + D(P_2||P_1) \right), \quad (44)$$

where the probability measures $P_{Y|X=x}^{(h,P_1,\lambda)}$ and $P_{Y|X=x}^{(h,P_2,\lambda)}$ are, respectively, $(h, P_1, \lambda)$- and $(h, P_2, \lambda)$-Gibbs probability measures, with $\lambda \ne 0$.
In the case in which neither $P_1$ is absolutely continuous with respect to $P_2$; nor $P_2$ is absolutely continuous with respect to $P_1$, the reference measure $Q$ in Theorem 5 can always be chosen as a convex combination of $P_1$ and $P_2$. That is, for all Borel sets $\mathcal{A} \in \mathcal{B}(\mathbb{R}^m)$, $Q(\mathcal{A}) = \alpha P_1(\mathcal{A}) + (1-\alpha)P_2(\mathcal{A})$, with $\alpha \in (0, 1)$.

Theorem 5 can be specialized to the cases in which $Q$ is either the Lebesgue measure or the counting measure.

If $Q$ is the Lebesgue measure, then the probability measures $P_1$ and $P_2$ in (42) admit probability density functions $f_1$ and $f_2$, respectively. Moreover, the terms $-D(P_1||Q)$ and $-D(P_2||Q)$ are Shannon's differential entropies [5] induced by $P_1$ and $P_2$, denoted by $h(P_1)$ and $h(P_2)$, respectively. That is, for all $i \in \{1, 2\}$,

$$h(P_i) \triangleq - \int f_i(x) \log f_i(x) dx. \qquad (45)$$

The probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \neq 0$, $x \in \mathbb{R}^n$, and $Q$ the Lebesgue measure, possesses a probability density function, denoted by $f_{Y|X=x}^{(h,Q,\lambda)}: \mathbb{R}^m \to (0, +\infty)$, which satisfies

$$f_{Y|X=x}^{(h,Q,\lambda)}(y) = \frac{\exp(-\lambda h(x, y))}{\int \exp(-\lambda h(x, y)) dy}. \qquad (46)$$

If $Q$ is the counting measure, then the probability measures $P_1$ and $P_2$ in (42) admit probability mass functions $p_1: \mathcal{Y} \to [0, 1]$ and $p_2: \mathcal{Y} \to [0, 1]$, with $\mathcal{Y}$ a countable subset of $\mathbb{R}^m$. Moreover, $-D(P_1||Q)$ and $-D(P_2||Q)$ are, respectively, Shannon's discrete entropies [5] induced by $P_1$ and $P_2$, denoted by $H(P_1)$ and $H(P_2)$, respectively. That is, for all $i \in \{1, 2\}$,

$$H(P_i) \triangleq - \sum_{y \in \mathcal{Y}} p_i(y) \log p_i(y). \qquad (47)$$

The probability measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \neq 0$ and $Q$ the counting measure, possesses a probability mass function, denoted by $p_{Y|X=x}^{(h,Q,\lambda)}: \mathcal{Y} \to (0, +\infty)$, which satisfies

$$p_{Y|X=x}^{(h,Q,\lambda)}(y) = \frac{\exp(-\lambda h(x, y))}{\sum_{y \in \mathcal{Y}} \exp(-\lambda h(x, y))}. \qquad (48)$$

These observations lead to the following corollary of Theorem 5.

**Corollary 7.** Given two probability measures $P_1$ and $P_2$, with probability density functions $f_1$ and $f_2$, respectively, the variation $G_h(x, P_1, P_2)$ in (2) satisfies,

$$G_h(x, P_1, P_2) = \frac{1}{\lambda} \left( D(P_1 || P_{Y|X=x}^{(h,Q,\lambda)}) - D(P_2 || P_{Y|X=x}^{(h,Q,\lambda)}) - h(P_2) + h(P_1) \right), (49)$$

where the probability density function of the measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \neq 0$ and $Q$ the Lebesgue measure, is defined in (46); and the entropy functional $h$ is
defined in (45). Alternatively, given two probability measures $P_1$ and $P_2$, with probability mass functions $p_1$ and $p_2$, respectively, the variation $G_h(x, P_1, P_2)$ in (2) satisfies,

$$G_h(x, P_1, P_2) = \frac{1}{\lambda} \left( D \left( P_1 \left\| P_{Y|X=x}^{(h,Q,\lambda)} \right. \right) - D \left( P_2 \left\| P_{Y|X=x}^{(h,Q,\lambda)} \right. \right) - H(P_2) + H(P_1) \right), (50)$$

where the probability mass function of the measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \neq 0$ and $Q$ the counting measure, is defined in (48); and the entropy functional $H$ is defined in (47).

# 4 Characterizations of $\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4)

The main result of this section is a characterization of $\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4).

**Theorem 8.** Consider the variation $\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4) and assume that for all $x \in \text{supp } P_X$, the probability measures $P_{Y|X=x}^{(1)}$ and $P_{Y|X=x}^{(2)}$ are both absolutely continuous with respect to a $\sigma$-measure $Q$. Then,

$$\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X) = \frac{1}{\lambda} \int \left( D \left( P_{Y|X=x}^{(1)} \left\| P_{Y|X=x}^{(h,Q,\lambda)} \right. \right) - D \left( P_{Y|X=x}^{(2)} \left\| P_{Y|X=x}^{(h,Q,\lambda)} \right. \right) \right. \\ \left. + D \left( P_{Y|X=x}^{(2)} \left\| Q \right. \right) - D \left( P_{Y|X=x}^{(1)} \left\| Q \right. \right) \right) dP_X(x), \quad (51)$$

where the probability measure $P_{Y|X}^{(h,Q,\lambda)}$, with $\lambda \neq 0$, is an $(h,Q,\lambda)$-Gibbs conditional probability measure.

*Proof.* The proof follows from (4) and Theorem 5. □

Two special cases are particularly noteworthy. When the reference measure $Q$ is the Lebesgue measure both $-\int D(P_{Y|X=x}^{(1)} \| Q) dP_X(x)$ and $-\int D(P_{Y|X=x}^{(2)} \| Q) dP_X(x)$ in (51) become Shannon's differential conditional entropies, denoted by $h(P_{Y|X}^{(1)}|P_X)$ and $h(P_{Y|X}^{(2)}|P_X)$, respectively. That is, for all $i \in \{1, 2\}$,

$$h(P_{Y|X}^{(i)}|P_X) \triangleq \int h(P_{Y|X=x}^{(i)}) dP_X(x), \quad (52)$$

where $h$ is the entropy functional in (45).

When the reference measure $Q$ is the counting measure both terms $-\int D(P_{Y|X=x}^{(1)} \| Q) dP_X(x)$ and $-\int D(P_{Y|X=x}^{(2)} \| Q) dP_X(x)$ in (51) become Shannon's discrete conditional entropies, denoted by $H(P_{Y|X}^{(1)}|P_X)$ and $H(P_{Y|X}^{(2)}|P_X)$,
respectively. That is, for all $i \in \{1, 2\}$,

$$ H(P_{Y|X}^{(i)}|P_X) \triangleq \int H(P_{Y|X=x}^{(i)}) dP_X(x), \quad (53) $$

where $H$ is the entropy functional in (47).

These observations lead to the following corollary of Theorem 8.

**Corollary 9.** Consider the variation $\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4) and assume that for all $x \in \text{supp } P_X$, the probability measures $P_{Y|X=x}^{(1)}$ and $P_{Y|X=x}^{(2)}$ possess probability density functions. Then,

$$ \begin{aligned} \bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X) &= \frac{1}{\lambda} \int \left( D(P_{Y|X=x}^{(1)} \| P_{Y|X=x}^{(h,Q,\lambda)} - D(P_{Y|X=x}^{(2)} \| P_{Y|X=x}^{(h,Q,\lambda)}) \right) dP_X(x) \\ &\quad - \frac{1}{\lambda} h(P_{Y|X}^{(2)}|P_X) + \frac{1}{\lambda} h(P_{Y|X}^{(1)}|P_X), \end{aligned} \quad (54) $$

where the probability density function of the measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \neq 0$ and $Q$ the Lebesgue measure, is defined in (46); and for all $i \in \{1, 2\}$, the conditional entropy $h(P_{Y|X}^{(i)}|P_X)$ is defined in (52). Alternatively, assume that for all $x \in \text{supp } P_X$, the probability measures $P_{Y|X=x}^{(1)}$ and $P_{Y|X=x}^{(2)}$ possess probability mass functions. Then,

$$ \begin{aligned} \bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X) &= \frac{1}{\lambda} \int \left( D(P_{Y|X=x}^{(1)} \| P_{Y|X=x}^{(h,Q,\lambda)} - D(P_{Y|X=x}^{(2)} \| P_{Y|X=x}^{(h,Q,\lambda)}) \right) dP_X(x) \\ &\quad - \frac{1}{\lambda} H(P_{Y|X}^{(2)}|P_X) + \frac{1}{\lambda} H(P_{Y|X}^{(1)}|P_X), \end{aligned} \quad (55) $$

where the probability mass function of the measure $P_{Y|X=x}^{(h,Q,\lambda)}$, with $\lambda \neq 0$ and $Q$ the counting measure, is defined in (48); and for all $i \in \{1, 2\}$, the conditional entropy $H(P_{Y|X}^{(i)}|P_X)$ is defined in (53).

The general expression for the expected variation $\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4) might be simplified according to Corollary 6. For instance, if for all $x \in \text{supp } P_X$, the probability measure $P_{Y|X=x}^{(1)}$ is absolutely continuous with respect to $P_{Y|X=x}^{(2)}$, the measure $P_{Y|X=x}^{(2)}$ can be chosen to be the reference measure in the calculation of $G_h(x, P_{Y|X=x}^{(1)}, P_{Y|X=x}^{(2)})$, with the functional $G_h$ in (1). This observation leads to the following corollary of Theorem 8.

**Corollary 10.** Consider the variation $\bar{G}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$ in (4) and assume that for all $x \in \text{supp } P_X$, $P_{Y|X=x}^{(1)} \ll P_{Y|X=x}^{(2)}$. Then,
$$
\begin{equation}
\begin{split}
\bar{G}_h \left( P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X \right) ={}& \frac{1}{\lambda} \int \left( D \left( P_{Y|X=x}^{(1)} \left\| P_{Y|X=x}^{(h, P_{Y|X=x}^{(2)}, \lambda)} \right. \right) - D \left( P_{Y|X=x}^{(2)} \left\| P_{Y|X=x}^{(h, P_{Y|X=x}^{(2)}, \lambda)} \right. \right) \right. \\
& \left. - D \left( P_{Y|X=x}^{(1)} \left\| P_{Y|X=x}^{(2)} \right. \right) \right) dP_X(x).
\end{split}
\tag{56}
\end{equation}
$$

Alternatively, if for all $x \in \text{supp } P_X$, the probability measure $P_{Y|X=x}^{(2)}$ is absolutely continuous with respect to $P_{Y|X=x}^{(1)}$, then,

$$
\begin{equation}
\begin{split}
\bar{G}_h \left( P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X \right) ={}& \frac{1}{\lambda} \int \left( D \left( P_{Y|X=x}^{(1)} \left\| P_{Y|X=x}^{(h, P_{Y|X=x}^{(1)}, \lambda)} \right. \right) - D \left( P_{Y|X=x}^{(2)} \left\| P_{Y|X=x}^{(h, P_{Y|X=x}^{(1)}, \lambda)} \right. \right) \right) \\
& + D \left( P_{Y|X=x}^{(2)} \left\| P_{Y|X=x}^{(1)} \right. \right) dP_X(x),
\end{split}
\tag{57}
\end{equation}
$$

where the measures $P_{Y|X=x}^{(h, P_{Y|X=x}^{(1)}, \lambda)}$ and $P_{Y|X=x}^{(h, P_{Y|X=x}^{(2)}, \lambda)}$ are, respectively, $(h, P_{Y|X=x}^{(1)}, \lambda)$- and $(h, P_{Y|X=x}^{(2)}, \lambda)$-Gibbs probability measures.

The Gibbs probability measures $P_{Y|X=x}^{(h, P_{Y|X=x}^{(1)}, \lambda)}$ and $P_{Y|X=x}^{(h, P_{Y|X=x}^{(2)}, \lambda)}$ in Corollary 10 are particularly interesting as their reference measures depend on $x$. Gibbs measures of this form appear, for instance, in Corollary 10 in [8].

5 Characterizations of $\bar{G}_h(P_Y, P_{Y|X}, P_X)$ in (10)

The main result of this section is a characterization of $\bar{G}_h(P_Y, P_{Y|X}, P_X)$ in (10), which describes the variation of the expectation of the function $h$ when the probability measure changes from the joint probability measure $P_{Y|X}P_X$ to the product of its marginals $P_Y P_X$. This result is presented hereunder and involves the mutual information $I(P_{Y|X}; P_X)$ and lautum information $L(P_{Y|X}; P_X)$, defined as follows:

$$
I(P_{Y|X}; P_X) \triangleq \int D(P_{Y|X=x} \| P_Y) dP_X(x); \text{ and} \tag{58}
$$

$$
L(P_{Y|X}; P_X) \triangleq \int D(P_Y \| P_{Y|X=x}) dP_X(x). \quad (59)
$$

**Theorem 11.** Consider the expected variation $\bar{G}_h(P_Y, P_{Y|X}, P_X)$ in (10) and assume that, for all $x \in \text{supp } P_X$:

1. The probability measures $P_Y$ and $P_{Y|X=x}$ are both absolutely continuous with respect to a given $\sigma$-finite measure $Q$; and

2. The probability measures $P_Y$ and $P_{Y|X=x}$ are mutually absolutely continuous.
Then,

$$ 
\begin{aligned}
\bar{G}_h(P_Y, P_{Y|X}, P_X) = & \frac{1}{\lambda} \left( I(P_{Y|X}; P_X) + L(P_{Y|X}; P_X) \right. \\
& \left. + \iint \log \left( \frac{dP_{Y|X=x}}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP_Y(y) dP_X(x) \right. \\
& \left. - \iint \log \left( \frac{dP_{Y|X=x}}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP_{Y|X=x}(y) dP_X(x) \right), \quad (60)
\end{aligned}
$$

where the probability measure $P_{Y|X}^{(h,Q,\lambda)}$, with $\lambda \neq 0$, is an $(h, Q, \lambda)$-Gibbs conditional probability measure.

*Proof.* The proof is presented in Appendix A. □

An alternative expression for $\bar{G}_h(P_Y, P_{Y|X}, P_X)$ in (10) involving only relative entropies is presented by the following theorem.

**Theorem 12.** Consider the expected variation $\bar{G}_h(P_Y, P_{Y|X}, P_X)$ in (10) and assume that, for all $x \in \text{supp } P_X$, the probability measure $P_{Y|X=x}$ is absolutely continuous with respect to a given $\sigma$-finite measure $Q$. Then, it follows that

$$ \bar{G}_h(P_Y, P_{Y|X}, P_X) = \frac{1}{\lambda} \iint \left( D(P_{Y|X=x_2} \| P_{Y|X=x_1}^{(h,Q,\lambda)}) - D(P_{Y|X=x_2} \| P_{Y|X=x_2}^{(h,Q,\lambda)}) \right) dP_X(x_1)dP_X(x_2), (61) $$

where $P_{Y|X}^{(h,Q,\lambda)}$, with $\lambda \neq 0$, is an $(h, Q, \lambda)$-Gibbs conditional probability measure.

*Proof.* The proof is presented in Appendix B. □

Theorem 12 expresses the variation $\bar{G}_h(P_Y, P_{Y|X}, P_X)$ in (10) as difference of two relative entropies. The former compares $P_{Y|X=x_1}$ with $P_{Y|X=x_2}^{(h,Q,\lambda)}$, where $(x_1, x_2) \in \mathcal{X} \times \mathcal{X}$ are independently sampled from the same probability measure $P_X$. The latter compares these two conditional measures conditioning on the same element of $\mathcal{X}$. That is, it compares $P_{Y|X=x_2}$ with $P_{Y|X=x_2}^{(h,Q,\lambda)}$.

An interesting observation from Theorems 11 and 12 is that the last two terms in the right-hand side of (60) are both zero in the case in which $P_{Y|X}$ is an $(h, Q, \lambda)$-Gibbs conditional probability measure. Similarly, in such a case, the second term in the right-hand side of (61) is also zero. This observation is highlighted by the following corollary.

**Corollary 13.** Consider an $(h, Q, \lambda)$-Gibbs conditional probability measure, denoted by $P_{Y|X}^{(h,Q,\lambda)} \in \Delta(\mathbb{R}^m|\mathbb{R}^n)$, with $\lambda \neq 0$; and a probability measure $P_X \in \Delta(\mathbb{R}^n)$. Let the measure $P_{Y|X}^{(h,Q,\lambda)} \in \Delta(\mathbb{R}^m)$ be such that for all sets $\mathcal{A} \in \mathcal{B}(\mathbb{R}^m)$,

$$ P_{Y}^{(h,Q,\lambda)}(\mathcal{A}) = \int P_{Y|X=x}^{(h,Q,\lambda)}(\mathcal{A}) dP_X(x). \quad (62) $$
Then,

$$ \bar{G}_h \left( P_Y^{(h,Q,\lambda)}, P_{Y|X}^{(h,Q,\lambda)}, P_X \right) = \frac{1}{\lambda} \left( I \left( P_{Y|X}^{(h,Q,\lambda)}; P_X \right) + L \left( P_{Y|X}^{(h,Q,\lambda)}; P_X \right) \right) \quad (63) $$

$$ = \frac{1}{\lambda} \iint D \left( P_{Y|X=x_2}^{(h,Q,\lambda)} \| P_{Y|X=x_1}^{(h,Q,\lambda)} \right) dP_X(x_1)dP_X(x_2). \quad (64) $$

Please note that mutual information and lautum information are both nonnegative information measures, which from Corollary 13, implies that $\bar{G}_h(P_Y^{(h,Q,\lambda)}, P_{Y|X}^{(h,Q,\lambda)}, P_X)$ in (64) might be either positive or negative depending exclusively on the sign of the regularization factor $\lambda$. The following corollary exploits such an observation to present a property of Gibbs conditional probability measures and their corresponding marginal probability measures.

**Corollary 14.** Given a probability measure $P_X \in \Delta(\mathbb{R}^n)$, the $(h, Q, \lambda)$-Gibbs conditional probability measure $P_{Y|X}^{(h,Q,\lambda)}$ in (14) and the probability measure $P_Y^{(h,Q,\lambda)}$ in (62) satisfy

$$ \iint h(x, y)dP_Y^{(h,Q,\lambda)}(y)dP_X(x) \geq \iint h(x, y)dP_{Y|X=x}^{(h,Q,\lambda)}(y)dP_X(x) \text{ if } \lambda > 0; \quad (65) $$

or

$$ \iint h(x, y)dP_Y^{(h,Q,\lambda)}(y)dP_X(x) \leq \iint h(x, y)dP_{Y|X=x}^{(h,Q,\lambda)}(y)dP_X(x) \text{ if } \lambda < 0. \quad (66) $$

Corollary 14 highlights the fact that a deviation from the joint probability measure $P_{Y|X}^{(h,Q,\lambda)}$ $P_X \in \Delta(\mathcal{Y} \times \mathcal{X})$ to the product of its marginals $P_Y^{(h,Q,\lambda)}$ $P_X \in \Delta(\mathcal{Y} \times \mathcal{X})$ might increase or decrease the expectation of the function $h$ depending on the sign of $\lambda$.

# 6 Examples

An immediate application of the results presented above is the analysis of the generalization error of machine-learning algorithms [11], which was the scenario in which these results were originally discovered. In the remainder of this section, such results are presented as consequences of the more general results presented in this work.

Let $\mathcal{M}$, $\mathcal{X}$ and $\mathcal{Y}$, with $\mathcal{M} \subseteq \mathbb{R}^d$, be sets of models, patterns, and labels, respectively. A pair $(x, y) \in \mathcal{X} \times \mathcal{Y}$ is referred to as a data point. A dataset $z \in (\mathcal{X} \times \mathcal{Y})^n$ is a tuple of $n$ data points of the form:

$$ z = ((x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathcal{Y})^n. \quad (67) $$

Consider the function

$$ L: \begin{cases} (\mathcal{X} \times \mathcal{Y})^n \times \mathcal{M} \rightarrow [0, +\infty] \\ (z, \theta) \mapsto \frac{1}{n} \sum_{i=1}^{n} \ell(x_i, y_i, \theta), \end{cases} \quad (68) $$
where $\ell(x_i, y_i, \theta)$ is the risk or loss induced by the model $\theta$ with respect to the data point $(x_i, y_i)$. Given a fixed dataset $z$ of the form in (67), consider also the functional

$$ R_z: \begin{cases} \Delta(\mathcal{M}) \to \mathbb{R} \\ P \mapsto \int \mathbb{L}(z, \theta) dP(\theta), \end{cases} \qquad (69) $$

where the function $\mathbb{L}$ is defined in (68). Using this notation, the empirical risk induced by the model $\theta$ with respect to the dataset $z$ is $\mathbb{L}(z, \theta)$. The expectation of the empirical risk with respect to a fixed dataset $z$ when models are sampled from a probability measure $P \in \Delta(\mathcal{M})$ is $R_z(P)$.

A machine-learning algorithm is represented by a conditional probability measure $P_{\theta|Z} \in \Delta(\mathcal{M}|(\mathcal{X} \times \mathcal{Y})^n)$. The instance of such an algorithm generated by training it upon the dataset $z$ in (67) is represented by the probability measure $P_{\theta|Z=z} \in \Delta(\mathcal{M})$. The generalization error induced by the algorithm $P_{\theta|Z}$ is defined as follows.

**Definition 4 (Generalization Error).** The generalization error induced by the algorithm $P_{\theta|Z} \in \Delta(\mathcal{M}|(\mathcal{X} \times \mathcal{Y})^n)$, under the assumption that training and test datasets are independently sampled from a probability measure $P_Z \in \Delta((\mathcal{X} \times \mathcal{Y})^n)$, is denoted by $\overline{\overline{\mathcal{G}}}(P_{\theta|Z}, P_Z)$, and

$$ \overline{\overline{\mathcal{G}}}(P_{\theta|Z}, P_Z) \triangleq \iint (R_u(P_{\theta|Z=z}) - R_z(P_{\theta|Z=z})) dP_Z(u) dP_Z(z), \quad (70) $$

where the functionals $R_u$ and $R_z$ are defined in (69).

Often, the term $R_u(P_{\theta|Z=z})$ is recognized to be the test error induced by the algorithm $P_{\theta|Z}$ with respect to a test dataset $u \in (\mathcal{X} \times \mathcal{Y})^n$ when it is trained upon the dataset $z \in (\mathcal{X} \times \mathcal{Y})^n$. Alternatively, the term $R_z(P_{\theta|Z=z})$ is recognized to be the training error induced by the algorithm $P_{\theta|Z}$ when it is trained upon the dataset $z \in (\mathcal{X} \times \mathcal{Y})^n$. From this perspective, the generalization error $\overline{\overline{\mathcal{G}}}(P_{\theta|Z}, P_Z)$ in (70) is the expectation of the difference between test error and training error when the test and training datasets are independently sampled from the same probability measure $P_Z$. The key observation is that such generalization error $\overline{\overline{\mathcal{G}}}(P_{\theta|Z}, P_Z)$ can be written as a variation of an expectation of the empirical risk function $\mathbb{L}$ in (68), as shown hereunder.

**Lemma 15 (Lemma 3 in [11]).** Consider the generalization error $\overline{\overline{\mathcal{G}}}(P_{\theta|Z}, P_Z)$ in (70) and assume that for all $z$, the probability measure $P_{\theta|Z=z}$ is absolutely continuous with respect to the probability measure $P_{\theta} \in \Delta(\mathcal{M})$, which satisfies for all measurable subsets $C$ of $\mathcal{M}$,

$$ P_{\theta}(C) = \int P_{\theta|Z=z}(C) dP_Z(z). \qquad (71) $$

Then,

$$ \overline{\overline{\mathcal{G}}}(P_{\theta|Z}, P_Z) = \overline{\mathcal{G}}_{\mathbb{L}}(P_{\theta}, P_{\theta|Z}, P_Z), \qquad (72) $$
where the functional $\bar{G}_L$ and the function $L$ are defined in (3) and (68), respectively.

From Theorems 11 and 15, the following holds.

**Theorem 16** (Theorem 14 in [11]). Consider the generalization error $\bar{\bar{G}}(P_{\Theta|Z}, P_Z)$ in (70) and assume that for all $z \in (X \times \mathcal{Y})^n$:

(a) The probability measures $P_{\Theta}$ in (71) and $P_{\Theta|Z=z}$ are both absolutely continuous with respect to some $\sigma$-finite measure $Q \in \Delta(\mathcal{M})$;

(b) The measure $Q$ is absolutely continuous with respect to $P_{\Theta}$; and

(c) The measure $P_{\Theta}$ is absolutely continuous with respect to $P_{\Theta|Z=z}$.

Then,

$$ \bar{\bar{G}}(P_{\Theta|Z}, P_Z) = \lambda \left( I(P_{\Theta|Z}; P_Z) + L(P_{\Theta|Z}; P_Z) \right) \\ +\lambda \iint \log \frac{dP_{\Theta|Z=z}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dP_{\Theta}(\theta) dP_Z(z) \\ -\lambda \iint \log \frac{dP_{\Theta|Z=z}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dP_{\Theta|Z=z}(\theta) dP_Z(z), \quad (73) $$

where the measure $P_{\Theta|Z}^{(Q,\lambda)}$ is an $(L, Q, \lambda)$-Gibbs conditional probability measure, with the function $L$ defined in (68).

Theorem 16 shows one of many closed-form expressions that can be obtained for the generalization error $\bar{\bar{G}}(P_{\Theta|Z}, P_Z)$ in (70) in terms of information measures. A complete exposition of several equivalent alternative expressions, as well as a discussion on their relevance, is presented in [11].

The important observation in this example is that the measure $P_{\Theta|Z}^{(Q,\lambda)}$ in (73) is an $(L, Q, \lambda)$-Gibbs conditional probability measure, which represents the celebrated *Gibbs algorithm* in statistical machine learning [41]. Thus, the term $\log \frac{dP_{\Theta|Z=z}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta)$ in (73) can be interpreted as a log-likelihood ratio in a hypothesis test in which the objective is to distinguish the probability measures $P_{\Theta|Z=z}$ and $P_{\Theta|Z=z}^{(Q,\lambda)}$ based on the observation of the model $\theta$. The former represents the algorithm under study trained upon $z$, whereas the latter represents a Gibbs algorithm trained upon the same dataset $z$.

From this perspective, the difference between the last two terms in (73), i.e.,

$$ \iint \log \frac{dP_{\Theta|Z=z}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dP_{\Theta}(\theta) dP_Z(z) - \iint \log \frac{dP_{\Theta|Z=z}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dP_{\Theta|Z=z}(\theta) dP_Z(z), $$

can be interpreted as the variation of the expectation of the log-likelihood ratio

$$ \log \frac{dP_{\Theta|Z=z}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) \quad (74) $$
when the probability measure from which the model $\theta$ and dataset $z$ are drawn
changes from the ground-truth distribution $P_{\theta|z}P_z$ to the product of the cor-
responding marginals $P_{\theta}P_z$. As originally suggested in [11], Theorem 16 estab-
lishes an interesting connection between hypothesis testing, information mea-
sures, and generalization error. Nonetheless, this connection goes beyond this
application in statistical machine learning as the same connection can be es-
tablished directly from Theorem 11. This establishes a connection between the
variation of the expectation due to changes in the probability measure, infor-
mation measures, and hypothesis testing.

# 7 Final Remarks

A simple reformulation of Varadhan's variational representation of relative en-
tropy (Lemma 2) yields an explicit expression for the variation of the expecta-
tion of a real function when the probability measure shifts from a Gibbs measure
to an arbitrary probability measure (Lemma 3). This result connects directly
with information-projection methods, Pythagorean identities for relative en-
tropy, and optimization problems with a constraint on the relative entropy of
the measure to be optimized with respect to a reference measure (Lemma 4).
A simple algebraic manipulation of Lemma 3 provides a general formula. It de-
scribes how the expectation changes with respect to changes in the probability
measure (Theorem 5). The result is general, and the only assumption is that
both initial (before the variation) and final (after the variation) measures are
absolutely continuous with respect to a common reference. A key insight is the
central role of Gibbs measures in this framework. The change in expectation
is described through relative entropy comparisons between the initial and final
measures, each with respect to a specific Gibbs measure built from the func-
tion under study. Notably, the reference measure of these Gibbs distributions
need not be a probability measure. It can be a $\sigma$-finite measure, such as the
Lebesgue measure or the counting measure. In such cases, the resulting expres-
sions include Shannon's fundamental information measures, including entropy
and conditional entropy (Corollary 7). Building on these results, the variation of
expectations under changes in joint probability measures is studied. Two cases
are of special interest. In the former, one marginal remains unchanged (Theorem
8). In the latter, the joint measure changes to the product of its marginals (The-
orems 11 and 12). In the case of Gibbs joint measures, the resulting expressions
involve only standard information quantities: mutual information, lautum infor-
mation, and relative entropy. These results show a broad connection between
the variation in the expectation of measurable functions, induced by changes
in probability measure, and information measures such as mutual and lautum
information.
# Appendices

## A Proof of Theorem 11

The proof follows from Theorem 8, which holds under assumption (a) and leads to

$$ \begin{aligned} & \bar{G}_h(P_Y, P_{Y|X}, P_X) \\ & = \frac{1}{\lambda} \int \left( D(P_Y \| P_{Y|X=x}^{(h,Q,\lambda)}) - D(P_{Y|X=x} \| P_{Y|X=x}^{(h,Q,\lambda)}) + D(P_{Y|X=x} \| Q) - D(P_Y \| Q) \right) dP_X(x). \end{aligned} \quad (75) $$

The proof continues by noticing that

$$ \int D(P_{Y|X=x} \| Q) dP_X(x) = \iint \log \left( \frac{dP_{Y|X=x}}{dQ}(y) \right) dP_{Y|X=x}(y) dP_X(x) \quad (76) $$

$$ = \iint \log \left( \frac{dP_{Y|X=x}}{dP_Y}(y) \frac{dP_Y}{dQ}(y) \right) dP_{Y|X=x}(y) dP_X(x) \quad (77) $$

$$ = \iint \log \left( \frac{dP_{Y|X=x}}{dP_Y}(y) \right) dP_{Y|X=x}(y) dP_X(x) $$

$$ + \iint \log \left( \frac{dP_Y}{dQ}(y) \right) dP_{Y|X=x}(y) dP_X(x) \quad (78) $$

$$ = \iint \log \left( \frac{dP_{Y|X=x}}{dP_Y}(y) \right) dP_{Y|X=x}(y) dP_X(x) $$

$$ + \iint \frac{dP_{Y|X=x}}{dP_Y}(y) \log \left( \frac{dP_Y}{dQ}(y) \right) dP_Y(y) dP_X(x) \quad (79) $$

$$ = \iint \log \left( \frac{dP_{Y|X=x}}{dP_Y}(y) \right) dP_{Y|X=x}(y) dP_X(x) $$

$$ + \iint \log \left( \frac{dP_Y}{dQ}(y) \right) \left( \iint \frac{dP_{Y|X=x}}{dP_Y}(y) dP_X(x) \right) dP_Y(y) \quad (80) $$

$$ = \iint \log \left( \frac{dP_{Y|X=x}}{dP_Y}(y) \right) dP_{Y|X=x}(y) dP_X(x) $$

$$ + \iint \log \left( \frac{dP_Y}{dQ}(y) \right) dP_Y(y) \quad (81) $$

$$ = I(P_{Y|X}; P_X) + D(P_Y \| Q), \quad (82) $$

where (77) follows from Theorem 4 in [38]; (79) follows from Theorem 2 in [38]; and (81) follows from Theorem 10 in [38], which implies that for all $y \in \mathbb{R}^m$, $\int \frac{dP_{Y|X=x}}{dP_Y}(y) dP_X(x) = 1$.
Note also that

$$ \int D(P_Y \| P_{Y|X=x}^{(h,Q,\lambda)}) dP_X(x) = \int \int \log \left( \frac{dP_Y}{dP_{Y|X=x}^{(h,Q,\lambda)}} \right) dP_Y(y) dP_X(x) \quad (83) $$

$$ = \int \int \log \left( \frac{dP_Y}{dP_{Y|X=x}}(y) \frac{dP_{Y|X=x}}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP_Y(y) dP_X(x) \quad (84) $$

$$ = \int \int \log \left( \frac{dP_Y}{dP_{Y|X=x}}(y) \right) dP_Y(y) dP_X(x) $$

$$ + \int \int \log \left( \frac{dP_{Y|X=x}}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP_Y(y) dP_X(x) \quad (85) $$

$$ = L(P_{Y|X=x}; P_X) + \iint \log \left( \frac{dP_{Y|X=x}}{dP_{Y|X=x}^{(h,Q,\lambda)}}(y) \right) dP_Y(y) dP_X(x), (86) $$

where (84) follows from Theorem 4 in [38]. Finally, using (82) and (86) in (75) yields (60), which completes the proof.

## B Proof of Theorem 12

The proof follows by observing that the functional $\bar{G}_h$ in (10) satisfies

$$ \bar{G}_h (P_Y, P_{Y|X}, P_X) = \iint \left( \int h(x_2, y) dP_{Y|X=x_1}(y) - \int h(x_1, y) dP_{Y|X=x_1}(y) \right) dP_X(x_2) dP_X(x_1) (87) $$

Using the functional $G_h$ in (1), the terms above can be written as follows

$$ \int h(x_1, y) dP_{Y|X=x_1}(y) = G_h(x_1, P_{Y|X=x_1}, P_{Y|X=x_1}^{(h,Q,\lambda)}) + \int h(x_1, y) dP_{Y|X=x_1}^{(h,Q,\lambda)}(y), (88) $$

and

$$ \int h(x_2, y) dP_{Y|X=x_1}(y) = G_h(x_2, P_{Y|X=x_1}, P_{Y|X=x_2}^{(h,Q,\lambda)}) + \int h(x_2, y) dP_{Y|X=x_2}^{(h,Q,\lambda)}(y). (89) $$
Using (88) and (89) in (87) yields

$$ \bar{G}_h (P_Y, P_{Y|X}, P_X) = \iint \left( G_h (x_2, P_{Y|X=x_1}, P_{Y|X=x_2}^{(h,Q,\lambda)}) + \int h(x_2, y) dP_{Y|X=x_2}^{(h,Q,\lambda)}(y) \right) \quad (90) $$

$$ -G_h(x_1, P_{Y|X=x_1}, P_{Y|X=x_1}^{(h,Q,\lambda)}) - \int h(x_1, y) dP_{Y|X=x_1}^{(h,Q,\lambda)}(y) dP_X(x_2) dP_X(x_1), $$

$$ = \iint \left( G_h(x_2, P_{Y|X=x_1}, P_{Y|X=x_2}^{(h,Q,\lambda)}) - \int G_h(x_1, P_{Y|X=x_1}, P_{Y|X=x_1}^{(h,Q,\lambda)}) dP_X(x_2) dP_X(x_1) \right) \quad (91) $$

$$ = \frac{1}{\lambda} \iint \left( D(P_{Y|X=x_2} \| P_{Y|X=x_1}^{(h,Q,\lambda)}) - D(P_{Y|X=x_2} \| P_{Y|X=x_2}^{(h,Q,\lambda)}) \right) dP_X(x_1) dP_X(x_2), \quad (92) $$

where the last equality holds from Lemma 3, which implies

$$ \begin{aligned} & \iint G_h (x_2, P_{Y|X=x_1}, P_{Y|X=x_2}^{(h,Q,\lambda)}) dP_X(x_2) dP_X(x_1) \\ = & \frac{1}{\lambda} \iint D(P_{Y|X=x_1} \| P_{Y|X=x_2}^{(h,Q,\lambda)}) \frac{1}{\lambda} dP_X(x_2) dP_X(x_1) + \frac{1}{\lambda} \int D(P_{Y|X=x_2}^{(h,Q,\lambda)} \| Q) dP_X(x_2) \\ & - \frac{1}{\lambda} \int D(P_{Y|X=x_1} \| Q) dP_X(x_1), \end{aligned} \quad (93) $$

and

$$ \begin{aligned} & \iint G_h (x_1, P_{Y|X=x_1}, P_{Y|X=x_1}^{(h,Q,\lambda)}) dP_X(x_2) dP_X(x_1) \\ = & \int G_h (x_1, P_{Y|X=x_1}, P_{Y|X=x_1}^{(h,Q,\lambda)}) dP_X(x_1) \end{aligned} \quad (94) $$

$$ = \frac{1}{\lambda} \int D(P_{Y|X=x_1} \| P_{Y|X=x_1}^{(h,Q,\lambda)}) dP_X(x_1) + \frac{1}{\lambda} \int D(P_{Y|X=x_1}^{(h,Q,\lambda)} \| Q) dP_X(x_1) $$

$$ -\frac{1}{\lambda} \int D(P_{Y|X=x_1} \| Q) dP_X(x_1), \quad (95) $$

which completes the proof.
References

[1] S. M. Perlaza and G. Bisson, "Variations on the expectation due to changes in the probability measure," *Entropy*, vol. 27, no. 8:865, Aug. 2025.

[2] J. Gama, P. Medas, G. Castillo, and P. Rodrigues, "Learning with drift detection," in *Proceedings of the 17th Brazilian Symposium on Artificial Intelligence*, Sao Luis, Maranhao, Brazil, Oct. 2004, pp. 286-295.

[3] G. I. Webb, L. K. Lee, B. Goethals, and F. Petitjean, "Analyzing concept drift and shift from sample data," *Data Mining and Knowledge Discovery*, vol. 32, pp. 1179-1199, 2018.

[4] G. H. F. M. Oliveira, L. L. Minku, and A. L. Oliveira, "Tackling virtual and real concept drifts: An adaptive Gaussian mixture model approach," *IEEE Transactions on Knowledge and Data Engineering*, vol. 35, no. 2, pp. 2048-2060, 2021.

[5] C. E. Shannon, "A mathematical theory of communication," *The Bell System Technical Journal*, vol. 27, pp. 379-423, Jul. 1948.

[6] D. P. Palomar and S. Verdú, "Lautum information," *IEEE Transactions on Information Theory*, vol. 54, no. 3, pp. 964-975, Mar. 2008.

[7] S. M. Perlaza, I. naki Esnaola, G. Bisson, and H. V. Poor, "On the validation of Gibbs algorithms: Training datasets, test datasets and their aggregation," in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Taipei, Taiwan, Jun. 2023.

[8] S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, "Empirical risk minimization with relative entropy regularization," *IEEE Transactions on Information Theory*, vol. 70, no. 7, pp. 5122-5161, Jul. 2024.

[9] X. Zou, S. M. Perlaza, I. Esnaola, and E. Altman, "Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measure," in *Proceedings of the AAAI Conference on Artificial Intelligence*, Vancouver, Canada, Feb. 2024.

[10] X. Zou, S. M. Perlaza, I. Esnaola, E. Altman, and H. V. Poor, "The worst-case data-generating probability measure in statistical learning," *IEEE Journal on Selected Areas in Information Theory*, vol. 5, p. 175-189, Apr. 2024.

[11] S. M. Perlaza and X. Zou, "The generalization error of machine learning algorithms," *arXiv preprint arXiv:2411.12030*, 2024.

[12] N. N. Chentsov, "Nonsymmetrical distance between probability distributions, entropy and the theorem of pythagoras," *Mathematical notes of the Academy of Sciences of the USSR*, vol. 4, no. 3, pp. 686-691, 1968.
[13] I. Csiszár and F. Matus, "Information projections revisited," *IEEE Transactions on Information Theory*, vol. 49, no. 6, pp. 1474-1490, 2003.

[14] A. Müller, "Integral probability metrics and their generating classes of functions," *Advances in applied probability*, vol. 29, no. 2, pp. 429-443, 1997.

[15] V. M. Zolotarev, "Probability metrics," *Teoriya Veroyatnostei i ee Prime-neniya*, vol. 28, no. 2, pp. 264-287, 1983.

[16] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, "A kernel two-sample test," *Journal of Machine Learning Research*, vol. 13, no. 25, pp. 723-773, 2012.

[17] C. Villani, *Optimal transport: Old and new*, 1st ed. Berlin, Germany: Springer, 2009.

[18] W. Liu, G. Yu, L. Wang, and R. Liao, "An information-theoretic framework for out-of-distribution generalization with applications to stochastic gradient Langevin dynamics," *arXiv preprint arXiv:2403.19895*, 2024.

[19] —, "An information-theoretic framework for out-of-distribution generalization," in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Athens, Greece, July 2024, pp. 2670-2675.

[20] R. Agrawal and T. Horel, "Optimal bounds between f-divergences and integral probability metrics," *Journal of Machine Learning Research*, vol. 22, no. 128, pp. 1-59, 2021.

[21] H. Rahimian and S. Mehrotra, " framworks and results in distributionally robust optimization," *Open Journal of Mathematical Optimization*, vol. 3, pp. 1-85, 2022.

[22] C. Xu, J. Lee, X. Cheng, and Y. Xie, "Flow-based distributionally robust optimization," *IEEE Journal on Selected Areas in Information Theory*, vol. 5, pp. 62 – 77, Apr. 2024.

[23] Z. Hu and L. J. Hong, "Kullback-Leibler divergence constrained distributionally robust optimization," *Optimization Online*, vol. 1, no. 2, p. 9, 2013.

[24] J. Radon, *Theorie und Anwendungen der absolut additiven Mengenfunktionen*, 1st ed. Vienna, Austria: Hölder, 1913.

[25] O. Nikodym, "Sur une généralisation des intégrales de mj radon," *Fundamenta Mathematicae*, vol. 15, no. 1, pp. 131-179, 1930.

[26] G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell, "An exact characterization of the generalization error for the Gibbs algorithm," *Advances in Neural Information Processing Systems*, vol. 34, pp. 8106-8118, Dec. 2021.
[27] S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, “Empirical risk minimization with relative entropy regularization: Optimality and sensitivity,” in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Espoo, Finland, Jul. 2022, pp. 684–689.

[28] W. Jiang and M. A. Tanner, “Gibbs posterior for variable selection in high-dimensional classification and data mining,” *The Annals of Statistics*, vol. 36, no. 5, pp. 2207–2231, Oct. 2008.

[29] P. Alquier, J. Ridgway, and N. Chopin, “On the properties of variational approximations of Gibbs posteriors,” *Journal of Machine Learning Research*, vol. 17, no. 1, pp. 8374–8414, Dec. 2016.

[30] Y. Bu, G. Aminian, L. Toni, G. W. Wornell, and M. Rodrigues, “Characterizing and understanding the generalization error of transfer learning with Gibbs algorithm,” in *Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS)*, Virtual Conference, Mar. 2022, pp. 8673–8699.

[31] M. Raginsky, A. Rakhlin, M. Tsao, Y. Wu, and A. Xu, “Information-theoretic analysis of stability and bias of learning algorithms,” in *Proceedings of the IEEE Information Theory Workshop (ITW)*, Cambridge, UK, Sep. 2016, pp. 26–30.

[32] B. Zou, L. Li, and Z. Xu, “The generalization performance of ERM algorithm with strongly mixing observations,” *Machine Learning*, vol. 75, no. 3, pp. 275–295, Feb. 2009.

[33] H. He, G. Aminian, Y. Bu, M. Rodrigues, and V. Y. Tan, “How does pseudo-labeling affect the generalization error of the semi-supervised Gibbs algorithm?” in *Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS)*, Valencia, Spain, Apr. 2023, pp. 8494–8520.

[34] F. Hellström, G. Durisi, B. Guedj, and M. Raginsky, “Generalization bounds: Perspectives from information theory and PAC-Bayes,” *Foundations and Trends® in Machine Learning*, vol. 18, no. 1, pp. 1–223, Jan. 2025.

[35] E. T. Jaynes, “Information theory and statistical mechanics I,” *Physical Review Journals*, vol. 106, no. 4, pp. 620–630, May 1957.

[36] ——, “Information theory and statistical mechanics II,” *Physical Review Journals*, vol. 108, no. 2, pp. 171–190, Oct. 1957.

[37] J. N. Kapur, *Maximum Entropy Models in Science and Engineering*, 1st ed. New York, NY, USA: Wiley, 1989.
[38] Y. Bermudez, G. Bisson, I. Esnaola, and S. M. Perlaza, “Proofs for folk-lore theorems on the Radon-Nikodym derivative,” INRIA, Centre Inria d’Université Côte d’Azur, Sophia Antipolis, France, Tech. Rep. RR-9591, July 2025.

[39] M. D. Donsker and S. S. Varadhan, “Asymptotic evaluation of certain Markov process expectations for large time, I,” *Communications on pure and applied mathematics*, vol. 28, no. 1, pp. 1–47, 1975.

[40] T. L. Heath, *The Thirteen Books of Euclid's Elements*, 2nd ed. New York: Dover Publications, Inc., 1956.

[41] W. Azizian, F. Lutzeler, J. Malick, and P. Mertikopoulos, “What is the long-run distribution of stochastic gradient descent? A large deviations analysis,” in *Proceedings of the 41st International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, vol. 235, Jul. 2024, pp. 2168–2229.
informatics mathematics
Inria

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex

Publisher
Inria
Domaine de Voluceau - Rocquencourt
BP 105 - 78153 Le Chesnay Cedex
inria.fr

ISSN 0249-6399