# Machine Unlearning for Gibbs Supervised Learning Algorithms

Yaiza Bermudez, Samir M. Perlaza, Iñaki Esnaola

<figure><img src="image_1.png" /></figure>

## To cite this version:

Yaiza Bermudez, Samir M. Perlaza, Iñaki Esnaola. Machine Unlearning for Gibbs Supervised Learning Algorithms. RR-9610, Inria. 2026. hal-05471666

HAL Id: hal-05471666

https://inria.hal.science/hal-05471666v1

Submitted on 22 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
informatics mathematics
Inria

# Machine Unlearning for Gibbs Supervised Learning Algorithms

Yaiza Bermudez, Samir M. Perlaza, and Iñaki Esnaola

## RESEARCH REPORT

### N° 9610

January 2026

Project-Team NEO

ISSN 0249-6399 ISRN INRIA/RR-9610-FR+ENG
<figure><img src="image_3.png" /></figure>
# Machine Unlearning for Gibbs Supervised
Learning Algorithms

Yaiza Bermudez, Samir M. Perlaza, and Iñaki Esnaola

Project-Team NEO

Research Report n° 9610 — January 2026 — 24 pages

**Abstract:** In this report, a method for achieving exact unlearning for Gibbs supervised learning algorithms is proposed using a variational formulation inspired by empirical risk minimization subject to relative entropy regularization (ERM-RER). Such a method consists of maximizing the expected empirical risk over the dataset to be unlearned subject to a regularization by relative entropy with respect to the original algorithm. The optimization variable is a probability measure on the models; and the solution is another Gibbs probability measure that represents a new Gibbs supervised learning algorithm. The method guarantees exact unlearning in the sense that the new Gibbs algorithm coincides in distribution with the algorithm that would have been obtained by retraining from scratch on the dataset to be retained. As a byproduct, a framework for reweighting data points in ERM-RER by strategically choosing both the reference measure and the regularization factor is obtained. In this framework, exact unlearning is the special case in which zero-weight is assigned to the contribution of the data points to be unlearned. More generally, depending on the choice of certain parameters, data points can be up-weighted or down-weighted in ERM-RER problems for particular purposes, e.g., controlling the generalization error of Gibbs algorithms. This paves the way to new constructive or adversarial views on classical reweighting data points in ERM-RER.

**Key-words:** Machine Unlearning, Empirical Risk Minimization, Relative Entropy Regularization, Gibbs Probability Measures, Gibbs Algorithms

Yaiza Bermudez and Samir M. Perlaza are with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis 06902, France. Samir M. Perlaza is also with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia. Iñaki Esnaola is with the School of Electrical and Electronic Engineering at The University of Sheffield, Sheffield S1 3JD, UK. Samir M. Perlaza and Iñaki Esnaola also are with the Electrical and Computer Engineering Department at Princeton University, Princeton N.J. 08544, USA.

This research was supported in part by the European Commission through the H2020MSCA-RISE-2019 project 872172; the French National Agency for Research (ANR) through the Project ANR-21-CE25-0013 and the project ANR-22-PEFT-0010 of the France 2030 program PEPR Réseaux du Futur; and in part by the Agence de l'innovation de défense (AID) through the project UK-FR 2024352.

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex
# Désapprentissage pour les algorithmes d'apprentissage supervisé de Gibbs.

**Résumé** : Dans ce rapport, une méthode permettant d'obtenir un désapprentissage (unlearning) exact pour les algorithmes d'apprentissage supervisé de Gibbs est proposée, au moyen d'une formulation variationnelle inspirée de la minimisation du risque empirique sous régularisation par entropie relative (ERM-RER). Une telle méthode consiste à maximiser le risque empirique espéré sur l'ensemble de données à désapprendre, sous une régularisation par entropie relative par rapport à l'algorithme original. La variable d'optimisation est une mesure de probabilité sur l'espace des modèles ; et la solution est une autre mesure de probabilité de Gibbs qui représente un nouvel algorithme d'apprentissage supervisé de Gibbs. La méthode garantit un désapprentissage exact au sens où le nouvel algorithme de Gibbs coïncide en distribution avec l'algorithme qui aurait été obtenu en réentraînant depuis zéro sur l'ensemble de données à conserver. Comme résultat annexe, on obtient un cadre permettant de repondérer les points de données dans ERM-RER en choisissant stratégiquement à la fois la mesure de référence et le facteur de régularisation. Dans ce cadre, le désapprentissage exact correspond au cas particulier où l'on attribue un poids nul à la contribution des points de données à désapprendre. Plus généralement, selon le choix de certains paramètres, les points de données peuvent être surpondérés ou sous-pondérés dans les problèmes ERM-RER à des fins particulières, par exemple pour augmenter ou diminuer l'erreur de généralisation des algorithmes de Gibbs. Cela ouvre la voie à de nouvelles perspectives, constructives ou adversariales, sur la repondération classique des points de données dans ERM-RER.

**Mots-clés** : Désapprentissage, Minimisation du Risque Empirique, Régularisation par Entropie Relative, Mesures de Probabilité de Gibbs, Algorithmes de Gibbs.
# Contents

<table><tr><td>1</td><td>Introduction</td><td>4</td></tr><tr><td>1.1</td><td>Contributions</td><td>4</td></tr><tr><td>1.2</td><td>Notation and Preliminaries</td><td>5</td></tr><tr><td>2</td><td>Supervised Machine Learning</td><td>6</td></tr><tr><td>3</td><td>Gibbs Algorithms</td><td>7</td></tr><tr><td>4</td><td>Main Result</td><td>9</td></tr><tr><td>4.1</td><td>Reweighting Data Points in ERM-RER</td><td>10</td></tr><tr><td>4.2</td><td>Exact Unlearning</td><td>12</td></tr><tr><td>5</td><td>Extension to Decentralized Learning</td><td>14</td></tr><tr><td>6</td><td>Conclusions and Final Remarks</td><td>15</td></tr><tr><td></td><td>References</td><td>16</td></tr><tr><td></td><td>Appendices</td><td>19</td></tr><tr><td>A</td><td>Preliminaries for the Proof of Theorem 5</td><td>19</td></tr><tr><td>B</td><td>Proof of Theorem 5</td><td>20</td></tr><tr><td>C</td><td>Proof of Theorem 6</td><td>21</td></tr><tr><td>D</td><td>Complementary Results</td><td>22</td></tr></table>
# 1 Introduction

Modern learning systems are routinely trained on data that may later become unavailable or impermissible to use, e.g., due to user requests, contractual constraints, or regulatory requirements such as the right to erasure in Article 17 of the GDPR [1], also known as the “Right to be Forgotten”. In this context, *machine unlearning* refers to the task of removing the influence of a part of the training dataset from a learning algorithm while avoiding the computational cost of retraining from scratch [2–4]. Early formulations of data deletion in learning formalize this task as producing an updated algorithm that is statistically consistent with training on the retained data only [5, 6]. A particularly stringent requirement is *exact unlearning*, in which the distribution of the post-unlearning algorithm is required to be identical to the algorithm obtained by retraining on the dataset to be retained [6]. These guarantees are difficult to obtain and practical approaches often trade computational efficiency for approximate removal [7–9], often referred to as *approximate unlearning* [10].

In this work, exact unlearning is formulated as a relative-entropy-regularized variational problem, which allows using existing tools previously developed in the framework of empirical risk minimization with $f$-divergence regularization [11–15].

## 1.1 Contributions

The main contribution of this work is an exact unlearning method for Gibbs algorithms. More specifically, the method consists in implementing a Gibbs algorithm via the maximization of the empirical risk with respect to the dataset to be unlearned using as reference measure the Gibbs algorithm from which data must be unlearned. Interestingly, this method guarantees exact unlearning in the sense that the post-unlearning algorithm coincides with the solution to an ERM-RER problem with respect to the retained training dataset. The key observation is that exact unlearning is achieved thanks to the strategic choice of the reference measure and the regularization factor, also known as temperature coefficient [16]. This choice results in an effect that can be assimilated to a reweighting of the contribution of the data points to be unlearned. In particular, unlearning consists in assigning zero-weight to those data points. From this perspective, the results presented here can be placed in a more general scenario than unlearning. In particular, it can be placed in the scenario of reweighting data points in ERM-RER. More generally, this reweighting might be oriented to decrease the generalization error [17] of Gibbs algorithms by controlling the contribution of particular data points. On the other hand, reweighting data points in ERM-RER might be performed adversarially. In this case, the objective might be to increase the generalization error of Gibbs algorithms by up-weighting the outliers within the training dataset.

The report is organized as follows. Section 2 formalizes the supervised learning setting. Section 3 defines Gibbs conditional probability measures and their characterizations via ERM-RER optimization problems. Section 4 presents
the main results: a reweighting interpretation through a general ERM-RER reformulation, and an exact unlearning method for Gibbs algorithms obtained by maximizing the empirical risk. Finally, Section 6 concludes the report and outlines open research directions. The proofs are presented in the appendices.

## 1.2 Notation and Preliminaries

The set of all probability measures on the measurable space $(\mathcal{X}, \mathcal{F}_{\mathcal{X}})$ is denoted by $\Delta(\mathcal{X}, \mathcal{F}_{\mathcal{X}})$, or simply $\Delta(\mathcal{X})$. Moreover, the set of probability measures in $\Delta(\mathcal{X})$ that are absolutely continuous with respect to $Q$ is denoted by $\Delta_Q(\mathcal{X})$. Using this notation, a conditional probability measure is defined hereunder.

**Definition 1** (Conditional Probability). A family $P_{Y|X} \triangleq (P_{Y|X=x})_{x \in \mathcal{X}}$ of elements of $\Delta(\mathcal{Y}, \mathcal{F}_{\mathcal{Y}})$ indexed by $\mathcal{X}$ is said to be a conditional probability measure if, for all sets $\mathcal{B} \in \mathcal{F}_{\mathcal{Y}}$, the map

$$ \begin{cases} \mathcal{X} \to [0, 1] \\ x \mapsto P_{Y|X=x}(\mathcal{B}) \end{cases} \qquad (1) $$

is Borel measurable. The set of such conditional probability measures is denoted by $\Delta(\mathcal{Y}|\mathcal{X})$.

Given two measures $P$ and $Q$ on the same measurable space, the notation $P \ll Q$ stands for “the measure $P$ is absolutely continuous with respect to $Q$”. Using this notation, the relative entropy or KL-divergence is defined hereunder.

**Definition 2** (Relative Entropy). Given a probability measure $P$ and a $\sigma$-finite measure $Q$, both on the same measurable space, with $P \ll Q$. The relative entropy of $P$ with respect to $Q$ is

$$ D(P \parallel Q) \triangleq \int \frac{dP}{dQ}(x) \log \left( \frac{dP}{dQ}(x) \right) dQ(x). \qquad (2) $$

**Lemma 1.** Consider three probability measures $P, Q$, and $R$ on the same measurable space such that $R \ll P \ll Q$, it then follows that

$$ \int \log \left( \frac{dP}{dQ}(\theta) \right) dR(\theta) = D(R \parallel Q) - D(R \parallel P). \qquad (3) $$

*Proof:* Under the absolute continuity assumptions, i.e., $R \ll P \ll Q$, from Definition 2 it follows that

$$ \int \log \left( \frac{dP}{dQ}(\theta) \right) dR(\theta) = \int \log \left( \frac{dP}{dR}(\theta) \frac{dR}{dQ}(\theta) \right) dR(\theta) \qquad (4) $$

$$ = \int -\log \left( \frac{dR}{dP}(\theta) \right) + \log \left( \frac{dR}{dQ}(\theta) \right) dR(\theta) \qquad (5) $$

$$ = D(R \parallel Q) - D(R \parallel P), \qquad (6) $$

where the equality in (4) follows from [18, Theorem 5] and the equality in (5) follows from [18, Theorem 4]. ■
## 2 Supervised Machine Learning

Let $\mathcal{M}$, $\mathcal{X}$ and $\mathcal{Y}$, with $\mathcal{M} \subseteq \mathbb{R}^d$ and $d \in \mathbb{N}$, be sets of models, patterns, and labels, respectively. The training data available is assumed to be partitioned into $K$ smaller datasets. For all $k \in \{1, 2, \dots, K\}$, the $k$-th dataset consists of $n_k$ data points $(x_{k,1}, y_{k,1}), \dots, (x_{k,n_k}, y_{k,n_k})$, which are elements of the set $\mathcal{Z} \triangleq \mathcal{X} \times \mathcal{Y}$. Such data points form the dataset $z_k \in \mathcal{Z}^{n_k}$, which can be explicitly written as

$$z_k \triangleq ((x_{k,1}, y_{k,1}), \dots, (x_{k,n_k}, y_{k,n_k})) \in \mathcal{Z}^{n_k}. \quad (7)$$

The aggregated training dataset is then defined explicitly as,

$$z_0 \triangleq (z_1, \dots, z_K) \in \mathcal{Z}^{n_1} \times \mathcal{Z}^{n_2} \times \dots \times \mathcal{Z}^{n_K}, \quad (8)$$

with size $n_0 \triangleq \sum_{k=1}^K n_k$ data points. Given a model $\theta \in \mathcal{M}$, the loss induced by such a model with respect to a data point $(x, y) \in \mathcal{Z}$ is $\ell(x, y, \theta)$, where the function

$$\ell: \mathcal{Z} \times \mathcal{M} \to [0, +\infty), \quad (9)$$

is referred to as the loss function and is assumed to be Borel measurable. Using such a loss function, the empirical risk induced by a model $\theta \in \mathcal{M}$, with respect to a dataset $z_k \in \mathcal{Z}^{n_k}$, with $k \in \{0, 1, \dots, K\}$, is determined by the function

$$L_k: \begin{cases} \mathcal{Z}^{n_k} \times \mathcal{M} \to [0, +\infty) \\ (z_k, \theta) \mapsto \frac{1}{n_k} \sum_{i=1}^{n_k} \ell(x_{k,i}, y_{k,i}, \theta). \end{cases} \quad (10)$$

The functions $L_0, L_1, \dots, L_K$ exhibit the following property.

**Lemma 2.** Consider the training dataset $z_0$ in (8), then the empirical risk functions $L_0, L_1, \dots, L_K$ in (10) satisfy for all $\theta \in \mathcal{M}$,

$$L_0(z_0, \theta) = \sum_{k=1}^{K} \frac{n_k}{n_0} L_k(z_k, \theta), \quad (11)$$

*Proof:* The proof is similar to the proof of [14, Lemma 4].

<figure><img src="image_4.png" /></figure>

In the following, a supervised machine learning algorithm is represented by a conditional probability measure, as defined hereunder.

**Definition 3 (Algorithm).** A conditional probability measure $P_{\Theta|\mathcal{Z}_k} \in \Delta(\mathcal{M} | (\mathcal{X} \times \mathcal{Y})^{n_k})$ is said to represent a supervised machine learning algorithm. The instance of such an algorithm trained upon the dataset $z_k$ in (7) is denoted by $P_{\Theta|\mathcal{Z}_k=z_k}$, which is simply a probability measure in $\Delta(\mathcal{M})$.

A class of algorithms that are central in this work are known as Gibbs algorithms. These algorithms are introduced in the following section.
## 3 Gibbs Algorithms

Gibbs algorithms are represented by Gibbs conditional probability measures. Such conditional probability measures are parametrized by the empirical risk function L_k in (10); a σ-finite measure Q ∈ Δ(M); and a dataset z_k ∈ Z^{n_k}, with k ∈ {0, 1, ..., K}. In order to define such Gibbs measure consider the following function:

$$ K_{k,Q,z_k}: \begin{cases} \mathbb{R} \to \mathbb{R} \\ t \mapsto \log \left( \int \exp \left( t L_k (z_k, \boldsymbol{\theta}) \right) dQ \left( \boldsymbol{\theta} \right) \right). \end{cases} \quad (12) $$

Under the assumption that the reference measure Q is a probability measure, the function K_{k,Q,z_k} in (12) is the cumulant generating function of the random variable L_k(z_k, θ), for some fixed dataset z_k ∈ Z^{n_k}, when the model θ is sampled from Q. Using this notation, the definition of Gibbs conditional probability measures on the models is presented hereunder.

**Definition 4.** Given the function L_k in (10), with k ∈ {0, 1, ..., K}; a σ-finite measure Q ∈ Δ(M); and a λ_k ∈ R \ {0}, the probability measure P_{θ|Z_k}^{(Q,λ_k)} ∈ Δ(M | Z^{n_k}) is said to be an (L_k, Q, λ_k)-Gibbs conditional probability measure if

$$ \forall z_k \in S_k, K_{k,Q,z_k} \left( -\frac{1}{\lambda_k} \right) < +\infty; \quad (13) $$

for some set S_k ⊆ Z^{n_k}; and for all (z_k, θ) ∈ S_k × supp Q,

$$ \frac{dP_{\Theta|Z_k=z_k}^{(Q,\lambda_k)}}{dQ}(\theta) = \exp \left( -\frac{1}{\lambda_k} L(z_k, \theta) - K_{k,Q,z_k} \left( -\frac{1}{\lambda_k} \right) \right), \quad (14) $$

Note that if λ_k < 0, the condition in (13) may fail. Therefore, throughout this work, the parameter λ_k is chosen so that (13) holds. Moreover, if the probability measure from which datasets are sampled is P_{Z_k} ∈ Δ(Z^{n_k}), the set S_k in Definition 4 is chosen to be the support of P_{Z_k}. Note that, while P_{θ|Z_k}^{(Q,λ_k)} in (14) is referred to as a Gibbs conditional probability measure, the measure P_{θ|Z_k=z_k}^{(Q,λ_k)} obtained by conditioning upon a given dataset z_k ∈ Z^{n_k}, is referred to as a Gibbs probability measure. Consider the functional R_{z_k} defined as follows,

$$ R_{z_k}: \begin{cases} \Delta(M) \to [0, +\infty) \\ P \mapsto \int L_k(z_k, \theta) dP(\theta), \end{cases} \quad (15) $$

where the function L_k is defined in (10). The Gibbs probability measure P_{θ|Z_k=z_k}^{(Q,λ_k)} in (14) is related to the following optimization problem, under the assumption that λ_k > 0:

$$ \min_{P \in \Delta_Q(M)} R_{z_k}(P) + \lambda_k D(P \parallel Q). \quad (16) $$
Alternatively, when $\lambda_k < 0$, such a Gibbs probability measure is related to the following optimization problem:

$$ \max_{P \in \Delta_Q(M)} \mathbf{R}_{z_k}(P) + \lambda_k D(P \| Q), \qquad (17) $$

where the functional $\mathbf{R}_{z_k}$ is defined in (15). The importance of these optimization problems stems from the fact that the unlearning problem can be posed as the optimization problem in (17). The following lemma shows the connections between the Gibbs probability measure $P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)}$ in (14) and the optimization problems in (16) and (17).

**Lemma 3.** The $(L_k, Q, \lambda_k)$-Gibbs probability measure $P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)}$ in (14) is the unique solution to (16) (respectively, to (17)), if $\lambda_k > 0$ (respectively, if $\lambda_k < 0$).

*Proof:* The proof is immediate from [19, Lemma 1]. ■

Two optimization problems that are also closely related to the probability measure $P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)}$ in (14) are the following. When $\lambda_k > 0$, the optimization problem is:

$$ \min_{P \in \Delta_Q(M)} \mathbf{R}_{z_k}(P) \qquad (18a) $$

$$ \text{s.t.} \quad D(P \| Q) \le \gamma_k, \qquad (18b) $$

for some $\gamma_k > 0$. Alternatively, when $\lambda_k < 0$, the optimization problem is:

$$ \max_{P \in \Delta_Q(M)} \mathbf{R}_{z_k}(P) \qquad (19a) $$

$$ \text{s.t.} \quad D(P \| Q) \le \gamma_k. \qquad (19b) $$

The following lemma formalizes these observations.

**Lemma 4.** Assume that $\lambda_k > 0$ (respectively, $\lambda_k < 0$) is such that

$$ D(P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)} \| Q) = \gamma_k. \qquad (20) $$

Then, the $(L_k, Q, \lambda_k)$-Gibbs probability measure $P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)}$ in (14) is the unique solution to (18) (respectively, to (19)).

*Proof:* The proof follows from [19, Lemma 4]. ■

Lemma 4 implies that the probability measure $P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)}$ in (14) is the one that minimizes (respectively, maximizes) the training empirical risk over all probability measures in the following neighborhood of $Q$,

$$ \{P \in \Delta_Q(M) : D(P \| Q) \le D(P_{\Theta|Z_k=z_k}^{(Q, \lambda_k)} \| Q)\}. \qquad (21) $$

The relevance of Gibbs algorithms, in particular when $\lambda_k > 0$, stems from the observation that the conditional probability measure $P_{\Theta|Z_k}^{(Q, \lambda_k)}$ in (14) is the long-run distribution of a stochastic gradient descent algorithm, as highlighted in [20].
# 4 Main Result

The main result of this work is presented using the following ($L_0$, $Q$, $\lambda_0$)-Gibbs probability measure

$$ P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \in \Delta(\mathcal{M}), \qquad (22a) $$

with $\lambda_0 > 0$, which represents a Gibbs algorithm trained upon the aggregated dataset $z_0$ in (8). That is, for all $\theta \in \text{supp } Q$,

$$ \frac{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}{\mathrm{d}Q}(\theta) = \frac{\exp\left(-\frac{1}{\lambda_0}L_0(z_0, \theta)\right)}{\int \exp\left(-\frac{1}{\lambda_0}L_0(z_0, \nu)\right) \mathrm{d}Q(\nu)}. \qquad (22b) $$

From Lemma 4, it follows that such an algorithm is optimal in the sense that the corresponding probability measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22) minimizes the expected training empirical risk with respect to $z_0$, over all probability measures in the following neighborhood of $Q$,

$$ \mathcal{G}_0 \triangleq \left\{ P \in \Delta_Q(\mathcal{M}) : D(P \parallel Q) \le D\left(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \parallel Q\right) \right\}. \qquad (23) $$

Consider that the dataset $z_0$ in (8) is partitioned into two smaller datasets $z_1 \in \mathcal{Z}^{n_1}$ and $z_2 \in \mathcal{Z}^{n_2}$, i.e. $K = 2$. The objective is to transform the ($L_0$, $Q$, $\lambda_0$)-Gibbs probability measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22) into an ($L_2$, $Q$, $\lambda_2$)-Gibbs probability measure

$$ P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)} \in \Delta(\mathcal{M}), \qquad (24a) $$

for some $\lambda_2 > 0$, which represents a Gibbs algorithm exclusively trained upon dataset $z_2$. That is, for all $\theta \in \text{supp } Q$,

$$ \frac{\mathrm{d}P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)}}{\mathrm{d}Q}(\theta) = \frac{\exp\left(-\frac{1}{\lambda_2}L_2(z_2, \theta)\right)}{\int \exp\left(-\frac{1}{\lambda_2}L_2(z_2, \nu)\right) \mathrm{d}Q(\nu)}. \qquad (24b) $$

From Lemma 4, it follows that the Gibbs algorithm represented by the measure $P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)}$ in (24) minimizes the expected empirical risk with respect to $z_2$ over all probability measures in the following neighborhood of $Q$,

$$ \mathcal{G}_1 \triangleq \left\{ P \in \Delta_Q(\mathcal{M}) : D(P \parallel Q) \le D\left(P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)} \parallel Q\right) \right\}. \qquad (25) $$

In a nutshell, the objective is to unlearn the dataset $z_1$ from the Gibbs algorithm represented by the probability measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22), while retaining the dataset $z_2$, which is often referred to as exact unlearning.

**Definition 5 (Exact unlearning).** Consider the algorithm represented by the ($L_0$, $Q$, $\lambda_0$)-Gibbs probability measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22) and the datasets $z_1$ and $z_2$ to be unlearned and retained, respectively, with $z_0$, $z_1$ and $z_2$ in (8). An
algorithm $P \in \Delta_Q(M)$ is said to achieve exact unlearning of $z_1$ from $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$, if for all $\theta \in \text{supp } Q$,

$$ \frac{dP}{dP_{\Theta|Z_2=z_2}^{(Q, \lambda_2)}} (\theta) = 1, \qquad (26) $$

where the probability measure $P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)}$ is defined in (24), for some $\lambda_2 > 0$.

The main result of this work consists in showing that exact unlearning of $z_1$ from $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ is achieved by solving the optimization problem

$$ \max_{P \in \Delta_Q(M)} \mathbb{R}_{z_1}(P) + \lambda_1 D \left( P \;\middle\|\; P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \right), \qquad (27) $$

for some $\lambda_1 < 0$, where the functional $\mathbb{R}_{z_1}$ is defined in (15). From Lemma 3, it follows that the unique solution to (27) is an $(L_1, P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)$-Gibbs probability measure

$$ P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)} \in \Delta(M). \qquad (28) $$

The following subsection makes explicit a reweighting role of $\lambda_1$ in ERM-RER, which is the key ingredient used later on to achieve exact unlearning.

## 4.1 Reweighting Data Points in ERM-RER

The following theorem shows a reweighting structure when solving an empirical risk minimization (or maximization) with relative entropy regularization on the dataset $z_1$ while using, as reference measure, the $(L_0, Q, \lambda_0)$-Gibbs probability measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22).

**Theorem 5.** The $(L_1, P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)$-Gibbs probability measure $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)}$ in (28), with $\lambda_0 > 0$ and $\lambda_1 \in \mathbb{R} \setminus \{0\}$, satisfies for all $\theta \in \text{supp } Q$,

$$ \frac{dP_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)}}{dQ}(\theta) = \frac{\exp\left(-\frac{1}{\lambda_0}\left(\left(\frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0}\right)L_1(z_1, \theta) + \frac{n_2}{n_0}L_2(z_2, \theta)\right)\right)}{\int \exp\left(-\frac{1}{\lambda_0}\left(\left(\frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0}\right)L_1(z_1, \nu) + \frac{n_2}{n_0}L_2(z_2, \nu)\right)\right)dQ(\nu)}, \qquad (29) $$

where the function $L_2$ is defined in (10).

*Proof:* The proof is presented in Appendix B

<figure><img src="image_5.png" /></figure>

The reweighting structure in Theorem 5 can be obtained either from an ERM-RER minimization or, equivalently, from an ERM-RER maximization on the dataset $z_1$ with reference measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22). Specifically, for $\lambda_1 > 0$, consider the minimization problem

$$ \min_{P \in \Delta_P^{(Q, \lambda_0)}(M)} \mathbb{R}_{z_1}(P) + \lambda_1 D \left( P \;\middle\|\; P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \right), \qquad (30) $$

Inria
while for $\lambda_1 < 0$ the same solution is characterized as the maximizer of the optimization problem in (27). In both cases, the unique solution is

the $(L_1, P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)$-Gibbs probability measure $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)}$ in (28).

In general if $\lambda_0 > 0$ and $\lambda_1 \in \mathbb{R} \setminus \{0\}$, Theorem 5 unveils the following observation:

$$ \begin{aligned} & \arg\min_{P \in \Delta_Q(M)} \left( \frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0} \right) \mathrm{R}_{z_1}(P) + \frac{n_2}{n_0} \mathrm{R}_{z_2}(P) + \lambda_0 D(P \parallel Q) \\ &= \arg\min_{P \in \Delta_Q(M)} \frac{\lambda_0}{\lambda_1} \mathrm{R}_{z_1}(P) + \mathrm{R}_{z_0}(P) + \lambda_0 D(P \parallel Q) \end{aligned} \quad (31) $$

$$ = \left\{ \begin{array}{ll} \arg\min_{P \in \Delta_Q(M)} \mathrm{R}_{z_1}(P) + \lambda_1 D(P \parallel P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) & \text{if } \lambda_1 > 0; \\ \arg\max_{P \in \Delta_Q(M)} \mathrm{R}_{z_1}(P) + \lambda_1 D(P \parallel P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) & \text{if } \lambda_1 < 0 \end{array} \right. \quad (32) $$

$$ = \left\{ P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)} \right\}, \quad (33) $$

where the equality in (31) follows from the fact that for all $P \in \Delta_Q(M)$,

$$ \mathrm{R}_{z_0}(P) = \frac{n_1}{n_0} \mathrm{R}_{z_1}(P) + \frac{n_2}{n_0} \mathrm{R}_{z_2}(P); \quad (34) $$

the equality in (32) follows from Theorem 5 and Lemma 3; and the equality in (33) follows from Lemma 3.

The equality in (31) highlights that $\lambda_1$ reweights the contribution of the dataset $z_1$ in the optimization problem whose solution is the measure $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)}$ in (28), relative to the optimization problem whose solution is the measure $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22). More specifically, from Lemma 3, it holds that

$$ \left\{ P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \right\} = \arg\min_{P \in \Delta_Q(M)} \left( \frac{n_1}{n_0} \mathrm{R}_{z_1}(P) + \frac{n_2}{n_0} \mathrm{R}_{z_2}(P) \right) + \lambda_0 D(P \parallel Q), \quad (35) $$

and from (33), it holds that

$$ \left\{ P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}, \lambda_1)} \right\} = \arg\min_{P \in \Delta_Q(M)} \left( \frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0} \right) \mathrm{R}_{z_1}(P) + \frac{n_2}{n_0} \mathrm{R}_{z_2}(P) \\ + \lambda_0 D(P \parallel Q). \quad (36) $$

The contribution of the dataset $z_1$ in the optimization problem in (35) is proportional to $\frac{n_1}{n_0}$, while in the optimization problem in (36), it is proportional to $\frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0}$. Interestingly, when $\lambda_1 > 0$, the contribution of the dataset $z_1$ is up-weighted in the optimization problem in (36). That is, $\frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0} > \frac{n_1}{n_0}$. The converse holds when $\lambda_1 < 0$. At the same time, it is important to highlight that the contribution of the dataset $z_2$ remains invariant in both optimization
problems (35) and (36). In particular, Theorem 5 makes explicit the role of $\lambda_1$ as a reweighting parameter of the contribution of the dataset $z_1$. Thus, exact unlearning can be achieved by choosing $\lambda_1$ to completely eliminate the contribution of dataset $z_1$. That is, $\lambda_1$ is chosen such that

$$ \frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0} = 0. \qquad (37) $$

The construction leading to this cancellation is detailed in the following section.

## 4.2 Exact Unlearning

The following theorem, which is the main result of this work, shows that a specific choice of $\lambda_1$, and $\lambda_2$ makes the $(L_1, P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)$-Gibbs probability measure in (28) identical to the probability measure $P_{\Theta|Z_2=z_2}^{(Q,\lambda_2)}$ in (24), in the sense of [18, Theorem 3]. Thus, achieving exact unlearning in the sense of Definition 5.

**Theorem 6.** Assume that

$$ \lambda_1 = -\frac{n_0}{n_1}\lambda_0 \quad \text{and} \quad \lambda_2 = \frac{n_0}{n_2}\lambda_0, \qquad (38) $$

for some $\lambda_0 > 0$. Then, the $(L_2, Q, \lambda_2)$-Gibbs probability measure $P_{\Theta|Z_2=z_2}^{(Q,\lambda_2)}$ in (24) and the $(L_1, P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)$-Gibbs probability measure $P_{\Theta|Z_1=z_1}^{(Q,\lambda_0)}$ in (28) satisfies for all $\theta \in \text{supp} Q$,

$$ \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_2=z_2}^{(Q,\lambda_2)}} (\theta) = 1. \qquad (39) $$

*Proof:* The proof is presented in Appendix C.

<figure><img src="image_6.png" /></figure>

Theorem 6 shows that the probability measure $P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}$ in (22) can be transformed into $P_{\Theta|Z_2=z_2}^{(Q,\lambda_2)}$ in (24) via the optimization problem in (27), achieving exact unlearning in the sense of Definition 5. An important observation is that only the dataset to be unlearned, i.e., dataset $z_1$, is used in the optimization problem in (27). The dependence of such an optimization problem on the dataset to be retained, i.e., $z_2$, is via the reference measure $P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}$, which represents the Gibbs algorithm from which the dataset $z_1$ must be unlearned. Hence, the optimization problem in (27) implies that unlearning the dataset $z_1$ from the Gibbs algorithm $P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}$ is achieved through training a new Gibbs algorithm exclusively upon the dataset to be unlearned, i.e., dataset $z_1$, while using as reference measure the Gibbs algorithm $P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}$ itself. Interestingly, this establishes a proof of exact unlearning as, under the scaling in (38), the $(L_1, P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)$-Gibbs probability measure $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}$ in (28) coincides with the Gibbs measure
that would have been obtained by training only on the retained dataset $z_2$, i.e., the measure $P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)}$ in (24).

Another consequence of Theorem 6, and the uniqueness of the solutions to (16) and (27) implied by Lemma 3, is that

$$ 
\begin{aligned}
& \arg\max_{P \in \Delta_Q(\mathcal{M})} \mathrm{R}_{z_1}(P) - \frac{n_0}{n_1} \lambda_0 D(P \parallel P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) \\
& = \arg\min_{P \in \Delta_Q(\mathcal{M})} \mathrm{R}_{z_2}(P) + \frac{n_0}{n_2} \lambda_0 D(P \parallel Q).
\end{aligned} 
\quad (40) $$

From the perspective of Lemma 4, the equality in (40) implies that the probability measure in $\Delta_Q(\mathcal{M})$ that minimizes the expected empirical risk with respect to $z_2$ (dataset to be retained) over all measures in the set $\mathcal{G}_1$ in (25) is identical to the measure that maximizes the expected empirical risk with respect to $z_1$ (dataset to be unlearned) over all measures in the set

$$ \mathcal{G}_2 \triangleq \left\{ P \in \Delta_Q(\mathcal{M}) : D(P \parallel P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) \le D\left(P\left(P_{\Theta|Z_0=z_1}^{(Q, \lambda_0)}, \frac{n_0}{n_1} \lambda_0\right) \middle\| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}\right) \right\}. \quad (41) $$

Figure 1 provides a representation of (40) under the assumption in (38): it depicts how the common optimizer in (40) can be equivalently characterized either as an empirical-risk minimizer over $\mathcal{G}_1$ in (25) (relative to $z_2$) or as an empirical-risk maximizer over $\mathcal{G}_2$ in (41) (relative to $z_1$), while also highlighting the relation with $\mathcal{G}_0$ in (23). In particular, the figure shows the sets $\mathcal{G}_0$, $\mathcal{G}_1$, and $\mathcal{G}_2$ together with the probability measures $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22), $P_{\Theta|Z_2=z_2}^{(Q, \lambda_2)}$ in (24), and $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}$ in (28). In such a figure, the following notation is used:

$$ \alpha_0 \triangleq D\left(P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \middle\| Q\right); \quad (42) $$

$$ \alpha_1 \triangleq D\left(P_{\Theta|Z_2=z_2}^{(Q, \frac{n_0}{n_2} \lambda_0)} \middle\| Q\right); \quad \text{and} \quad (43) $$

$$ \alpha_2 \triangleq D\left(P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \frac{n_0}{n_1} \lambda_0)} \middle\| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}\right), \quad (44) $$

with $\alpha_0 < \alpha_1$.

Finally, note that the parameter $\lambda_0 > 0$ remains a free parameter that can be further optimized, e.g., to minimize the generalization error of the resulting Gibbs algorithm $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \frac{n_0}{n_1} \lambda_0)}$. See for instance [17, 22–25] and references therein for this purpose.
<figure><img src="image_7.png" /><figcaption>Figure 1: Representation, under the assumption in (38), of the set $G_0$ in (23) as a solid black circle; the set $G_1$ in (25) as a dashed red circle; and the set $G_2$ in (41) as a dotted blue circle. The involved probability measures are defined in (22), (24), and (28). The definitions of $\alpha_0, \alpha_1$, and $\alpha_2$ are in (42), (43) and (44), respectively.</figcaption></figure>

# 5 Extension to Decentralized Learning

A key ingredient in the unlearning mechanism is the availability of the centralized Gibbs algorithm trained on $z_0$, namely $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22). In the decentralized setting, such a centralized Gibbs measure can be obtained without explicit data aggregation by using the sequential decentralized Gibbs construction in [26]. This construction operates directly on Gibbs measures by propagating a reference measure across clients through successive local updates, while keeping raw datasets local.

The equivalence between the decentralized construction and the pooled-data/ centralized Gibbs algorithm in (22) relies on two mild modeling assumptions introduced in [26]. First, the clients may draw data from different domains, hence the local samples satisfy $z_k \in \mathcal{Z}_k^{n_k}$, and the aggregation in (8) is understood through the corresponding product space, i.e., $\mathcal{Z}_1^{n_1} \times \mathcal{Z}_2^{n_2} \times \dots \times \mathcal{Z}_K^{n_K}$. Second, all clients use the same loss function, i.e. the local losses satisfy

$$ \ell_1 = \ell_2 = \dots = \ell_K, \qquad (45) $$

so that the local empirical risks are defined consistently across clients. Under these assumptions and under the sample-size scaling of the regularization factors in [26], the final output of the decentralized Gibbs construction coincides with the centralized Gibbs algorithm that would have been obtained from $z_0$, $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ in (22).
Once the centralized Gibbs algorithm is available, the unlearning mechanism developed in this report can be implemented in a decentralized manner. Fix $j \in \{1, \dots, K\}$ and consider the deletion request of client $j$, i.e. the dataset $z_j \in \mathcal{Z}_j^{nj}$. Client $j$ solves an empirical risk maximization of the form in (27), on the dataset to be forgotten $z_j$ while taking $P_{\Theta|z_0=z_0}^{(Q, \Lambda_0)}$ in (22) as reference measure. This update requires access only to $z_j$ and to the reference Gibbs algorithm, and does not require explicit access to the retained datasets $z_1, z_2, \dots, z_{k-1}, z_{k+1}, \dots, z_K$.

Moreover, under a similar scaling of regularization factors as in (38), the resulting Gibbs algorithm admits an exact unlearning interpretation: the contribution of $z_j$ is canceled and the updated measure coincides with the Gibbs measure that would have been obtained by training only on the retained data. Importantly, the latter retained-data Gibbs measure can itself be obtained in a decentralized manner by applying the decentralized Gibbs construction of [26] over the retained clients.

Hence, provided that the centralized Gibbs algorithm can be obtained in a decentralized manner under the assumptions $z_k \in \mathcal{Z}_k^{nk}$ and (45), the unlearning mechanism of the previous sections extends naturally to decentralized learning systems while preserving the central operational advantages: no explicit transmission of raw datasets and no retraining from scratch on the retained clients.

The same principle extends to multiple deletion requests. If $\mathcal{J} \subseteq \{1, \dots, K\}$ denotes the set of clients to be forgotten, the corresponding updates can be performed sequentially, each time taking the current centralized Gibbs algorithm (or its updated version) as reference. Hence, the results of the previous sections extend naturally to decentralized learning systems, provided that the centralized Gibbs algorithm associated with $z_0$ can be obtained in a decentralized manner as in [26].

# 6 Conclusions and Final Remarks

In this work, an exact unlearning method for Gibbs algorithms has been proposed in Theorem 6. The method consists in implementing a new Gibbs algorithm via the maximization of the empirical risk with respect to the dataset to be unlearned using as reference measure the algorithm from which data must be unlearned. Under the corresponding scaling, this construction guarantees exact unlearning in the sense that the new Gibbs algorithm coincides with the Gibbs algorithm that would have been obtained by retraining from scratch exclusively on the dataset to be retained. An important feature is that the corresponding optimization problem depends exclusively on the dataset to be unlearned, while the dependence on the dataset to be retained is captured solely through the reference measure. The proof of Theorem 6 relies on Theorem 5, which can be placed in a more general scenario than the problem of unlearning. In particular, it can be placed in the scenario of reweighting data points in ERM-RER [11]. From this perspective, unlearning appears as a special case in which zero weight is assigned to the contribution of the data points to be unlearned. More broadly,
this reweighting offers a principled mechanism to modulate the contribution of particular data points, which can be leveraged to improve the generalization error by attenuating the effect of atypical or overly influential data points. Conversely, the same mechanism can be used adversarially: by up-weighting outliers, an attacker may deliberately increase the generalization error of Gibbs algorithms, yielding a model-poisoning strategy within the ERM-RER learning framework. The strategic choice of the parameters in Theorem 5, beyond unlearning, is left out of the scope of this paper. Nonetheless, the theorem provides a unifying theoretical foundation for exact unlearning and data points reweighting in ERM-RER. This opens the door to developing both robustness-oriented reweighting and its adversarial counterpart in statistical machine learning.

## References

[1] European Parliament and Council of the European Union, “Regulation (EU) 2016/679 (General Data Protection Regulation),” Official Journal of the European Union, OJ L 119, 4.5.2016, pp. 1–88, 2016, art. 17.

[2] T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W.-C. Liew, H. Yin, and Q. V. H. Nguyen, “A survey of machine unlearning,” *ACM Transactions on Intelligent Systems and Technology*, vol. 16, no. 5, pp. 1–46, Oct. 2025.

[3] S. Sai, U. Mittal, V. Chamola, K. Huang, I. Spinelli, S. Scardapane, Z. Tan, and A. Hussain, “Machine un-learning: An overview of techniques, applications, and future directions,” *Cognitive Computation*, vol. 16, no. 2, pp. 482–506, Mar. 2024.

[4] J. Xu, Z. Wu, C. Wang, and X. Jia, “Machine unlearning: Solutions and challenges,” *IEEE Transactions on Emerging Topics in Computational Intelligence*, vol. 8, no. 3, pp. 2150–2168, Jun. 2024.

[5] A. Ginart, M. Guan, G. Valiant, and J. Y. Zou, “Making AI forget you: Data deletion in machine learning,” in *Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS)*, vol. 32. Vancouver, Canada: Curran Associates, Inc., 2019.

[6] Y. Cao and J. Yang, “Towards making systems forget with machine unlearning,” in *Proceedings of the IEEE Symposium on Security and Privacy*, San Jose, CA, USA, May 2015, pp. 463–480.

[7] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearning,” in *Proceedings of the IEEE Symposium on Security and Privacy (SP)*, San Francisco, CA, USA, May 2021, pp. 141–159.

[8] M. Egger, R. Bitar, and R. L. Urbanke, “Efficient machine unlearning by model splitting and core sample selection,” in *Proceedings of the IEEE*
*Information Theory Workshop (ITW)*, Sydney, NSW, Australia, Sep. 2025, pp. 1–6.

[9] Y. Jiang, C.-W. Tan, and K.-Y. Lam, “Feduhb: Accelerating federated unlearning via Polyak heavy ball method,” in *Proceedings of the IEEE Information Theory Workshop (ITW)*, Shenzhen, China, Nov. 2024, pp. 235–240.

[10] C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, “Certified data removal from machine learning models,” in *Proceedings of the 37th International Conference on Machine Learning (ICML)*, vol. 119, Virtual Event, Jul. 2020, pp. 3832–3842.

[11] S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, “Empirical risk minimization with relative entropy regularization,” *IEEE Transactions on Information Theory*, vol. 70, no. 7, pp. 5122–5161, Jul. 2024.

[12] X. Zou, S. M. Perlaza, I. Esnaola, and E. Altman, “Generalization analysis of machine learning algorithms via the worst-case data-generating probability measure,” in *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 38, no. 15, Vancouver, Canada, Feb. 2024, pp. 17271–17279.

[13] F. Daunas, I. Esnaola, S. M. Perlaza, and H. V. Poor, “Asymmetry of the relative entropy in the regularization of empirical risk minimization,” *IEEE Transactions on Information Theory*, vol. 71, no. 8, pp. 6198–6226, Aug. 2025.

[14] S. M. Perlaza, I. Esnaola, G. Bisson, and H. V. Poor, “On the validation of Gibbs algorithms: Training datasets, test datasets and their aggregation,” in *Proceedings of the International Symposium on Information Theory (ISIT)*, Taipei, Taiwan, June 2023, pp. 328–333.

[15] F. Daunas, I. Esnaola, S. M. Perlaza, and H. V. Poor, “Equivalence of empirical risk minimization to regularization on the family of *f*-divergences,” in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Athens, Greece, Jul. 2024, pp. 759–764.

[16] Y. Bu, “Towards optimal inverse temperature in the Gibbs algorithm,” in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Athens, Greece, Jul. 2024, pp. 2257–2262.

[17] S. M. Perlaza and X. Zou, “The generalization error of machine learning algorithms,” preprint arXiv:2411.12030, 2024.

[18] Y. Bermudez, G. Bisson, I. Esnaola, and S. M. Perlaza, “Proofs for folk-lore theorems on the Radon-Nikodym derivative,” INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis, France, Tech. Rep. RR-9591, July 2025.
[19] S. M. Perlaza and G. Bisson, "Variations on the expectation due to changes in the probability measure," *Entropy*, vol. 27, no. 8:865, pp. 1-20, Aug. 2025.

[20] W. Azizian, F. Lutzeler, J. Malick, and P. Mertikopoulos, "What is the long-run distribution of stochastic gradient descent? A large deviations analysis," in *Proceedings of the International Conference on Machine Learning (ICML)*, Vienna, Austria, July 2024, pp. 2168 – 2229.

[21] H. Jeffreys, "An invariant form for the prior probability in estimation problems," *Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences*, vol. 186, no. 1007, pp. 453-461, Sept. 1946.

[22] G. Aminian, Y. Bu, L. Toni, M. R. D. Rodrigues, and G. W. Wornell, "Information-theoretic characterizations of generalization error for the Gibbs algorithm," *IEEE Transactions on Information Theory*, vol. 70, no. 1, pp. 632-655, 2024.

[23] X. Zou, S. M. Perlaza, I. Esnaola, E. Altman, and H. V. Poor, "The worst-case data-generating probability measure in statistical learning," *IEEE Journal on Selected Areas in Information Theory*, vol. 5, p. 175–189, Apr. 2024.

[24] G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell, "An exact characterization of the generalization error for the Gibbs algorithm," in *Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS)*, vol. 34, Virtual Event, Dec. 2021, pp. 8106–8118.

[25] A. Xu and M. Raginsky, "Information-theoretic analysis of generalization capability of learning algorithms," in *Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS)*, vol. 30, Long Beach, CA, USA, Dec. 2017, pp. 2521–2530.

[26] Y. Bermudez, S. M. Perlaza, and I. Esnaola, "Decentralized machine learning with centralized performance guarantees via Gibbs algorithms," INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis, France, Tech. Rep. RR-9608, January 2026.
# Appendices

## A Preliminaries for the Proof of Theorem 5

**Lemma 7.** The $(L_1, P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)$-Gibbs probability measure $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}$ in (28), with $\lambda_0 > 0$ and $\lambda_1 \in \mathbb{R} \setminus \{0\}$, satisfies for all $\theta \in \text{supp } Q$,

$$ \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}}(\theta) = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}}{\mathrm{d}Q}(\theta) \exp(C), \quad (46) $$

where the measure $P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}$ is an $(L_1, Q, \lambda_1)$-Gibbs probability measure (Definition 4), and $C \in \mathbb{R}$ satisfies

$$ C = \log \left( \frac{\exp \left( K_{0,Q,z_0} \left( -\frac{1}{\lambda_0} \right) + K_{1,Q,z_1} \left( -\frac{1}{\lambda_1} \right) \right)}{\int \exp \left( -\frac{1}{\lambda_1} L_1 (z_1, \nu) - \frac{1}{\lambda_0} L_0 (z_0, \nu) \right) dQ (\nu)} \right), \quad (47) $$

where the functions $K_{0,Q,z_0}$ and $K_{1,Q,z_1}$ are defined as in (12).

*Proof:* From Definition 4, for all $\theta \in \text{supp } Q$, it follows that

$$ \frac{\mathrm{d}P_{\Theta|Z=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}}(\theta) = \exp\left(\frac{-1}{\lambda_1}L_1(z_1, \theta) - K_{1,P_{\Theta|Z_0=z_0},z_1}^{(Q,\lambda_0)}\left(\frac{-1}{\lambda_1}\right)\right) \quad (48) $$

$$ = \frac{\exp\left(\frac{-1}{\lambda_1}L_1(z_1, \theta)\right)}{\int \exp\left(\frac{-1}{\lambda_1}L_1(z_1, \nu)\right) dP_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}(\nu)} \quad (49) $$

$$ = \frac{\exp\left(\frac{-1}{\lambda_1}L_1(z_1, \theta)\right)}{\int \exp\left(\frac{-1}{\lambda_1}L_1(z_1, \nu)\right) \frac{dP_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}}{dQ}(\nu) dQ(\nu)} \quad (50) $$

$$ = \frac{\exp\left(-\frac{1}{\lambda_1}L_1(z_1, \theta)\right) \exp\left(K_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right)\right)}{\int \exp\left(-\frac{1}{\lambda_1}L_1(z_1, \nu) - \frac{1}{\lambda_0}L_0(z_0, \nu)\right) dQ(\nu)} \quad (51) $$

$$ = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}(\theta)}{\mathrm{d}Q} \frac{\exp\left(K_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + K_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)}{\int \exp\left(-\frac{1}{\lambda_1}L_1(z_1, \nu) - \frac{1}{\lambda_0}L_0(z_0, \nu)\right) dQ(\nu)}, \quad (52) $$

where the equality in (50) follows from [18, Theorem 2] and the equality in (51) follows from Definition 4. The second factor in (52) does not depend on $\theta$ and
is denoted by

$$ \exp(C) \triangleq \frac{\exp\left(K_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + K_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)}{\int \exp\left(-\frac{1}{\lambda_1}L_1(z_1, \nu) - \frac{1}{\lambda_0}L_0(z_0, \nu)\right) dQ(\nu)}. \quad (53) $$

Combining (52) and (53) yields for all $\theta \in \text{supp } Q_1$,

$$ \frac{\mathrm{d}P_{\Theta|Z=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}(\theta) = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}}{\mathrm{d}Q}(\theta) \exp(C), \quad (54) $$

which completes the proof of (46).

<figure><img src="image_8.png" /></figure>

## B Proof of Theorem 5

From Lemma 7, the following identity holds for all $\theta \in \text{supp } Q$

$$ \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}(\theta) = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}}{\mathrm{d}Q}(\theta) \frac{\exp\left(K_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + K_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)}{\int \exp\left(-\frac{1}{\lambda_1}L_1(z_1, \nu) - \frac{1}{\lambda_0}L_0(z_0, \nu)\right) dQ(\nu)}. \quad (55) $$

From [11, Lemma 3] and [18, Theorem 2] the Radon-Nikodym derivative $\frac{\mathrm{d}Q}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}$ is well defined and from [18, Theorem 4], for all $\theta \in \text{supp } Q$, it follows that

$$ \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}(\theta) = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{\mathrm{d}Q}(\theta) \frac{\mathrm{d}Q}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}(\theta). \quad (56) $$

Combining (55) and (56) yields for all $\theta \in \text{supp } Q$,

$$ \begin{aligned} & \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{\mathrm{d}Q}(\theta) \\ &= \left(\frac{\mathrm{d}Q}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}(\theta)\right)^{-1} \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}}{\mathrm{d}Q}(\theta) \frac{\exp\left(K_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + K_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)}{\int \exp\left(-\frac{1}{\lambda_1}L_1(z_1, \nu) - \frac{1}{\lambda_0}L_0(z_0, \nu)\right) dQ(\nu)}. \quad (57) \end{aligned} $$

$$ = \frac{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}}{\mathrm{d}Q}(\theta) \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}}{\mathrm{d}Q}(\theta) \frac{\exp\left(K_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + K_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)}{\int \exp\left(-\frac{1}{\lambda_1}L_1(z_1, \nu) - \frac{1}{\lambda_0}L_0(z_0, \nu)\right) dQ(\nu)}. \quad (58) $$
$$ = \frac{\exp\left(-\frac{1}{\lambda_1}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta}) - \frac{1}{\lambda_0}\mathbf{L}_0(\mathbf{z}_0, \boldsymbol{\theta})\right)}{\exp\left(\mathbf{K}_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + \mathbf{K}_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)} \exp\left(\mathbf{K}_{0,Q,z_0}\left(-\frac{1}{\lambda_0}\right) + \mathbf{K}_{1,Q,z_1}\left(-\frac{1}{\lambda_1}\right)\right)} \int \exp\left(-\frac{1}{\lambda_1}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\nu}) - \frac{1}{\lambda_0}\mathbf{L}_0(\mathbf{z}_0, \boldsymbol{\nu})\right) dQ(\boldsymbol{\nu}) \quad (59) $$

$$ = \frac{\exp\left(-\frac{1}{\lambda_1}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta}) - \frac{1}{\lambda_0}\mathbf{L}_0(\mathbf{z}_0, \boldsymbol{\theta})\right)}{\int \exp\left(-\frac{1}{\lambda_1}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\nu}) - \frac{1}{\lambda_0}\mathbf{L}_0(\mathbf{z}_0, \boldsymbol{\nu})\right) dQ(\boldsymbol{\nu})} \quad (60) $$

$$ = \frac{\exp\left(-\frac{1}{\lambda_0}\left(\frac{n_1}{n_0}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta}) + \frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\theta})\right) + \frac{-1}{\lambda_1}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta})\right)}{\int \exp\left(-\frac{1}{\lambda_0}\left(\frac{n_1}{n_0}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta}) + \frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\nu})\right) + \frac{-1}{\lambda_1}\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\nu})\right) dQ(\boldsymbol{\nu})} \quad (61) $$

$$ = \frac{\exp\left(-\frac{1}{\lambda_0}\left(\left(\frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0}\right)\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta}) + \frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\theta})\right)\right)}{\int \exp\left(-\frac{1}{\lambda_0}\left(\left(\frac{\lambda_0}{\lambda_1} + \frac{n_1}{n_0}\right)\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\nu}) + \frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\nu})\right)\right) dQ(\boldsymbol{\nu})} \quad (62) $$

where the equality in (58) follows from [18, Theorem 5]; the equality in (59) follows from Definition 4; and the equality in (61) follows from Lemma 2. This completes the proof.

## C Proof of Theorem 6

Under the assumptions in (38), it holds that $\lambda_1 = -\frac{n_0}{n_1}\lambda_0$ and $\lambda_2 = \frac{n_0}{n_2}\lambda_0$, from Theorem 5, for all $\boldsymbol{\theta} \in \text{supp } Q$, it follows that

$$ \frac{dP_{\boldsymbol{\theta}|\mathbf{Z}_1=\mathbf{z}_1}^{(P_{\boldsymbol{\theta}|\mathbf{Z}_0=\mathbf{z}_0}, \lambda_1)}(\boldsymbol{\theta})}{dQ} = \frac{\exp\left(-\frac{1}{\lambda_0}\left(\left(\frac{\lambda_0}{-\frac{n_0}{n_1}\lambda_0} + \frac{n_1}{n_0}\right)\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\theta}) + \frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\theta})\right)\right)}{\int \exp\left(-\frac{1}{\lambda_0}\left(\left(\frac{\lambda_0}{-\frac{n_0}{n_1}\lambda_0} + \frac{n_1}{n_0}\right)\mathbf{L}_1(\mathbf{z}_1, \boldsymbol{\nu}) + \frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\nu})\right)\right) dQ(\boldsymbol{\nu})} \quad (63) $$

$$ = \frac{\exp\left(-\frac{1}{\lambda_0}\left(\frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\theta})\right)\right)}{\int \exp\left(-\frac{1}{\lambda_0}\left(\frac{n_2}{n_0}\mathbf{L}_2(\mathbf{z}_2, \boldsymbol{\nu})\right)\right) dQ(\boldsymbol{\nu})} \quad (64) $$

$$ = \frac{dP_{\boldsymbol{\theta}|\mathbf{Z}_2=\mathbf{z}_2}^{(Q, \frac{n_0}{n_2}\lambda_0)}(\boldsymbol{\theta})}{dQ} \quad (65) $$

where the equality in (65) follows from (24b).

Finally, from [18, Theorem 5] and (65), for all $\boldsymbol{\theta} \in \text{supp } Q$,

$$ 1 = \frac{dP_{\boldsymbol{\theta}|\mathbf{Z}_1=\mathbf{z}_1}^{(P_{\boldsymbol{\theta}|\mathbf{Z}_0=\mathbf{z}_0}, \frac{n_0}{n_1}\lambda_0)}(\boldsymbol{\theta})}{dQ} \frac{dQ}{dP_{\boldsymbol{\theta}|\mathbf{Z}_2=\mathbf{z}_2}^{(Q, \frac{n_0}{n_2}\lambda_0)}(\boldsymbol{\theta})} \quad (66) $$
$$ = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \frac{n_0}{n_1}\lambda_0)}}{\mathrm{d}P_{\Theta|Z_2=z_2}^{(Q, \frac{n_0}{n_2}\lambda_0)}} (\theta), \qquad (67) $$

where the equality in (67) follows from [18, Theorem 4] and completes the proof.

# D Complementary Results

**Lemma 8.** The value C in (47) satisfies

$$ \begin{aligned} C &= D(P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)} \| P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}) - D(P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)} \| P_{\Theta|Z_0=z_0}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}) - D(P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)} \| Q) \quad (68) \\ &= D(Q \| P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}) - D(Q \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}) + D(Q \| P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}), \quad (69) \end{aligned} $$

where the measure $P_{\Theta|Z=z_1}^{(Q,\lambda_1)}$ is an $(L_1, Q, \lambda_1)$-Gibbs probability measure; the measures $P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}$ and $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}$ are defined respectively in (22) and (28).

*Proof:* From Lemma 7, for all $\theta \in \text{supp } Q$, it follows that

$$ \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}}(\theta) = \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}}{\mathrm{d}Q}(\theta) \exp(C), \quad (70) $$

with

$$ C = \log \left( \frac{\exp \left( K_{0,Q,z_0} \left( -\frac{1}{\lambda_0} \right) + K_{1,Q,z_1} \left( -\frac{1}{\lambda_1} \right) \right)}{\int \exp \left( -\frac{1}{\lambda_1} L_1(z_1, \nu) - \frac{1}{\lambda_0} L_0(z_0, \nu) \right) dQ(\nu)} \right). \quad (71) $$

By taking a logarithm and integrating with respect to $P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}$ in (70) it follows that

$$ \begin{aligned} & \int \log \left( \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}, \lambda_1)}}{\mathrm{d}P_{\Theta|Z_0=z_0}^{(Q,\lambda_0)}}(\theta) \right) \mathrm{d}P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}(\theta) \\ &= \int \log \left( \frac{\mathrm{d}P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}}{\mathrm{d}Q}(\theta) \right) \mathrm{d}P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)}(\theta) + C \end{aligned} \quad (72) $$

$$ = D(P_{\Theta|Z_1=z_1}^{(Q,\lambda_1)} \| Q) + C. \quad (73) $$
On the other hand, from Lemma 1 it follows that

$$ 
\begin{aligned}
& \int \log \left( \frac{dP_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{dP_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}} \right) dP_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \\
& = D \left( P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \right) - D \left( P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)} \right). \quad (74)
\end{aligned}
$$

Combining (73) and (74) yields

$$ C = D(P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) - D(P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}) - D(P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| Q) \quad (75) $$

which completes the proof of (68).

Following the same method, if the integral in (70) is taken with respect to $Q$, it then follows that,

$$ 
\begin{aligned}
& \int \log \left( \frac{dP_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{dP_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}} (\theta) \right) dQ(\theta) \\
& = \int \log \left( \frac{dP_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}}{dQ} (\theta) \right) dQ(\theta) + C \quad (76)
\end{aligned}
$$

$$ = \int \log \left( \left( \frac{dQ}{dP_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}(\theta)} \right)^{-1} \right) dQ(\theta) + C \quad (77) $$

$$ = - \int \log \left( \frac{dQ}{dP_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}(\theta)} \right) dQ(\theta) + C \quad (78) $$

$$ = -D(Q \| P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}) + C, \quad (79) $$

where (77) follows from [11, Lemma 3], [18, Theorem 2] and [18, Theorem 5].

On the other hand, from Lemma 1 it follows that

$$ 
\begin{aligned}
& \int \log \left( \frac{dP_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}}{dP_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}} \right) dP_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \\
& = D \left( P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)} \right) - D \left( P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)} \right). \quad (80)
\end{aligned}
$$

Combining (79) and (80) yields
$$ C = D(Q \| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) - D(Q \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}) + D(Q \| P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}), \quad (81) $$

which completes the proof of (69).

<figure><img src="image_9.png" /></figure>

Lemma 8 implies a characterization of the Jeffreys divergence [21] between an $(L_1, Q, \lambda_1)$-Gibbs probability measure denoted by $P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \in \Delta(M)$ and its reference $Q$, as shown by the following corollary.

**Corollary 9.** The $(L_1, Q, \lambda_1)$-Gibbs probability measure denoted by, $P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \in \Delta(M)$ satisfies

$$ \begin{aligned} & D(P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| Q) + D(Q \| P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)}) \\ = & D(P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) - D(Q \| P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}) \\ & + D(P_{\Theta|Z_1=z_1}^{(Q, \lambda_1)} \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}) - D(Q \| P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}), \end{aligned} \quad (82) $$

where the measures $P_{\Theta|Z_0=z_0}^{(Q, \lambda_0)}$ and $P_{\Theta|Z_1=z_1}^{(P_{\Theta|Z_0=z_0}, \lambda_1)}$ are respectively defined in (22) and (28).
informatics mathematics
Inria

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex

Publisher
Inria
Domaine de Voluceau - Rocquencourt
BP 105 - 78153 Le Chesnay Cedex
inria.fr

ISSN 0249-6399