informatics mathematics
Inria

A Dual Optimization
View to Empirical Risk
Minimisation with
$f$-Divergence
Regularization

Francisco Daunas, Iñaki Esnaola, and Samir M. Perlaza

RESEARCH
REPORT

Nº 9594

July, 2025

Project-Team NEO
<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# A Dual Optimization View to Empirical Risk Minimisation with $f$-Divergence Regularization

Francisco Daunas, Iñaki Esnaola, and Samir M. Perlaza

Project-Team NEO

Research Report n° 9594 — July, 2025 — 27 pages

**Abstract:** The dual formulation of empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is introduced. The solution of the dual optimization problem to the ERM-$f$DR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide a nonlinear ODE expression to the normalization function. Furthermore, the nonlinear ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-$f$DR solution under a mild condition.

**Key-words:** Empirical risk minimization, $f$-divergence regularization, statistical learning.

Francisco Daunas and Iñaki Esnaola are with the Department of Automatic Control and Systems Engineering at the University of Sheffield, Sheffield, UK. Francisco Daunas and Samir M. Perlaza are with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis, France. Samir M. Perlaza is also with the GAATI Laboratory of the Université de la Polynésie Française, Faaa, French Polynesia. Iñaki Esnaola, and Samir M. Perlaza are with the ECE Department at Princeton University, Princeton, NJ.

This work is supported in part by a University of Sheffield ACSE PGR scholarship; University of Sheffield PGR Publication Scholarship scheme; the European Commission through the H2020-MSCA-RISE-2019 project 872172; the French National Agency for Research (ANR) through the Project ANR-21-CE25-0013 and the project ANR-22-PEFT-0010 of the France 2030 program PEPR Réseaux du Futur; and the Agence de l'innovation de défense (AID) through the project UK-FR 2024352.

Part of this work will appear in the Proceedings of the IEEE Information Theory Workshop (ITW) held in Sydney, Australia in September 2025 [1].
Une Approche d'Optimisation Duale de la
Minimisation du Risque Empirique avec
Régularisation par f-Divergence

**Résumé** : La formulation duale de la minimisation du risque empirique avec régularisation par divergence *f* (ERM-*f*DR) est introduite. La solution du problème d'optimisation dual de l'ERM-*f*DR est reliée à la notion de fonction de normalisation introduite comme une fonction implicite. Cette approche duale exploite la transformée de Legendre-Fenchel et le théorème des fonctions implicites pour fournir une expression sous forme d'EDO non linéaire de la fonction de normalisation. De plus, l'expression de l'EDO non linéaire et ses propriétés offrent une méthode de calcul efficace pour déterminer la fonction de normalisation de la solution de l'ERM-*f*DR sous une condition faible.

**Mots-clés** : Minimisation des risques empiriques, régularisation par des *f*-divergences, apprentissage statistique.
# Contents

<table><tr><td>1</td><td>Introduction</td><td>4</td></tr><tr><td>2</td><td>Preliminaries</td><td>4</td></tr><tr><td>3</td><td>The Learning Problem</td><td>5</td></tr><tr><td>4</td><td>ERM-fDR Dual Problem</td><td>7</td></tr><tr><td>5</td><td>Analysis of the Normalization Function</td><td>10</td></tr><tr><td>5.1</td><td>Characterization and Properties</td><td>10</td></tr><tr><td>5.2</td><td>Proof of Theorem 5.1</td><td>10</td></tr><tr><td>5.3</td><td>Discussion of Results</td><td>14</td></tr><tr><td>6</td><td>Conclusions</td><td>15</td></tr><tr><td></td><td>References</td><td>15</td></tr><tr><td></td><td>Appendices</td><td>19</td></tr><tr><td>A</td><td>Preliminaries</td><td>19</td></tr><tr><td>B</td><td>Proof of Lemma 5.1</td><td>24</td></tr></table>
# 1 Introduction

Empirical risk minimization (ERM) [2-7] is often posed as an optimization problem regularized by a statistical distance between the probability measure to be optimized and a given reference measure [8-14]. A well-studied case for such statistical distance is the relative entropy, which leads to the celebrated ERM with relative entropy regularization (ERM-RER) studied, for instance, in [15-22]. Other works address the general case for f-divergences, known as ERM with f-divergence regularization (ERM-fDR) [23-26]. The discrete case of ERM-fDR is explored in [23] and [24], while more general settings are covered in [25] and [26]. Recent results have shown that f-divergence regularization can improve the robustness of learning algorithms in the context of distributionally robust optimization (DRO) [27-30].

However, a main challenge in both the theory and practice of ERM-fDR is that the solution is known only up to a normalization factor [23-25]. For many f-divergences, a closed-form expression for the normalization factor is not known. This hinders sampling methods like Markov Chain Monte Carlo (MCMC) [31], where it appears in the likelihood ratio, which is needed for valid transition probabilities. In the case of rejection sampling [32], the absence of such a constant prevents defining an efficient proposal distribution. Even when the normalization factor has a known expression, such as the log-partition function in the case of relative entropy regularization [33], it can be intractable for general loss functions. More broadly, in ERM-fDR problems, computing this normalization factor is especially difficult because it requires evaluating the empirical risk across all models in the support of the reference measure, a task that is #P-hard [34-36].

This paper addresses this challenge by formalizing the concept of the normalization function introduced in [26] and deriving the dual optimization problem for ERM-fDR. The benefits of studying the dual optimization problem and its properties are twofold. First, insights for computing the normalization factor are obtained, and second, a characterization of the normalization function for ERM-fDR problems is presented. This characterization is obtained by leveraging tools from convex analysis, notably the Legendre-Fenchel transform [37,38], and the implicit function theorem [39].

# 2 Preliminaries

Let $\Omega$ be an arbitrary subset of $\mathbb{R}^d$, with $d \in \mathbb{N}$, and let $\mathcal{B}(\Omega)$ denote the Borel $\sigma$-field on $\Omega$. The set of probability measures that can be defined upon the measurable space $(\Omega, \mathcal{B}(\Omega))$ is denoted by $\Delta(\Omega)$. Given a probability measure $Q \in \Delta(\Omega)$ the set exclusively containing the probability measures in $\Delta(\Omega)$ that are absolutely continuous with respect to $Q$ is denoted by $\Delta_Q(\Omega)$. That is,

$$ \Delta_Q(\Omega) \triangleq \{P \in \Delta(\Omega) : P \ll Q\}, \quad (1) $$
where the notation $P \ll Q$ stands for the measure $P$ being absolutely continuous with respect to the measure $Q$. The Radon-Nikodym derivative of the measure $P$ with respect to $Q$ is denoted by $\frac{dP}{dQ} : \Omega \to [0, \infty)$.

Using this notation, an $f$-divergence is defined as follows.

**Definition 2.1** ($f$-divergence [40]). Let $f: [0, \infty) \to \mathbb{R}$ be a convex function with $f(1) = 0$ and $f(0) \triangleq \lim_{x \to 0^+} f(x)$. Let $P$ and $Q$ be two probability measures on the same measurable space, with $P \ll Q$. The $f$-divergence of $P$ with respect to $Q$, denoted by $D_f(P||Q)$, is

$$D_f(P||Q) \triangleq \int f\left(\frac{dP}{dQ}(\omega)\right)dQ(\omega). \qquad (2)$$

In the case in which the function $f$ in (2) is continuous and differentiable, the derivative of the function $f$ is denoted by

$$\dot{f} : (0, +\infty) \to \mathbb{R}. \qquad (3)$$

If the inverse of the function $\dot{f}$ exists, it is denoted by

$$\dot{f}^{-1} : \mathbb{R} \to (0, +\infty). \qquad (4)$$

## 3 The Learning Problem

Let $\mathcal{M}$, $\mathcal{X}$ and $\mathcal{Y}$, with $\mathcal{M} \subseteq \mathbb{R}^d$ and $d \in \mathbb{N}$, be sets of models, patterns, and labels, respectively. A pair $(x, y) \in \mathcal{X} \times \mathcal{Y}$ is referred to as a labeled pattern or data point, and a dataset is represented by the tuple $((x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathcal{Y})^n$. Let the function $h: \mathcal{M} \times \mathcal{X} \to \mathcal{Y}$ be such that the label assigned to a pattern $x \in \mathcal{X}$ according to the model $\theta \in \mathcal{M}$ is $h(\theta, x)$. Then, given a dataset

$$z = ((x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathcal{Y})^n, \qquad (5)$$

the objective is to obtain a model $\theta \in \mathcal{M}$, such that, for all $i \in \{1, 2, \dots, n\}$, the label assigned to the pattern $x_i$, which is $h(\theta, x_i)$, is “close” to the label $y_i$. This notion of “closeness” is formalized by the function

$$\ell: \mathcal{Y} \times \mathcal{Y} \to [0, +\infty), \qquad (6)$$

such that the loss or risk induced by choosing the model $\theta \in \mathcal{M}$ with respect to the labeled pattern $(x_i, y_i)$, with $i \in \{1, 2, \dots, n\}$, is $\ell(h(\theta, x_i), y_i)$. The risk function $\ell$ is assumed to be nonnegative and to satisfy $\ell(y, y) = 0$, for all $y \in \mathcal{Y}$.

The empirical risk induced by a model $\theta$ with respect to the dataset $z$ in (5) is determined by the function $L_z: \mathcal{M} \to [0, +\infty)$, which satisfies

$$L_z(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell(h(\theta, x_i), y_i). \qquad (7)$$
The expectation of the empirical risk $L_z(\theta)$ in (7), when $\theta$ is sampled from a probability measure $P \in \Delta(\mathcal{M})$, is determined by the functional $R_z : \Delta(\mathcal{M}) \to [0, +\infty)$, such that

$$R_z(P) = \int L_z(\theta) dP(\theta). \qquad (8)$$

The ERM-fDR problem is parametrized by a probability measure $Q \in \Delta(\mathcal{M})$, a positive real $\lambda$, and a function $f: [0, \infty) \to \mathbb{R}$ that satisfies the conditions in Definition 2.1. The measure $Q$ is referred to as the *reference measure* and $\lambda$ as the *regularization factor*.

Given the dataset $z \in (\mathcal{X} \times \mathcal{Y})^n$ in (5), the ERM-fDR problem, with parameters $Q$, $\lambda$ and $f$, is given by the following optimization problem

$$\min_{P \in \Delta_Q(\mathcal{M})} R_z(P) + \lambda D_f(P \| Q), \qquad (9)$$

where the functional $R_z$ is defined in (8). The set of solutions to (9) is the singleton $\{Q\}$ in the case in which for all $\theta \in \text{supp } Q$, $L_z(\theta) = c$, for some $c > 0$. This distinction is mathematically significant but can be ignored in practice, as it arises only when $R_z(P)$ in (8) is constant for all measures $P$. In order to avoid the above case, the notion of separable empirical risk functions [19, Definition 5] is adopted.

The solution to the ERM-fDR problems in (9) was first presented in [26, Theorem 1] under the following assumptions:

(a) The function $f$ is strictly convex and twice differentiable;

(b) There exists a $\beta$ such that

$$\beta \in \left\{ t \in \mathbb{R} : \forall \theta \in \text{supp } Q, 0 < \dot{f}^{-1}\left(-\frac{t + L_z(\theta)}{\lambda}\right) \right\}, \qquad (10a)$$

and

$$\int \dot{f}^{-1}\left(-\frac{\beta + L_z(\theta)}{\lambda}\right) dQ(\theta) = 1, \qquad (10b)$$

where the function $L_z$ is defined in (7); and

(c) The function $L_z$ in (7) is separable with respect to the probability measure $Q$.

**Theorem 3.1** ([26, Theorem 1]). Under Assumptions (a) and (b), the solution to the optimization problem in (9), denoted by $P_{\Theta|Z=z}^{(Q,\lambda)} \in \Delta_Q(\mathcal{M})$, is unique, and for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \dot{f}^{-1}\left(-\frac{\beta + L_z(\theta)}{\lambda}\right). \qquad (11)$$
The equality in (11) can be written in terms of the normalization function, introduced in [26] and defined hereunder.

**Definition 3.1** (Normalization Function). The normalization function of the problem in (9), denoted by

$$N_{Q,z} : \mathcal{A}_{Q,z} \to \mathcal{B}_{Q,z}, \qquad (12a)$$

with $\mathcal{A}_{Q,z} \subseteq (0, \infty)$ and $\mathcal{B}_{Q,z} \subseteq \mathbb{R}$, is such that for all $\lambda \in \mathcal{A}_{Q,z}$,

$$\int \dot{f}^{-1} \left( - \frac{N_{Q,z}(\lambda) + \mathbf{L}_z(\boldsymbol{\theta})}{\lambda} \right) dQ(\boldsymbol{\theta}) = 1. \qquad (12b)$$

The set $\mathcal{A}_{Q,z}$ in (12a) contains all the regularization factors $\lambda$ for which Assumption (b) is satisfied. More specifically, it contains the regularization factors $\lambda$ for which the problem in (9) has a solution. Furthermore, the equality in (12b) justifies referring to the function $N_{Q,z}$ as the normalization function, as it ensures that the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (11) is a probability measure.

This section ends by highlighting that the probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ in (11) are mutually absolutely continuous [26, Corollary 1]. This is important, as it ensures the Radon-Nikodym derivative of $Q$ with respect to $P_{\Theta|Z=z}^{(Q,\lambda)}$ is well defined, a property that is used repeatedly throughout the proofs.

# 4 The ERM-fDR Dual Problem

The duality principle [38, Chapter 5] enables the analysis of the optimization problem in (9) by studying an alternative form, known as the dual problem. In this section, this dual problem is derived using the Legendre-Fenchel transform [38], which is defined below.

**Definition 4.1** (Legendre-Fenchel transform [38]). Consider a function $f: \mathcal{I}_1 \to \mathcal{I}_2$, with $\mathcal{I}_i \subseteq \mathbb{R}$, with $i \in \{1,2\}$. The Legendre-Fenchel transform of the function $f$, denoted by $f^* : \mathcal{J} \to \mathbb{R}$, is

$$f^*(t) = \sup_{x \in \mathcal{I}_1} (tx - f(x)), \qquad (13)$$

with

$$\mathcal{J} = \{t \in \mathbb{R} : f^*(t) < \infty\}. \qquad (14)$$

Using this notation, consider the following problem

$$\min_{\beta \in \mathbb{R}} \lambda \int f^* \left( -\frac{1}{\lambda}(\beta + \mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) + \beta, \qquad (15)$$

where the real $\lambda$, the measure $Q$ and the function $f$ are those in (9); and the functions $\mathbf{L}_z$ and $f^*$ are defined in (7) and (13), respectively. The following theorem introduces the solution to the problem in (15).
**Theorem 4.1.** Under Assumptions (a) and (b), the solution to the optimization problem in (15) is $N_{Q,z}(\lambda)$, where the function $N_{Q,z}$ is defined in (12).

*Proof:* Let $G: \mathbb{R} \to \mathbb{R}$ be a function such that

$$G(\beta) = \lambda \int \left(-\frac{1}{\lambda}(\beta + \mathbf{L}_z(\boldsymbol{\theta}))\right) dQ(\boldsymbol{\theta}) + \beta, \quad (16)$$

which is the objective function of the optimization problem in (15). Note that from Assumption (a) and Definition 4.1, the function $G$ in (16) is a strictly convex function, which satisfies

$$\frac{d}{d\beta} G(\beta) = \frac{d}{d\beta} \left( \lambda \int f^* \left( -\frac{1}{\lambda}(\beta + \mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) + \beta \right) \quad (17)$$

$$= \lambda \int \frac{d}{d\beta} f^* \left( -\frac{1}{\lambda}(\beta + \mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) + 1 \quad (18)$$

$$= -\int f^* \left( -\frac{1}{\lambda}(\beta + \mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) + 1, \quad (19)$$

where $f^*$ is the derivative of the function $f^*$ in (15). Let the solution to the optimization problem in (16) be denoted by $\hat{\beta} \in \mathbb{R}$ and note that the derivative of the function $G$ evaluated at $\hat{\beta}$ is equal to zero, that is

$$\int f^* \left( -\frac{1}{\lambda}(\hat{\beta} + \mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) = 1. \quad (20)$$

From [37, Corollary 23.5.1] and Assumption (a), the following equality holds for all $t \in \mathcal{J}$, with $\mathcal{J}$ in (14),

$$\frac{d}{dt} f^*(t) = f^*(t) = \dot{f}^{-1}(t), \quad (21)$$

where the functions $\dot{f}^{-1}$ and $\dot{f}^*$ are defined in (4) and (19), respectively. From (20) and (21), it follows that

$$\int \dot{f}^{-1} \left( -\frac{1}{\lambda} (\hat{\beta} + \mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) = 1, \quad (22)$$

which combined with (12b) and Assumption (b) yields

$$N_{Q,z}(\lambda) = \hat{\beta}. \quad (23)$$

Note that $\hat{\beta}$ is unique as it is the minimum of a strictly convex function. This observation completes the proof. ■

The following lemma establishes that the problem in (15) is the dual problem to the ERM-fDR problem in (9) and characterizes the difference between their optimal values, which is often referred to as duality gap [41, Section 8.3].

**Lemma 4.1.** Under Assumptions (a) and (b), the optimization problem in (15) is the dual problem to the ERM-fDR problem in (9). Moreover, the duality gap is zero.
Proof: Under Assumption (a) and [38, Section 3.3.2], it can be verified that for all $t \in \mathcal{J}$, with $\mathcal{J}$ in (14), the function $f^*$ in (13) satisfies

$$f^*(t) = t\dot{f}^*(t) - f(\dot{f}^*(t)), \qquad (24)$$

where the function $\dot{f}^*$ is the same as in (21). From Assumption (a) and (21), the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (11) satisfies for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \dot{f}^*\left(-\frac{N_{Q,z}(\lambda) + L_z(\theta)}{\lambda}\right), \qquad (25)$$

where the functions $L_z$ and $N_{Q,z}$ are defined in (7) and (12), respectively. Then, from (24) and (25), with some algebraic manipulations, it holds that for all $\theta \in \text{supp } Q$,

$$\begin{aligned} & L_z(\theta) + \lambda f\left(\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)\right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) \\ & = -\lambda f^*\left(-\frac{N_{Q,z}(\lambda) + L_z(\theta)}{\lambda}\right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) - N_{Q,z}(\lambda). \end{aligned} \qquad (26)$$

Taking the expectation in both sides of (26) with respect to the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (11) yields

$$\begin{aligned} & R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) + \lambda D_f(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) \\ & = -\lambda \int f^*\left(-\frac{1}{\lambda}(N_{Q,z}(\lambda) + L_z(\theta))\right) dQ(\theta) - N_{Q,z}(\lambda). \end{aligned} \qquad (27)$$

Using Theorem 3.1 and Theorem 4.1 in the left-hand and right-hand sides of (27), respectively, yields

$$\begin{aligned} & \min_{P \in \Delta_Q(\mathcal{M})} R_z(P) + \lambda D_f(P \| Q) \\ & = \max_{\beta \in \mathbb{R}} -\lambda \int f^*\left(-\frac{1}{\lambda}(\beta + L_z(\theta))\right) dQ(\theta) - \beta. \end{aligned} \qquad (28)$$

The claim that the optimization problem in (15) is the dual to the ERM-fDR problem in (9) follows from (28) and [42, Theorem 1, Section 8.4]. The zero duality gap is established by the equality in (28), which completes the proof.

<figure><img src="image_3.png" /></figure>

The zero duality gap in Lemma 4.1 ensures strong duality, which implies that the optimal value of the dual variables recovers the primal optimal measure. However, solving the dual problem in (15) remains equally challenging, as the solution must satisfy Assumption (b) in Theorem 3.1. Thus, unless the normalization function $N_{Q,z}$ in (12) is explicitly characterized, computing $N_{Q,z}(\lambda)$ involves a search over the real line to obtain the value that satisfies (22).
# 5 Analysis of the Normalization Function

## 5.1 Characterization and Properties

The purpose of this section is to characterize the function $N_{Q,z}$ and the sets $A_{Q,z}$ and $B_{Q,z}$ in (12). Given a real $\delta \in [0, \infty)$, consider the Rashomon set $\mathcal{L}_z(\delta)$, which is defined as follows

$$ \mathcal{L}_z(\delta) \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) \leq \delta\}. \quad (29) $$

Consider also the real numbers $\delta_{Q,z}^*$ and $\lambda_{Q,z}^*$ defined as follows

$$ \delta_{Q,z}^* \triangleq \inf\{\delta \in [0, \infty) : Q(\mathcal{L}_z(\delta)) > 0\}, \quad (30) $$

and

$$ \lambda_{Q,z}^* \triangleq \inf \mathcal{A}_{Q,z}. \quad (31) $$

Using this notation, the following theorem introduces one of the main properties of the function $N_{Q,z}$.

**Theorem 5.1.** The function $N_{Q,z}$ in (12) is strictly increasing and continuous within the interior of $\mathcal{A}_{Q,z}$ in (12a). Furthermore, for all $\lambda \in \mathcal{A}_{Q,z}$,

$$ N_{Q,z}(\lambda) = \lambda \frac{d}{d\lambda} N_{Q,z}(\lambda) - \mathbb{R}_z(P_N), \quad (32) $$

where the probability measure $P_N \in \triangle_Q(\mathcal{M})$ satisfies for all $\theta \in \text{supp } Q$,

$$ \frac{dP_N}{dQ}(\theta) = \frac{1}{\ddot{f}\left(\frac{dP_{\Theta|Z=z}^{(Q,a)}}{dQ}(\theta)\right)} \left( \int \frac{1}{\ddot{f}\left(\frac{dP_{\Theta|Z=z}^{(Q,a)}}{dQ}(\nu)\right)} dQ(\nu) \right)^{-1} \quad (33) $$

## 5.2 Proof of Theorem 5.1

The proof is divided into two parts. The first part leverages the properties of the function $f$ under Assumption (a) in Theorem 3.1 to prove that the normalization function $N_{Q,z}: \mathcal{A}_{Q,z} \to \mathcal{B}_{Q,z}$ in (12) is strictly increasing. The second part proves the continuity of the function $N_{Q,z}$ in (12).

The first part is as follows. Given a pair $(a, b) \in \mathcal{A}_{Q,z} \times \mathcal{B}_{Q,z}$, with $\mathcal{A}_{Q,z}$ and $\mathcal{B}_{Q,z}$ in (12), assume that

$$ N_{Q,z}(a) = b. \quad (34) $$

This implies that

$$ 1 = \int \frac{dP_{\Theta|Z=z}^{(Q,b)}}{dQ}(\theta) dQ(\theta) = \int \dot{f}^{-1}\left(-\frac{1}{a}(b + \mathcal{L}_z(\theta))\right) dQ(\theta). \quad (35) $$

Note that the inverse $\dot{f}^{-1}$ exists from the fact that $f$ is strictly convex, which implies that $\dot{f}$ is a strictly increasing function. Hence, $\dot{f}^{-1}$ is also a strictly increasing function in $\mathcal{B}_{Q,z}$ [43, Theorem 5.6.9]. Moreover, from the assumption that $f$ is strictly convex and differentiable, it holds that $\dot{f}$ is continuous [44,
Proposition 5.44]. This implies that $\dot{f}^{-1}$ is continuous. From Lemma A.1 in Appendix A, the function $\dot{f}^{-1}$ is strictly increasing such that for all $b \in \mathcal{B}_{Q,z}$ and for all $\theta \in \text{supp } Q$, it holds that

$$ \dot{f}^{-1}\left(-\frac{1}{a}(b+\mathbf{L}_z(\boldsymbol{\theta}))\right) \le \dot{f}^{-1}\left(-\frac{1}{a}(b+\delta_{Q,z}^*)\right) < \infty, \quad (36) $$

with $\delta_{Q,z}^*$ defined in (30) and equality holds if and only if $\mathbf{L}_z(\boldsymbol{\theta}) = \delta_{Q,z}^*$. Then, from (36) and $\mathcal{A}_{Q,z} \subseteq (0, \infty)$, which implies $a > 0$, it follows that

$$ \int \dot{f}^{-1}\left(-\frac{1}{a}(b+\mathbf{L}_z(\boldsymbol{\theta}))\right) dQ(\boldsymbol{\theta}) < \infty. \quad (37) $$

Note that from Lemma A.5 in Appendix A and the assumption in (35) for all $(a_1, a_2) \in \mathcal{A}_{Q,z}^2$, such that $a_1 < a < a_2$, it holds that

$$ \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b}{a_1}\right) dQ(\boldsymbol{\theta}) < 1 < \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b}{a_2}\right) dQ(\boldsymbol{\theta}). \quad (38) $$

Then, for $N_{Q,z}(a_i)$, with $i \in \{1, 2\}$, to satisfy,

$$ \int \dot{f}^{-1}\left(-\frac{1}{a_i}(\mathbf{L}_z(\boldsymbol{\theta}) + N_{Q,z}(a_i))\right) dQ(\boldsymbol{\theta}) = 1, \quad (39) $$

under the assumption that $a_1 < a < a_2$, it must hold that $N_{Q,z}(a_1)$ and $N_{Q,z}(a_2)$ satisfy

$$ N_{Q,z}(a_1) < b < N_{Q,z}(a_2). \quad (40) $$

This implies that the function $N_{Q,z}$ in (12) is strictly increasing.

Similarly, from Lemma A.5 in Appendix A and the assumption in (35) for all $(b_1, b_2) \in \mathcal{B}_{Q,z}^2$, such that $b_1 < b < b_2$, it holds that

$$ \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a}\right) dQ(\boldsymbol{\theta}) > 1 > \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_2}{a}\right) dQ(\boldsymbol{\theta}). \quad (41) $$

Then, for the $a_i$, with $i \in \{1, 2\}$ to satisfy,

$$ \int \dot{f}^{-1}\left(-\frac{1}{a_i}(\mathbf{L}_z(\boldsymbol{\theta}) + b_i)\right) dQ(\boldsymbol{\theta}) = 1, \quad (42) $$

under the assumption that $b_1 < b < b_2$, it holds that $a_1$ and $a_2$ satisfy

$$ a_1 < a < a_2, \quad (43) $$

which implies that the function $N_{Q,z}$ in (12) is strictly increasing. Furthermore, from (40) and (43) the function $N_{Q,z}$ maps one to one the elements of $\mathcal{A}_{Q,z}$ into $\mathcal{B}_{Q,z}$, which implies it is bijective. Thus, the inverse $N_{Q,z}^{-1}: \mathcal{B}_{Q,z} \to \mathcal{A}_{Q,z}$ is well-defined. This completes the proof of the first part.

In the second part, the objective is to prove the continuity of the function $N_{Q,z}$. To do so, an auxiliary function is introduced and proven to be continuous.
Under the assumptions (a), (b) and (c) from Theorem 3.1, the sets $A_{Q,z}$ and $B_{Q,z}$ in (12) are non-empty and the real values: $\bar{a} = \sup A_{Q,z}$; $\underline{a} = \inf A_{Q,z}$; $\bar{b} = \sup B_{Q,z}$; and $\underline{b} = \inf B_{Q,z}$, are such that

$$ \mathcal{A} = (\underline{a}, \bar{a}) \subseteq (0, \infty), \text{ and } \mathcal{B} = (\underline{b}, \bar{b}) \subseteq \mathbb{R}. \quad (44) $$

Let the function $F: \mathcal{A} \times \mathcal{B} \to (0, \infty)$ be

$$ F(a, b) = \int \dot{f}^{-1} \left( -\frac{b + \mathbf{L}_z(\boldsymbol{\theta})}{a} \right) dQ(\boldsymbol{\theta}) - 1. \quad (45) $$

The first step is to prove that the function $F$ in (45) is continuous on the sets $\mathcal{A}$ and $\mathcal{B}$ defined in (44), respectively. This is proved by showing that $F$ always exhibits a limit in $\mathcal{A}$ and $\mathcal{B}$, and that limit is the same as the function evaluated at that value.

Then, for all $(a, b) \in \mathcal{A} \times \mathcal{B}$ and for all $\boldsymbol{\theta} \in \text{supp } Q$, the function $\dot{f}^{-1}$ satisfies the bound in (36). Hence, from [37, Corollary 24.5.1] the function $\dot{f}^{-1}$ is continuous, such that for all $(a, b) \in \mathcal{A} \times \mathcal{B}$, it holds that

$$ \lim_{a \to \lambda} \lim_{b \to \beta} \dot{f}^{-1} \left( \frac{-b - \mathbf{L}_z(\boldsymbol{\theta})}{a} \right) = \dot{f}^{-1} \left( \frac{-\beta - \mathbf{L}_z(\boldsymbol{\theta})}{\lambda} \right). \quad (46) $$

Note that the function $F$ in (45) is bounded as a consequence of (36), and thus, from the *dominated convergence theorem* [45, Theorem 1.6.9], the following limits exist and satisfy

$$ \lim_{b \to \beta} F(a, b) = F(\beta, a), \quad (47) $$

and

$$ \lim_{a \to \lambda} F(a, b) = F(\lambda, b), \quad (48) $$

which proves that the function $F$ in (45) is continuous in $\mathcal{A}$ and $\mathcal{B}$. The proof continues by noting that from the definition of $\mathcal{A}$ and $\mathcal{B}$ in (44) there exists at least one point $(\lambda, \beta) \in \mathcal{A} \times \mathcal{B}$, such that $(\lambda, \beta) \in \mathcal{A}_{Q,z} \times \mathcal{B}_{Q,z}$, which implies that

$$ F(\lambda, \beta) = 0. \quad (49) $$

Note that from (47) and (48) the function $F$ is continuous and thus the partial derivative of $F$ satisfy

$$ \frac{\partial}{\partial a} F(a, b) = \int \frac{b + \mathbf{L}_z(\boldsymbol{\theta})}{a^2} \frac{1}{\ddot{f}\left(\dot{f}^{-1}\left(-\frac{b+\mathbf{L}_z(\boldsymbol{\theta})}{a}\right)\right)} dQ(\boldsymbol{\theta}), \quad (50) $$

where (50) follows from Lemma A.2; and

$$ \frac{\partial}{\partial b} F(a, b) = \int -\frac{1}{a} \frac{1}{\ddot{f}\left(\dot{f}^{-1}\left(-\frac{b+\mathbf{L}_z(\boldsymbol{\theta})}{a}\right)\right)} dQ(\boldsymbol{\theta}), \quad (51) $$
where (51) follows from Lemma A.2 in Appendix A. Then, from the *implicit function theorem* presented in [39, Theorem 4], the function $N_{Q,z}$ exists and is unique in the open interval $\mathcal{A}$ with $\mathcal{A}$ in (44) and for all $a \in \mathcal{A}$ satisfies that

$$N_{Q,z}(a) = b, \qquad (52)$$

such that $F(a, N_{Q,z}(a)) = 0$, which completes the proof of continuity for the normalization function $N_{Q,z}$. Additionally, from [39, Theorem 4] it follows that

$$\frac{d}{da} N_{Q,z}(a) = - \left( \frac{\partial}{\partial b} F(a, N_{Q,z}(a)) \right)^{-1} \frac{\partial}{\partial a} F(a, N_{Q,z}(a)), \qquad (53)$$

$$= \frac{N_{Q,z}(a)}{a} + \int \frac{L_z(\theta)}{a} g_a(\theta) dQ(\theta), \qquad (54)$$

with the function $g_a: \mathcal{M} \to \mathbb{R}$, such that for all $\theta \in \text{supp } Q$,

$$g_a(\theta) = \frac{dP_N}{dQ}(\theta), \qquad (55)$$

with $\frac{dP_N}{dQ}$ defined in (33). Note that from the assumption that $f$ is strictly convex and twice differentiable, the derivative $\dot{f}$ is increasing, and the second derivative $\ddot{f}$ is positive for all $\theta \in \text{supp } Q$. Also, the denominator of the fraction is the integral of the reciprocal of $\ddot{f}\left(\frac{dP^{(Q,a)}(\theta|z=z)}{dQ(z)}\right)$ with respect to the measure $Q$. This term serves as a normalization constant ensuring that the resulting function is a proper probability density such that $\int g_a(\theta)dQ(\theta) = 1$. Therefore, the function $g_a$ in (55) can be interpreted as the Radon-Nikodym derivative of a new probability measure $P^{(a)}$, parametrized by the regularization factor $a$ with respect to $Q$. Specifically, if we define a measure $P^{(a)}$ such that for any set $\mathcal{A} \in \mathcal{F}_{\mathcal{M}}$,

$$P^{(a)}(\mathcal{A}) = \int_{\mathcal{A}} g_a(\theta) dQ(\theta). \qquad (56)$$

From (54) and (56), it follows that

$$N_{Q,z}(a) = a \frac{d}{da} N_{Q,z}(a) - \mathbb{R}_z(P^{(a)}), \qquad (57)$$

with $\mathbb{R}_z$ defined in (8). This completes the proof of the derivative of the normalization function. ■

The continuity and monotonicity exhibited by the function $N_{Q,z}$ allow the following characterization of the set $\mathcal{A}_{Q,z}$.

**Lemma 5.1.** The set $\mathcal{A}_{Q,z}$ in (12a) is either empty or an interval that satisfies $(\lambda_{Q,z}^*, \infty) \subseteq \mathcal{A}_{Q,z} \subseteq [\lambda_{Q,z}^*, \infty)$, with $\lambda_{Q,z}^*$ in (31) and satisfies $\lambda_{Q,z}^* \ge 0$.

*Proof:* The proof is presented in Appendix B. ■

Lemma 5.1 highlights two facts. First, the set $\mathcal{A}_{Q,z}$ is a convex subset of positive reals. Second, if there exists a solution to the optimization problem in (9) for some $\lambda > 0$, then there exists a solution to such a problem when $\lambda$ is replaced by $\bar{\lambda} \in (\lambda, \infty)$.
## 5.3 Discussion of Results

A major bottleneck in ERM-fDR solutions stems from the computational challenges associated with: (i) evaluating expectations with respect to the prior $Q$, and (ii) determining the value of $N_{Q,z}(\lambda)$ in (11) for implementing the resulting algorithms. This subsection discusses the significance of the continuity and monotonicity properties of the normalization function $N_{Q,z}$ in (12) and their practical implications for addressing challenge (ii), under the assumption that expectations with respect to the prior $Q$ are computationally tractable. To begin, note that the function $N_{Q,z}$ induces a bijection between the sets $A_{Q,z}$ and $B_{Q,z}$, as a result of the monotonicity and continuity established in Theorem 5.1. For the cases in which no explicit expression for the normalization function is available, this bijection allows the ERM-fDR solution to be computed directly by selecting a normalization factor and using its inverse to determine the corresponding regularization parameter $\lambda$. For example, this approach is used in [18, Eq (26)] for the reverse relative entropy. More importantly, in cases in which neither the normalization function $N_{Q,z}$ nor its inverse can be explicitly defined, as is the case for several f-divergences (see [26]); the monotonicity and continuity of $N_{Q,z}$ allow replacing an exhaustive search over all real values with a root-finding algorithm to approximate the solution, ensuring the existence of the root in (49) and convergence to it. Thus, such an algorithm reduces the number of evaluations of the expectation with respect to $Q$ to find such an approximation. The following Algorithm makes use of the above properties of the normalization function $N_{Q,z}$.

<caption>**Algorithm 1** Algorithm for $N_{Q,z}(\lambda)$ via Root-finding</caption>

**Input:** $L_z : \mathcal{M} \to [0, \infty)$, $Q$, $f : (0, \infty) \to \mathbb{R}$, $\lambda > 0$, $\delta_{Q,z}^*$, tolerance $\epsilon > 0$, max iterations $N_{\max}$
**Output:** $\beta$ satisfying $\left|\int \dot{f}^{-1}\left(-\frac{L_z(\theta)+\beta}{\lambda}\right)dQ(\theta)-1\right| \le \epsilon$
**Initialisation:**
1: $b_{\text{low}} \leftarrow \delta_{Q,z}^* - \lambda \dot{f}(0)$, $b_{\text{high}} \leftarrow \lambda$, $n \leftarrow 0$, $b \leftarrow \frac{1}{2}(b_{\text{low}} + b_{\text{high}})$
2: $I \leftarrow \int \dot{f}^{-1}\left(-\frac{L_z(\theta)+b}{\lambda}\right)dQ(\theta)$
**Root-Finding Process:**
3: **while** $|I - 1| > \epsilon$ **and** $n < N_{\max}$ **do**
4: &nbsp;&nbsp;&nbsp;&nbsp;**if** $I > 1$ **then**
5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$b_{\text{high}} \leftarrow b$
6: &nbsp;&nbsp;&nbsp;&nbsp;**else**
7: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$b_{\text{low}} \leftarrow b$
8: &nbsp;&nbsp;&nbsp;&nbsp;**end if**
9: &nbsp;&nbsp;&nbsp;&nbsp;$b \leftarrow \frac{1}{2}(b_{\text{low}} + b_{\text{high}})$
10: $I \leftarrow \int \dot{f}^{-1}\left(-\frac{L_z(\theta)+b}{\lambda}\right)dQ(\theta)$, $n \leftarrow n + 1$
11: **end while; return** $b$
# 6 Conclusions

This paper establishes a connection between the ERM-fDR solution and the Legendre-Fenchel transform via the dual formulation in (15). By analyzing the relationship between the primal and dual problems in (9) and (15), it is shown that the duality gap is zero. The identity arising from strong duality is leveraged to prove the continuity and monotonicity of the normalization function, enabling application of the implicit function theorem. This, in turn, yields a nonlinear ODE characterizing the normalization function. These properties are then used to design an algorithm that approximates the normalization function for a tractable reference measure and fixed regularization factor. This improves over searching across the real line when an explicit characterization is unavailable.

# References

[1] F. Daunas, I. Esnaola, and S. M. Perlaza, "A dual optimization view to empirical risk minimization with f-divergence regularization," in *Proceedings of the IEEE Information Theory Workshop (ITW)*, Sydney, Australia, Sep. 2025.

[2] V. Vapnik and A. Chervonenkis, "On a perceptron class," *Avtomatika i Telemkhanika*, vol. 25, no. 1, pp. 112-120, Feb. 1964.

[3] —, *Theory of Pattern Recognition (in Russian)*. Moscow: Nauka, Oct. 1974.

[4] V. Vapnik, "Principles of risk minimization for learning theory," *Advances in Neural Information Processing Systems*, vol. 4, pp. 831-838, Jan. 1991.

[5] V. Vapnik and L. Bottou, "Local algorithms for pattern recognition and dependencies estimation," *Neural Computation*, vol. 5, no. 6, pp. 893-909, Nov. 1993.

[6] A. Krzyzak, T. Linder, and C. Lugosi, "Nonparametric estimation and classification using radial basis function nets and empirical risk minimization," *IEEE Transactions on Neural Networks*, vol. 7, no. 2, pp. 475-487, Mar. 1996.

[7] B. Zou, L. Li, and Z. Xu, "The generalization performance of ERM algorithm with strongly mixing observations," *Machine Learning*, vol. 75, no. 3, pp. 275-295, Feb. 2009.

[8] A. R. Esposito, M. Gastpar, and I. Issa, "Generalization error bounds via Rényi-, f-divergences and maximal leakage," *IEEE Transactions on Information Theory*, vol. 67, no. 8, pp. 4986-5004, May 2021.

[9] H. Wang, M. Diaz, J. C. S. Santos Filho, and F. P. Calmon, "An information-theoretic view of generalization via Wasserstein distance," in
*Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Paris, France, Jul. 2019, pp. 577–581.

[10] A. S. Dalalyan and A. G. Karagulyan, “User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient,” arXiv preprint arXiv:1710.00095, Feb. 2024.

[11] W. Li, N. Klein, B. Gifford, E. Sklute, C. Legett, and S. Clegg, “Regularization via f-divergence: An application to multi-oxide spectroscopic analysis,” arXiv preprint arXiv:2502.03755, Feb. 2025.

[12] F. Peng, M. Zhang, and M. Tang, “An information-theoretic analysis for federated learning under concept drift,” arXiv preprint arXiv:2506.21036, Jun. 2025.

[13] F. Futami and T. Iwata, “Information-theoretic analysis of test data sensitivity in uncertainty,” arXiv preprint arXiv:2307.12456, Jul. 2023.

[14] Z. Huang and S. Becker, “Stochastic gradient Langevin dynamics with variance reduction,” arXiv preprint arXiv:2102.06759, Jul. 2021.

[15] M. Raginsky, A. Rakhlin, M. Tsao, Y. Wu, and A. Xu, “Information-theoretic analysis of stability and bias of learning algorithms,” in *Proceedings of the IEEE Information Theory Workshop (ITW)*, Cambridge, UK, Sep. 2016, pp. 26–30.

[16] D. Russo and J. Zou, “How much does your data exploration overfit? Controlling bias via information usage,” *IEEE Transactions on Information Theory*, vol. 66, no. 1, pp. 302–323, Jan. 2019.

[17] S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, “Empirical risk minimization with relative entropy regularization: Optimality and sensitivity analysis,” in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Espoo, Finland, Jul. 2022, pp. 684–689.

[18] F. Daunas, I. Esnaola, S. M. Perlaza, and H. V. Poor, “Asymmetry of the relative entropy in the regularization of empirical risk minimization,” *IEEE Transactions on Information Theory*, vol. 71, no. 8, pp. 6198–6226, Aug. 2025.

[19] S. M. Perlaza, G. Bisson, I. Esnaola, A. Jean-Marie, and S. Rini, “Empirical risk minimization with relative entropy regularization,” *IEEE Transactions on Information Theory*, vol. 70, no. 7, pp. 5122–5161, Jul. 2024.

[20] G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell, “An exact characterization of the generalization error for the Gibbs algorithm,” *Advances in Neural Information Processing Systems*, vol. 34, pp. 8106–8118, Dec. 2021.

[21] Y. Bu, S. Zou, and V. V. Veeravalli, “Tightening mutual information-based bounds on generalization error,” *IEEE Journal on Selected Areas in Information Theory*, vol. 1, no. 1, pp. 121–130, Jan. 2020.
[22] X. Zou, S. M. Perlaza, I. Esnaola, E. Altman, and H. V. Poor, "The worst-case data-generating probability measure in statistical learning," *IEEE Journal on Selected Areas in Information Theory*, vol. 5, no. 1, pp. 175-189, Apr. 2024.

[23] M. Teboulle, "Entropic proximal mappings with applications to nonlinear programming," *Mathematics of Operations Research*, vol. 17, no. 3, pp. 670-690, Aug. 1992.

[24] A. Beck and M. Teboulle, "Mirror descent and nonlinear projected sub-gradient methods for convex optimization," *Operations Research Letters*, vol. 31, no. 3, pp. 167-175, Jan. 2003.

[25] P. Alquier, "Non-exponentially weighted aggregation: Regret bounds for unbounded loss functions," in *Proceedings of the 38th International Conference on Machine Learning (ICML)*, vol. 139, Jul. 2021, pp. 207-218.

[26] F. Daunas, I. Esnaola, S. M. Perlaza, and H. V. Poor, "Equivalence of empirical risk minimization to regularization on the family of f-divergences," in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Athens, Greece, Jul. 2024.

[27] J. Wei and Y. Liu, "When optimizing f-divergence is robust with label noise," arXiv preprint arXiv:2011.03687, Aug. 2021.

[28] Z. Liu, B. P. Van Parys, and H. Lam, "Smoothed f-divergence distributionally robust optimization," arXiv preprint arXiv:2306.14041, Oct. 2023.

[29] H. Namkoong and J. C. Duchi, "Stochastic gradient methods for distributionally robust optimization with f-divergences," *Advances in Neural Information Processing Systems*, vol. 26, pp. 1476-1485, Dec. 2016.

[30] Z. Hu and L. J. Hong, "Kullback-Leibler divergence constrained distributionally robust optimization," *Optimization Online*, vol. 1, no. 2, Nov. 2012.

[31] W. K. Hastings, "Monte Carlo sampling methods using Markov chains and their applications," *Biometrika*, vol. 57, no. 1, pp. 97-109, May 1970.

[32] W. R. Gilks and P. Wild, "Adaptive rejection sampling for Gibbs sampling," *Journal of the Royal Statistical Society: Series C (Applied Statistics)*, vol. 41, no. 2, pp. 337-348, Jun. 1992.

[33] S. M. Perlaza, I. Esnaola, G. Bisson, and H. V. Poor, "On the validation of Gibbs algorithms: Training datasets, test datasets and their aggregation," in *Proceedings of the IEEE International Symposium on Information Theory (ISIT)*, Taipei, Taiwan, Jun. 2023.

[34] A. Bulatov and M. Grohe, "The complexity of partition functions," *Theoretical Computer Science*, vol. 348, no. 2, pp. 148-186, Sep. 2005.

[35] A. A. Bulatov, "The complexity of the counting constraint satisfaction problem," *Journal of the ACM (JACM)*, vol. 60, no. 5, pp. 1-41, Sep. 2014.
[36] C. McQuillan, “The computational complexity of approximation of partition functions,” PhD thesis, University of Liverpool, Liverpool, UK, Jun. 2013.

[37] R. T. Rockafellar, *Conjugate Convex Functions in Optimal Control and the Calculus of Variations*, 2nd ed. Princeton, NJ, USA: Princeton University Press, 1970.

[38] S. Boyd, S. P. Boyd, and L. Vandenberghe, *Convex optimization*, 1st ed. Cambridge, UK: Cambridge University Press, 2004.

[39] O. de Oliveira, “The Implicit and Inverse Function Theorems: Easy Proofs,” *Real Analysis Exchange*, vol. 39, no. 1, pp. 207–218, 2013.

[40] I. Csiszár, “Information-type measures of difference of probability distributions and indirect observation,” *Studia Scientiarum Mathematicarum Hungarica*, vol. 2, no. 1, pp. 299–318, Jun. 1967.

[41] D. G. Luenberger, “Observing the state of a linear system,” *IEEE Transactions on Military Electronics*, vol. 8, no. 2, pp. 74–80, Apr. 1964.

[42] ——, *Optimization by Vector Space Methods*, 1st ed. New York, NY, USA: Wiley, 1997.

[43] R. G. Bartle and D. R. Sherbert, *Introduction to Real Analysis*, 3rd ed. New York, NY, USA: Wiley New York, 2000.

[44] J. Douchet, *Analyse : Recueil d'Exercices et Aide-Mémoire*, 3rd ed. Lausanne, Switzerland: PPUR, 2010, vol. 1.

[45] R. B. Ash and C. A. Doleans-Dade, *Probability and Measure Theory*, 2nd ed. Burlington, MA, USA: Academic Press, 2000.
# Appendices

## A Preliminaries

**Lemma A.1.** Given a strictly convex and differentiable function $f: \mathcal{I} \to \mathbb{R}$, the inverse of the derivative of $f$, denoted by the function $\dot{f}^{-1}: \mathcal{J} \to \mathcal{I}$, exists and is strictly increasing.

*Proof:* Under the assumption that the function $f: \mathcal{I} \to \mathbb{R}$ is strictly convex and differentiable, it follows from the definition of strict convexity (see [42, Definition 7.8]) that its derivative $\dot{f}: \mathcal{I} \to \mathcal{J}$ is strictly increasing and continuous. By the continuous inverse theorem (see [43, Theorem 5.6]), the inverse function $\dot{f}^{-1}: \mathcal{J} \to \mathcal{I}$ exists and is also strictly increasing and continuous. This completes the proof. ■

**Lemma A.2.** Given a strictly convex and twice differentiable function $f: \mathcal{I} \to \mathbb{R}$, and a differentiable function $h: \mathcal{I} \to \mathcal{J}$, for all $u \in \mathcal{I}$ it holds that

$$ \frac{d}{du} \dot{f}^{-1}(h(u)) = \frac{\dot{h}(u)}{\ddot{f}(\dot{f}^{-1}(h(u)))}. \qquad (58) $$

*Proof:* Let the function $g: \mathcal{I} \to \mathcal{J}$ be defined for all $u \in \mathcal{I}$ by

$$ g(u) = \dot{f}^{-1}(h(u)) \qquad (59) $$

By the definition of the inverse function, it follows that

$$ \dot{f}(g(u)) = h(u). \qquad (60) $$

Differentiating (60) with respect to $u$ yields

$$ \frac{d}{du} \dot{f}(g(u)) = \ddot{f}(g(u)) \dot{g}(u) \qquad (61) $$

$$ = \dot{h}(u). \qquad (62) $$

From (61), the derivative of the function $g$ in (60) is given by

$$ \dot{g}(u) = \frac{\dot{h}(u)}{\ddot{f}(g(u))} \qquad (63) $$

$$ = \frac{\dot{h}(u)}{\ddot{f}(\dot{f}^{-1}(h(u)))}, \qquad (64) $$

where (64) follows from (59). Hence, (59) and (64) imply that

$$ \frac{d}{du} \dot{f}^{-1}(h(u)) = \frac{\dot{h}(u)}{\ddot{f}(\dot{f}^{-1}(h(u)))}. \qquad (65) $$

This completes the proof. ■

<figure><img src="image_4.png" /></figure>
**Lemma A.3.** The Legendre-Fenchel transform of a strictly convex and differentiable function $f$, satisfies for all $v \in \mathcal{J}$, with $\mathcal{J}$ in (14),

$$f^*(v) = v\dot{f}^*(v) - f(\dot{f}^*(v)). \qquad (66)$$

*Proof:* From the Legendre-Fenchel transform in Definition 4.1, it holds that for all $v \in \mathcal{J}$, with $\mathcal{J}$ in (14),

$$f^*(v) = \sup_{u \in \mathcal{I}} (vu - f(u)), \qquad (67)$$

which implies that, for all $u \in \mathcal{I}$,

$$f^*(v) \geq vu - f(u). \qquad (68)$$

Note that (68) holds with equality if $u$ achieves the supremum in (67), which will be denoted by $u_v$. In other words, $u_v$ is the maximizing argument

$$u_v = \arg\max_{u \in \mathcal{I}} uv - f(u), \qquad (69)$$

such that

$$f^*(v) = vu_v - f(u_v). \qquad (70)$$

Note that under Assumption (a) in Theorem 3.1, the maximization problem

$$\max_{u \in \mathcal{I}} uv - f(u), \qquad (71)$$

is concave and possesses a unique solution that satisfies

$$\left. \frac{d}{du}(uv - f(u)) \right|_{u=u_v} = v - \dot{f}(u_v) = 0, \qquad (72)$$

and thus,

$$u_v = \dot{f}^{-1}(v) \qquad (73)$$

$$= \dot{f}^*(v), \qquad (74)$$

where (74) follows from [37, Corollary 23.5.1]. Hence, substituting (74) into (70) yields that for all $v \in \mathcal{J}$,

$$f^*(v) = v\dot{f}^*(v) - f(\dot{f}^*(v)), \qquad (75)$$

which completes the proof.

<figure><img src="image_5.png" /></figure>

**Theorem A.1.** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (11) satisfies

$$\begin{aligned} & \mathrm{R}_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) + \lambda \mathrm{D}_f \left( P_{\Theta|Z=z}^{(Q,\lambda)} \| Q \right) \\ & = -\lambda \int f^* \left( - \frac{\mathrm{L}_z(\theta) + N_{Q,z}(\lambda)}{\lambda} \right) dQ(\theta) - N_{Q,z}(\lambda), \end{aligned} \qquad (76)$$
and

$$
\begin{align}
& R_z(Q) + \lambda \int f \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) \right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \nonumber \\
&= -\lambda \int f^* \left( -\frac{L_z(\boldsymbol{\theta}) + N_{Q,z}(\lambda)}{\lambda} \right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}(\boldsymbol{\theta})} dQ(\boldsymbol{\theta}) \nonumber \\
&\quad - N_{Q,z}(\lambda), \tag{77}
\end{align}
$$

where $f^*$ is the Legendre-Fenchel transform of $f$ (see Definition 4.1), the function $N_{Q,z}$ is defined in (12), and the functional $R_z$ is defined in (8).

*Proof:* The Legendre-Fenchel transform of a strictly convex function $f: \mathcal{I} \to \mathbb{R}$ satisfies

$$f^*(t) \triangleq \sup_{s \in \mathcal{I}} (ts - f(s)). \quad (78)$$

From [37, Theorem 23.5] if $f$ is strictly convex then maximizing argument of the convex conjugate $f^*$ satisfies

$$f^*(t) = t \frac{d}{dt} f^*(t) - f\left(\frac{d}{dt} f^*(t)\right). \quad (79)$$

Furthermore, from [37, Corollary 23.5.1] the function $\dot{f}^{-1}$ is the derivative of the convex conjugate of $f$ in (79), which implies that Differential Equations

$$f^*(t) = t \dot{f}^{-1}(t) - f(\dot{f}^{-1}(t)). \quad (80)$$

From Theorem 3.1, let $t = -\frac{L_z(\boldsymbol{\theta})+\beta}{\lambda}$ in the optimization problem in (9), then it holds that for all $\boldsymbol{\theta} \in \text{supp } Q$,

$$
\begin{align}
& f^* \left( - \frac{L_z(\boldsymbol{\theta}) + \beta}{\lambda} \right) \nonumber \\
&= - \frac{L_z(\boldsymbol{\theta}) + \beta}{\lambda} \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) - f \left( \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) \right). \tag{81}
\end{align}
$$

Taking the integral of (81) with respect to the reference measure $Q$, yields

$$
\begin{align}
& \int f^* \left( - \frac{L_z(\boldsymbol{\theta}) + \beta}{\lambda} \right) dQ(\boldsymbol{\theta}) \nonumber \\
&= \int - \frac{L_z(\boldsymbol{\theta}) + \beta}{\lambda} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \nonumber \\
&\quad - \int f \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) \right) dQ(\boldsymbol{\theta}) \tag{82}
\end{align}
$$
$$ = -\frac{1}{\lambda} (R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) + \beta) - D_f(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q). \quad (83) $$

Arranging (83) results in

$$ \begin{aligned} & R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) + \lambda D_f(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) \\ &= -\lambda \int f^* \left(-\frac{L_z(\theta) + \beta}{\lambda}\right) dQ(\theta) - \beta, \end{aligned} \quad (84) $$

which completes the proof.

The second part of the proof is as follows. From Corollary A.4, equality (81) can be rewritten as

$$ \begin{aligned} \frac{L_z(\theta) + \beta}{\lambda} = & -f^*\left(-\frac{L_z(\theta) + \beta}{\lambda}\right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) \\ & -\frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) f\left(\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)\right). \end{aligned} \quad (85) $$

Taking the integral of (85) with respect to the reference measure $Q$, yields

$$ \begin{aligned} & \frac{1}{\lambda} R_z(Q) + \frac{\beta}{\lambda} \\ &= -\int f^*\left(-\frac{L_z(\theta) + \beta}{\lambda}\right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dQ(\theta) \\ & \quad -\int \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) f\left(\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)\right) dQ(\theta). \end{aligned} \quad (86) $$

Arranging (86) results in

$$ \begin{aligned} & R_z(Q) + \lambda \int \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) f\left(\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)\right) dQ(\theta) \\ &= -\lambda \int f^*\left(-\frac{L_z(\theta) + \beta}{\lambda}\right) \frac{dQ}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dQ(\theta) - \beta, \end{aligned} \quad (87) $$

which completes the proof.

<figure><img src="image_6.png" /></figure>

In Theorem 3.1, the condition that $\beta$ in (10b) satisfies (b) in (10a), leads to observing that for all $\theta \in \text{supp } Q$,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) > 0, \quad (88) $$

which leads to the following corollary.
**Corollary A.4.** Under Assumptions (a) and (b), the probability measures $Q$ and $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (11) are mutually absolutely continuous.

**Lemma A.5.** Let $Q$ be a probability measure on a measurable space $(M, \mathcal{F})$. Consider the pair $(a_1, b_1) \in (0, \infty) \times \mathbb{R}$ and let $g: \mathbb{R} \to (0, \infty)$ be a strictly increasing function such that

$$ \int g\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_1}{a_1}\right) dQ(\boldsymbol{\theta}) < \infty, \qquad (89) $$

where $\mathbf{L}_z$ is defined in (7). Then the following monotonicity properties hold: For any $a_2 \in (0, \infty)$ with $a_1 < a_2$,

$$ \int g\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_1}{a_1}\right) dQ(\boldsymbol{\theta}) < \int g\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_1}{a_2}\right) dQ(\boldsymbol{\theta}). \qquad (90) $$

For any $b_2 \in \mathbb{R}$ with $b_1 < b_2$:

$$ \int g\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_1}{a_1}\right) dQ(\boldsymbol{\theta}) > \int g\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_2}{a_1}\right) dQ(\boldsymbol{\theta}). \qquad (91) $$

*Proof:* For all $(a_1, a_2) \in (0, \infty)^2$, such that $a_1 < a_2$, it holds that for all $\boldsymbol{\theta} \in \text{supp} Q$,

$$ -\frac{1}{a_1}(\mathbf{L}_z(\boldsymbol{\theta}) + b_1) < -\frac{1}{a_2}(\mathbf{L}_z(\boldsymbol{\theta}) + b_1), \qquad (92) $$

which from the assumption that $g$ is monotonic implies that

$$ \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a_1}\right) < \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a_2}\right). \qquad (93) $$

From (35) and (93), it holds that

$$ \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a_1}\right) dQ(\boldsymbol{\theta}) < \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a_2}\right) dQ(\boldsymbol{\theta}). \qquad (94) $$

For all $(b_1, b_2) \in \mathbb{R}^2$, such that $b_1 < b_2$, it holds that for all $\boldsymbol{\theta} \in \text{supp} Q$,

$$ -\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_1}{a_1} > -\frac{\mathbf{L}_z(\boldsymbol{\theta}) + b_2}{a_1}, \qquad (95) $$

which from the assumption that $g$ is monotonic implies that

$$ \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a_1}\right) > \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_2}{a_1}\right). \qquad (96) $$

From (35) and (96), it holds that

$$ \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_1}{a}\right) dQ(\boldsymbol{\theta}) > \int \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})+b_2}{a}\right) dQ(\boldsymbol{\theta}). \qquad (97) $$

This completes the proof.

<figure><img src="image_7.png" /></figure>
## B Proof of Lemma 5.1

*Proof:* Given a reference measure $Q$, a dataset $z$, a strictly convex and differentiable function $f$ that induces an $f$-divergence and the empirical risk function $L_z$ in (7), under the assumption that there exists a $\lambda \in (0, \infty)$, such that the optimization problem (9) has a solution, the proof is concerned with characterizing the set of all regularization factors $\lambda$ for which a solution exists. This set of regularization factors is denoted by $A_{Q,z}$, where $A_{Q,z} \subseteq (0, \infty)$. The proof is divided into three parts. In the first part, the Legendre-Fenchel transform of $f$ is connected to the Radon-Nikodym derivative of the solution to the optimization problem in (9) presented in Theorem 3.1. In the second part, the strictly increasing property of the Radon-Nikodym derivative, obtained from the connection established with the Legendre-Fenchel transform of $f$, is used to evaluate the real values of $\lambda$ under which assumption (10a) holds. In the third part, the strictly increasing property is used to evaluate the real values of $\lambda$ under which assumption (10b) holds.

The first part is as follows. The Legendre-Fenchel transform of $f$ is defined as

$$f^*(t) \triangleq \sup_{s \in \mathcal{I}} (ts - f(s)), \qquad (98)$$

where $f^*: \mathcal{J} \to \mathbb{R}$. From Assumption (a) and [37, Theorem 23.5] the Legendre-Fenchel transform $f^*$ in (98) satisfies

$$f^*(t) = t \frac{d}{dt} f^*(t) - f\left(\frac{d}{dt} f^*(t)\right). \qquad (99)$$

Furthermore, from [37, Corollary 23.5.1], the function $\frac{d}{dt} f^*: \mathcal{J} \to \mathcal{I}$ satisfies

$$\frac{d}{dt} f^*(t) = \left( \frac{df}{dt} \right)^{-1}(t), \qquad (100)$$

which is the functional inverse of the derivative of $f$, denoted by $\dot{f}^{-1}$ for simplicity. Note that, given the assumption that $f$ is strictly convex and induces an $f$-divergence, it follows from [37, Theorem 12.2] that the function $f^*$ in (98) is also strictly convex. From the strict convexity of $f^*$, it follows from Lemma A.1 that $\dot{f}^{-1}$ in (100) is strictly increasing. Furthermore, from [37, Corollary 26.3.1] $f^*$ in (98) is bijective with $\frac{df}{ds}: \mathcal{I} \to \mathcal{J}$, which completes the first part of the proof.

The second part is as follows. Evaluating the real values of $\lambda$ under which assumption in (10a) holds requires to show that the function $\frac{dP^{(Q,\lambda)}}{dQ}$ belongs to the set of nonnegative measurable functions. From the $f$-divergence in Definition 2.1 and the fact that $\dot{f}^{-1}$ strictly increasing and bijective, the proof follows by showing that the limit

$$\lim_{x \to 0^+} \dot{f}(x) = t_0, \qquad (101)$$
satisfies for all $\theta \in \text{supp } Q$,

$$ -\frac{\mathbf{L}_z(\theta) + \beta}{\lambda} > t_0. \qquad (102) $$

Note that (102) is sufficient from the fact that the monotonicity of $\dot{f}^{-1}$ implies that for all $t > t_0$,

$$ \dot{f}^{-1}(t) > 0. \qquad (103) $$

To evaluate the real values of $\lambda$ under which assumption in (10a) holds, three cases must be considered for the limit in (101).

**Case 1:** Assume that

$$ \lim_{x \to 0^+} \dot{f}(x) = \infty. \qquad (104) $$

Under the above assumption, for all $\theta \in \text{supp } Q$,

$$ -\frac{\mathbf{L}_z(\theta) + \beta}{\lambda} < \infty, \qquad (105) $$

which implies that

$$ \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\theta) + \beta}{\lambda}\right) < 0. \qquad (106) $$

Hence, Assumption (b) in Theorem 3.1 is not satisfied and nothing can be stated about the solution.

**Case 2:** Assume that

$$ \lim_{x \to 0^+} \dot{f}(x) = a, \qquad (107) $$

where $a \in \mathbb{R}$. Under the above assumption, consider the set

$$ \mathcal{D} = \{\theta \in \text{supp } Q : -\mathbf{L}_z(\theta) < a\lambda + \beta\}. \qquad (108) $$

On one hand, note that if the function $\mathbf{L}_z$ in (7) is unbounded in $\text{supp } Q$, from (107) the set $\mathcal{D}$ in (108) is nonegible and measurable, such that for all $\theta \in \mathcal{D}$,

$$ -\frac{\mathbf{L}_z(\theta) + \beta}{\lambda} < a, \qquad (109) $$

which implies that

$$ \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\theta) + \beta}{\lambda}\right) < 0. \qquad (110) $$

Hence, Assumption (b) in Theorem 3.1 is not satisfied and nothing can be stated about the solution. On the other hand, if the function $\mathbf{L}_z$ in (7) is bounded in $\text{supp } Q$, such that

$$ M = \sup_{\theta \in \text{supp } Q} \mathbf{L}_z(\theta). \qquad (111) $$
Then, there exists a $\lambda_{Q,z} \in (0, \infty)$ such that

$$ -M = a\lambda_{Q,z} + \beta. \qquad (112) $$

From (112) for all $\lambda > \lambda_{Q,z}$, it holds that for all $\theta \in \text{supp } Q$,

$$ \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\theta) + \beta}{\lambda}\right) > 0. \qquad (113) $$

From (113), consider the following conditions: If there exists a model $\bar{\theta} \in \text{supp } Q$ such that $\mathbf{L}_z(\bar{\theta}) = M$, where $M$ is defined in (111), then the set of regularization factors $\lambda$ for which the function $\frac{dP^{(Q,\lambda)}_{\Theta|Z=z}}{dQ}$ is nonnegative is $[\lambda_{Q,z}, \infty)$. Alternatively, if for all models $\theta \in \text{supp } Q$, it holds that $\mathbf{L}_z(\theta) < M$, where $M$ is defined in (111), then the set of regularization factors $\lambda$ for which the function $\frac{dP^{(Q,\lambda)}_{\Theta|Z=z}}{dQ}$ is nonnegative is $(\lambda_{Q,z}, \infty)$.

**Case 3:** Assume that

$$ \lim_{x \to 0^+} \dot{f}(x) = -\infty. \qquad (114) $$

Under the above assumption, for all $\theta \in \text{supp } Q$,

$$ -\frac{\mathbf{L}_z(\theta) + \beta}{\lambda} > -\infty, \qquad (115) $$

which implies that

$$ \dot{f}^{-1}\left(-\frac{\mathbf{L}_z(\theta) + \beta}{\lambda}\right) > 0. \qquad (116) $$

Hence, for all $\lambda \in (0, \infty)$ Assumption (b) in Theorem 3.1 is satisfied such that the nonnegativity of the function $\frac{dP^{(Q,\lambda)}_{\Theta|Z=z}}{dQ}$ is guaranteed. This completes the second part of the proof.

The third part is as follows. Evaluating the values $\lambda$ under which assumption (10b) holds requires showing that there exists a real value $\beta \in \mathbb{R}$ such that the integral of $\frac{dP^{(Q,\lambda)}_{\Theta|Z=z}}{dQ}$ with respect to $Q$ is one. From Theorem 5.1 the monotonicity of the normalization function $N_{Q,z}$ in (12), there is a minimum regularization factor $\lambda_{Q,z}^*$ defined in (31). Furthermore, from Theorem 5.1 the continuity of the function $N_{Q,z}$ implies that for all $\lambda \in (\lambda_{Q,z}^*, \infty)$, there exists a unique $\beta \in \mathcal{B}_{Q,z}$ such that Assumption (b) of Theorem 3.1 is satisfied. From Theorem 5.1, it holds that

$$ \lim_{\lambda \to \lambda_{Q,z}^* +} N_{Q,z}(\lambda) = N_{Q,z}(\lambda_{Q,z}^*), \qquad (117) $$

with the function $N_{Q,z}$ defined in (12) and the limit from the right is well-defined from the fact that the set $\mathcal{A}_{Q,z}$ is convex. To determine whether the infimum in (31) belongs to the set $\mathcal{A}_{Q,z}$ two cases are considered.
**Case 1:** Assume that $\beta > N_{Q,z}(\lambda_{Q,z}^*)$, such that

$$ \int \dot{f}^{-1} \left( - \frac{\beta + L_z(\theta)}{\lambda_{Q,z}^*} \right) dQ(\theta) = \infty. \quad (118) $$

Notice that from (118) for all $\beta_1 \in [N_{Q,z}(\lambda_{Q,z}^*), \beta)$ and for all $\theta \in \text{supp } Q$, it holds that

$$ \dot{f}^{-1} \left( - \frac{\beta_1 + L_z(\theta)}{\lambda_{Q,z}^*} \right) > \dot{f}^{-1} \left( - \frac{\beta + L_z(\theta)}{\lambda_{Q,z}^*} \right). \quad (119) $$

Hence, under the above assumption, $N_{Q,z}(\lambda_{Q,z}^*) \notin \mathcal{B}_{Q,z}$ which implies that the set of all regularization $\mathcal{A}_{Q,z}$ in (12) that satisfy assumption (10b) is $\mathcal{A}_{Q,z} = (\lambda_{Q,z}^*, \infty)$.

**Case 2:** Assume that $\beta > N_{Q,z}(\lambda_{Q,z}^*)$, such that

$$ \int \dot{f}^{-1} \left( - \frac{\beta + L_z(\theta)}{\lambda_{Q,z}^*} \right) dQ(\theta) < \infty. \quad (120) $$

From the monotonicity of the solution in part one and continuity of the function $N_{Q,z}$ in (12) from Theorem 5.1, there exists a $\beta_{Q,z}^* \in \mathcal{B}_{Q,z}$ such that $N_{Q,z}(\lambda_{Q,z}^*) = \beta_{Q,z}^*$, which implies that the set of all regularization factors $\mathcal{A}_{Q,z} = (\lambda_{Q,z}^*, \infty)$.

Finally, from parts two and three of the proof, the set $\mathcal{A}_{Q,z}$ is a convex set such that the regularization factors for which the assumptions of Theorem 3.1 hold and are given by

$$ \mathcal{A}_{Q,z} = \begin{cases} [\lambda_{Q,z}^*, \infty) & \text{if } \int \dot{f}^{-1} \left( - \frac{\beta + L_z(\theta)}{\lambda_{Q,z}^*} \right) dQ(\theta) < \infty, \\ (\lambda_{Q,z}^*, \infty) & \text{otherwise,} \end{cases} \quad (121) $$

where $\beta > \lim_{\lambda \to \lambda_{Q,z}^*} N_{Q,z}(\lambda)$. This completes the proof.

<figure><img src="image_8.png" /></figure>
informatics mathematics
Inria

RESEARCH CENTRE
SOPHIA ANTIPOLIS – MÉDITERRANÉE

2004 route des Lucioles - BP 93
06902 Sophia Antipolis Cedex

Publisher
Inria
Domaine de Voluceau -
Rocquencourt
BP 105 - 78153 Le Chesnay
Cedex
inria.fr