# Empirical Risk Minimization with Relative Entropy Regularization

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini

► To cite this version:

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Relative Entropy Regularization. IEEE Transactions on Information Theory, 2024, 70 (7), pp.5122-5161. 10.1109/TIT.2024.3365728. hal-03849748v4

HAL Id: hal-03849748

https://hal.science/hal-03849748v4

Submitted on 8 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Empirical Risk Minimization with Relative Entropy Regularization

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini

► To cite this version:

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Relative Entropy Regularization. IEEE Transactions on Information Theory, 2024, 70 (7), pp.5122-5161. 10.1109/TIT.2024.3365728. hal-03849748v4

HAL Id: hal-03849748

https://hal.science/hal-03849748v4

Submitted on 8 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Empirical Risk Minimization with Relative Entropy Regularization

Samir M. Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, and Stefano Rini.

*Abstract*—The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a σ-finite measure, and not necessarily a probability measure. Under this assumption, which leads to a generalization of the ERM-RER problem allowing a larger degree of flexibility for incorporating prior knowledge, numerous relevant properties are stated. Among these properties, the solution to this problem, if it exists, is shown to be a unique probability measure, mutually absolutely continuous with the reference measure. Such a solution exhibits a probably-approximately-correct guarantee for the ERM problem independently of whether the latter possesses a solution. For a fixed dataset and under a specific condition, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the ERM-RER problem. The generalization capabilities of the solution to the ERM-RER problem (the Gibbs algorithm) are studied via the sensitivity of the expected empirical risk to deviations from such a solution towards alternative probability measures. Finally, an interesting connection between sensitivity, generalization error, and lautum information is established.

*Index Terms*—Supervised Learning, PAC-Learning, Relative Entropy Regularization, Empirical Risk Minimization, Gibbs Measure, Gibbs Algorithm, Generalization, and Sensitivity.

## I. INTRODUCTION

IN statistical machine learning, the problem of empirical risk minimization (ERM) with relative entropy regularization (ERM-RER) has been the workhorse for building probability measures on the set of models, without any additional assumption on the statistical description of the datasets. See for instance [3]–[5] and [6]. Instead of additional statistical assumptions on the datasets, which are typical in Bayesian methods [7], relative entropy regularization requires a reference probability measure on the set of models, which is external to the ERM problem. Often, such a reference measure represents prior knowledge or side information and is chosen for guiding the search of models towards those inducing low

empirical risks with high probability over seen and unseen datasets. From this perspective, the reference measure can be seen as an additional degree of freedom to improve the generalization capabilities of machine learning algorithms based on ERM-RER, e.g. Gibbs algorithms [5], [8]–[17] and [18]. This new degree of freedom is one of the main motivations for regularizing the ERM problem using relative entropy, or more generally, any *f*-divergence regularization, as discussed in [19], [20] and [21]. Beyond probability measures, as shown in this paper, the reference measure can be any *σ*-finite measure with arbitrary support. The flexibility introduced by this generalization becomes particularly relevant for the case in which priors are available in the form of probability distributions that can be evaluated up to some normalizing factor, cf. [22], or cannot be represented by probability distributions, e.g., equal preferences among elements of infinite countable sets. For some specific choices of *σ*-finite reference measures, the ERM-RER boils down to particular cases of special interest: (i) the information-risk minimization problem presented in [23]; (ii) the ERM with differential entropy regularization (ERM-DiffER); and (iii) the ERM with discrete entropy regularization (ERM-DisER). See for instance [24], [25] and references therein. From this perspective, the proposed ERM-RER formulation yields a unified mathematical framework that comprises a large class of problems.

When the reference measure is a probability measure, the solution to the ERM-RER problem is known to be unique and correspond to a Gibbs probability measure. Such a Gibbs probability measure has been studied using measure theoretic and information theoretic notions in [9], [23], [26]–[33]; statistical physics in [3]; PAC (Probably Approximately Correct)-Bayesian learning theory in [34]–[37]; and proved to be of particular interest in classification problems in [5], [12], [20], [24], [38], [39] and [40]. In the general case in which the reference is a *σ*-finite measure, a solution to the ERM-RER problem does not always exist. Nonetheless, if it exists, it is shown to be a unique Gibbs probability despite the fact that its partition function is defined with respect to a *σ*-finite measure. The condition for the existence is mild and is always satisfied when the reference measure is a probability measure, as highlighted above. Interestingly, such a solution is mutually absolutely continuous with the reference measure and most of the properties known for the classical ERM-RER problem are shown to hold in the most general case. For instance, under certain conditions, the empirical risk observed when models are sampled from the ERM-RER-optimal probability measure is a sub-Gaussian random variable that exhibits a PAC guarantee for the ERM problem without regularization.

Samir M. Perlaza is with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis 06902, France; also with the ECE Dept. at Princeton University, Princeton N.J. 08544, USA; and also with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia.

Gaetan Bisson is with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia.

Iñaki Esnaola is with the ACSE Dept. at The University of Sheffield, Sheffield S1 3JD, UK; and also with the ECE Dept. at Princeton University, Princeton N.J. 08544, USA.

Alain Jean-Marie is with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis 06902, France.

Stefano Rini is with the ECE Dept. at the National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan 30010, ROC.

This work was presented in part at the IEEE International Symposium on Information Theory (ISIT), Espoo, Finland, 2022 in [1]; and appears as an INRIA technical report in [2].
# Empirical Risk Minimization with Relative Entropy Regularization

Samir M. Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, and Stefano Rini.

*Abstract*—The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a σ-finite measure, and not necessarily a probability measure. Under this assumption, which leads to a generalization of the ERM-RER problem allowing a larger degree of flexibility for incorporating prior knowledge, numerous relevant properties are stated. Among these properties, the solution to this problem, if it exists, is shown to be a unique probability measure, mutually absolutely continuous with the reference measure. Such a solution exhibits a probably-approximately-correct guarantee for the ERM problem independently of whether the latter possesses a solution. For a fixed dataset and under a specific condition, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the ERM-RER problem. The generalization capabilities of the solution to the ERM-RER problem (the Gibbs algorithm) are studied via the sensitivity of the expected empirical risk to deviations from such a solution towards alternative probability measures. Finally, an interesting connection between sensitivity, generalization error, and lautum information is established.

*Index Terms*—Supervised Learning, PAC-Learning, Relative Entropy Regularization, Empirical Risk Minimization, Gibbs Measure, Gibbs Algorithm, Generalization, and Sensitivity.

## I. INTRODUCTION

IN statistical machine learning, the problem of empirical risk minimization (ERM) with relative entropy regularization (ERM-RER) has been the workhorse for building probability measures on the set of models, without any additional assumption on the statistical description of the datasets. See for instance [3]–[5] and [6]. Instead of additional statistical assumptions on the datasets, which are typical in Bayesian methods [7], relative entropy regularization requires a reference probability measure on the set of models, which is external to the ERM problem. Often, such a reference measure represents prior knowledge or side information and is chosen for guiding the search of models towards those inducing low

empirical risks with high probability over seen and unseen datasets. From this perspective, the reference measure can be seen as an additional degree of freedom to improve the generalization capabilities of machine learning algorithms based on ERM-RER, e.g. Gibbs algorithms [5], [8]–[17] and [18]. This new degree of freedom is one of the main motivations for regularizing the ERM problem using relative entropy, or more generally, any f-divergence regularization, as discussed in [19], [20] and [21]. Beyond probability measures, as shown in this paper, the reference measure can be any σ-finite measure with arbitrary support. The flexibility introduced by this generalization becomes particularly relevant for the case in which priors are available in the form of probability distributions that can be evaluated up to some normalizing factor, cf. [22], or cannot be represented by probability distributions, e.g., equal preferences among elements of infinite countable sets. For some specific choices of σ-finite reference measures, the ERM-RER boils down to particular cases of special interest: (i) the information-risk minimization problem presented in [23]; (ii) the ERM with differential entropy regularization (ERM-DiffER); and (iii) the ERM with discrete entropy regularization (ERM-DisER). See for instance [24], [25] and references therein. From this perspective, the proposed ERM-RER formulation yields a unified mathematical framework that comprises a large class of problems.

When the reference measure is a probability measure, the solution to the ERM-RER problem is known to be unique and correspond to a Gibbs probability measure. Such a Gibbs probability measure has been studied using measure theoretic and information theoretic notions in [9], [23], [26]–[33]; statistical physics in [3]; PAC (Probably Approximately Correct)-Bayesian learning theory in [34]–[37]; and proved to be of particular interest in classification problems in [5], [12], [20], [24], [38], [39] and [40]. In the general case in which the reference is a σ-finite measure, a solution to the ERM-RER problem does not always exist. Nonetheless, if it exists, it is shown to be a unique Gibbs probability despite the fact that its partition function is defined with respect to a σ-finite measure. The condition for the existence is mild and is always satisfied when the reference measure is a probability measure, as highlighted above. Interestingly, such a solution is mutually absolutely continuous with the reference measure and most of the properties known for the classical ERM-RER problem are shown to hold in the most general case. For instance, under certain conditions, the empirical risk observed when models are sampled from the ERM-RER-optimal probability measure is a sub-Gaussian random variable that exhibits a PAC guarantee for the ERM problem without regularization.

Samir M. Perlaza is with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis 06902, France; also with the ECE Dept. at Princeton University, Princeton N.J. 08544, USA; and also with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia.

Gaetan Bisson is with the GAATI Laboratory at the Université de la Polynésie Française, Faaa 98702, French Polynesia.

Iñaki Esnaola is with the ACSE Dept. at The University of Sheffield, Sheffield S1 3JD, UK; and also with the ECE Dept. at Princeton University, Princeton N.J. 08544, USA.

Alain Jean-Marie is with INRIA, Centre Inria d'Université Côte d'Azur, Sophia Antipolis 06902, France.

Stefano Rini is with the ECE Dept. at the National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan 30010, ROC.

This work was presented in part at the IEEE International Symposium on Information Theory (ISIT), Espoo, Finland, 2022 in [1]; and appears as an INRIA technical report in [2].
When the solution to the ERM-RER problem is used to sample models to label unseen patterns, the process is known as the Gibbs algorithm. One of the traditional performance metrics to evaluate the generalization capabilities of machine learning algorithms is the generalization error, for which closed-form expressions in terms of information measures are presented in [15]. When the reference measure is a probability measure, a closed-form expression for the generalization error of the Gibbs algorithm is presented in [9], while upper bounds have been derived in [18], [23], [31]–[37], [41]–[55], and references therein. In this work, a new performance metric coined sensitivity, which quantifies the variations of the expected empirical risk due to deviations from the solution of the ERM-RER problem is introduced. The sensitivity is defined as the difference between two quantities: (a) The expectation of the empirical risk with respect to the solution to the ERM-RER problem; and (b) the expectation of the empirical risk with respect to an alternative measure. The absolute value of the sensitivity is shown to be upper bounded by a term that is proportional to the squared-root of the relative entropy of the alternative measure with respect to the ERM-RER-optimal measure. Such bound allows providing lower and upper bounds on the expected empirical risk after a deviation from the ERM-RER-optimal measure towards an alternative probability measure. More interestingly, the expectation (with respect to the probability distribution of the datasets) of the sensitivity to deviations to a specific measure is shown to be equal to the generalization error of the Gibbs algorithm. Using this result, the closed-form expression for the generalization error of the Gibbs algorithm presented in [9] is shown to hold even in the case in which the reference measure is a $\sigma$-finite measure. Moreover, the generalization error is shown to be upper bounded by a term that is proportional to the squared-root of the lautum information between the models and the datasets, cf. [56]. This bound is reminiscent of the result in [33, Theorem 1] in which a similar bound is presented using the mutual information instead of the lautum information. While [33, Theorem 1] follows immediately from the variational representation of relative entropy, c.f., [57, Lemma 4.18 (Transportation Lemma)], the new result follows from the fact that the empirical risk when models are sampled from the ERM-RER-optimal probability measure is a sub-Gaussian random variable. Interestingly, the new upper-bound does not require any of the conditions in [33, Theorem 1].

The remainder of this work is organized as follows. Section II introduces two optimization problems: the ERM and the ERM-RER. The asymmetry of the relative entropy is analyzed in the context of the ERM-RER and two variants, coined Type-I and Type-II, are distinguished. The former considers the case in which the regularization is the relative entropy of the optimization measure with respect to the reference measure. The latter considers a regularization by the relative entropy of the reference measure with respect to the optimization measure. Section III presents the solution to the ERM-RER problem in the general case and introduces its main properties. Section IV introduces two new classes of reference measures and the solution of the ERM-RER problem is shown to

exhibit different properties for each class. This section ends by studying the ERM-RER problem in the special case in which the reference measure is a Gibbs probability measure. This special case exhibits a solution that is identical to the solution to an ERM-RER problem whose reference measure is the same used to build the above mentioned Gibbs measure. Section V studies the properties of the log-partition function of the ERM-RER-optimal probability measure. The first, second, and third cumulants of the empirical risk when the models are sampled from the ERM-RER-optimal measure and the reference measure are respectively characterized. Section VI and Section VII study the properties of the expectation and variance of the empirical risk when the models are sampled from the ERM-RER-optimal probability measure. This mean and variance are compared with the mean and variance of the empirical risk when models are sampled from the reference measure. Section VIII introduces several explicit expressions for the cumulant generating function of the empirical risk when the models are sampled from the ERM-RER-optimal measure. Using these equivalent expressions, under a specific condition, it is shown that the empirical risk is a sub-Gaussian random variable when models are sampled from the ERM-RER-optimal measure. Section IX describes the monotonic concentration of the ERM-RER-optimal probability measure when the regularization factor tends to zero. Section X show that the empirical risk when the models are sampled from the ERM-RER-optimal probability measure exhibits a PAC-type guarantee with respect to the ERM problem without regularization. Finally, Section XI studies the sensitivity of the expected empirical risk with respect to deviations from the ERM-RER-optimal measure to alternative measures and shows connections with the generalization error and the lautum information. Section XII ends this work with conclusions and a discussion on the results.

## II. EMPIRICAL RISK MINIMIZATION (ERM)

Let $\mathcal{M}, \mathcal{X}$ and $\mathcal{Y}$, with $\mathcal{M} \subseteq \mathbb{R}^d$ and $d \in \mathbb{N}$, be sets of models, patterns, and labels, respectively. A pair $(x, y) \in \mathcal{X} \times \mathcal{Y}$ is referred to as a labeled pattern or as a data point. Given $n$ data points, with $n \in \mathbb{N}$, denoted by $(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)$, the corresponding dataset is represented by the tuple

$$z = ((x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathcal{Y})^n. \quad (1)$$

Let the function $f: \mathcal{M} \times \mathcal{X} \to \mathcal{Y}$ be such that the label assigned to the pattern $x$ according to the model $\theta \in \mathcal{M}$ is $f(\theta, x)$. Let also the function

$$\ell: \mathcal{Y} \times \mathcal{Y} \to [0, +\infty] \quad (2)$$

be such that given a data point $(x, y) \in \mathcal{X} \times \mathcal{Y}$, the risk induced by a model $\theta \in \mathcal{M}$ is $\ell(f(\theta, x), y)$. In the following, the risk function $\ell$ is assumed to be nonnegative and for all $y \in \mathcal{Y}$, $\ell(y, y) = 0$.
When the solution to the ERM-RER problem is used to sample models to label unseen patterns, the process is known as the Gibbs algorithm. One of the traditional performance metrics to evaluate the generalization capabilities of machine learning algorithms is the generalization error, for which closed-form expressions in terms of information measures are presented in [15]. When the reference measure is a probability measure, a closed-form expression for the generalization error of the Gibbs algorithm is presented in [9], while upper bounds have been derived in [18], [23], [31]–[37], [41]–[55], and references therein. In this work, a new performance metric coined sensitivity, which quantifies the variations of the expected empirical risk due to deviations from the solution of the ERM-RER problem is introduced. The sensitivity is defined as the difference between two quantities: (a) The expectation of the empirical risk with respect to the solution to the ERM-RER problem; and (b) the expectation of the empirical risk with respect to an alternative measure. The absolute value of the sensitivity is shown to be upper bounded by a term that is proportional to the squared-root of the relative entropy of the alternative measure with respect to the ERM-RER-optimal measure. Such bound allows providing lower and upper bounds on the expected empirical risk after a deviation from the ERM-RER-optimal measure towards an alternative probability measure. More interestingly, the expectation (with respect to the probability distribution of the datasets) of the sensitivity to deviations to a specific measure is shown to be equal to the generalization error of the Gibbs algorithm. Using this result, the closed-form expression for the generalization error of the Gibbs algorithm presented in [9] is shown to hold even in the case in which the reference measure is a $\sigma$-finite measure. Moreover, the generalization error is shown to be upper bounded by a term that is proportional to the squared-root of the lautum information between the models and the datasets, cf. [56]. This bound is reminiscent of the result in [33, Theorem 1] in which a similar bound is presented using the mutual information instead of the lautum information. While [33, Theorem 1] follows immediately from the variational representation of relative entropy, c.f., [57, Lemma 4.18 (Transportation Lemma)], the new result follows from the fact that the empirical risk when models are sampled from the ERM-RER-optimal probability measure is a sub-Gaussian random variable. Interestingly, the new upper-bound does not require any of the conditions in [33, Theorem 1].

The remainder of this work is organized as follows. Section II introduces two optimization problems: the ERM and the ERM-RER. The asymmetry of the relative entropy is analyzed in the context of the ERM-RER and two variants, coined Type-I and Type-II, are distinguished. The former considers the case in which the regularization is the relative entropy of the optimization measure with respect to the reference measure. The latter considers a regularization by the relative entropy of the reference measure with respect to the optimization measure. Section III presents the solution to the ERM-RER problem in the general case and introduces its main properties. Section IV introduces two new classes of reference measures and the solution of the ERM-RER problem is shown to

exhibit different properties for each class. This section ends by studying the ERM-RER problem in the special case in which the reference measure is a Gibbs probability measure. This special case exhibits a solution that is identical to the solution to an ERM-RER problem whose reference measure is the same used to build the above mentioned Gibbs measure. Section V studies the properties of the log-partition function of the ERM-RER-optimal probability measure. The first, second, and third cumulants of the empirical risk when the models are sampled from the ERM-RER-optimal measure and the reference measure are respectively characterized. Section VI and Section VII study the properties of the expectation and variance of the empirical risk when the models are sampled from the ERM-RER-optimal probability measure. This mean and variance are compared with the mean and variance of the empirical risk when models are sampled from the reference measure. Section VIII introduces several explicit expressions for the cumulant generating function of the empirical risk when the models are sampled from the ERM-RER-optimal measure. Using these equivalent expressions, under a specific condition, it is shown that the empirical risk is a sub-Gaussian random variable when models are sampled from the ERM-RER-optimal measure. Section IX describes the monotonic concentration of the ERM-RER-optimal probability measure when the regularization factor tends to zero. Section X show that the empirical risk when the models are sampled from the ERM-RER-optimal probability measure exhibits a PAC-type guarantee with respect to the ERM problem without regularization. Finally, Section XI studies the sensitivity of the expected empirical risk with respect to deviations from the ERM-RER-optimal measure to alternative measures and shows connections with the generalization error and the lautum information. Section XII ends this work with conclusions and a discussion on the results.

## II. EMPIRICAL RISK MINIMIZATION (ERM)

Let $\mathcal{M}, \mathcal{X}$ and $\mathcal{Y}$, with $\mathcal{M} \subseteq \mathbb{R}^d$ and $d \in \mathbb{N}$, be sets of models, patterns, and labels, respectively. A pair $(x, y) \in \mathcal{X} \times \mathcal{Y}$ is referred to as a labeled pattern or as a data point. Given $n$ data points, with $n \in \mathbb{N}$, denoted by $(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)$, the corresponding dataset is represented by the tuple

$$z = ((x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathcal{Y})^n. \quad (1)$$

Let the function $f: \mathcal{M} \times \mathcal{X} \to \mathcal{Y}$ be such that the label assigned to the pattern $x$ according to the model $\theta \in \mathcal{M}$ is $f(\theta, x)$. Let also the function

$$\ell: \mathcal{Y} \times \mathcal{Y} \to [0, +\infty] \quad (2)$$

be such that given a data point $(x, y) \in \mathcal{X} \times \mathcal{Y}$, the risk induced by a model $\theta \in \mathcal{M}$ is $\ell(f(\theta, x), y)$. In the following, the risk function $\ell$ is assumed to be nonnegative and for all $y \in \mathcal{Y}$, $\ell(y, y) = 0$.
The empirical risk induced by the model $\theta$, with respect to the dataset $z$ in (1) is determined by the function $L_z : \mathcal{M} \to [0, +\infty]$, which satisfies

$$L_z(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell(f(\theta, x_i), y_i). \quad (3)$$

Using this notation, the ERM consists of the following optimization problem:

$$\min_{\theta \in \mathcal{M}} L_z(\theta). \quad (4)$$

Let the set of solutions to the ERM problem in (4) be denoted by

$$\mathcal{T}(z) \triangleq \arg \min_{\theta \in \mathcal{M}} L_z(\theta). \quad (5)$$

Note that if the set $\mathcal{M}$ is finite, the ERM problem in (4) always possesses a solution, and thus, $|\mathcal{T}(z)| > 0$. Nonetheless, in general, the ERM problem might not necessarily possess a solution, i.e., $|\mathcal{T}(z)| = 0$.

### A. Notation and Main Assumptions

In the following, given a measurable space $(\Omega, \mathcal{F})$, the notation $\triangleq (\Omega, \mathcal{F})$ is used to represent the set of $\sigma$-finite measures that can be defined over $(\Omega, \mathcal{F})$. Given a measure $Q \in \triangleq (\Omega, \mathcal{F})$, the subset $\triangle_Q (\Omega, \mathcal{F})$ of $\triangle (\Omega, \mathcal{F})$ contains all $\sigma$-finite measures that are absolutely continuous with respect to the measure $Q$. Alternatively, the subset $\triangledown_Q (\Omega, \mathcal{F})$ of $\triangle (\Omega, \mathcal{F})$ contains all probability measures $P$ such that $Q$ is absolutely continuous with respect to $P$. Given a set $\mathcal{A} \subset \mathbb{R}^d$, the Borel $\sigma$-field over $\mathcal{A}$ is denoted by $\mathcal{B}(\mathcal{A})$.

The main assumption adopted in this work is that the function $L_z$ in (3) is measurable with respect to the Borel measurable spaces $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and $([0, +\infty], \mathcal{B}([0, +\infty]))$.

### B. Relative Entropy Extended to $\sigma$-Finite Measures

In this work, the relative entropy, which is usually defined for probability measures, is extended to $\sigma$-finite measures.

**Definition 1 (Generalized Relative Entropy):** Given two $\sigma$-finite measures $P$ and $Q$ on the same measurable space, such that $P$ is absolutely continuous with respect to $Q$, the relative entropy of $P$ with respect to $Q$ is

$$D(P||Q) = \int \frac{dP}{dQ}(x) \log \left( \frac{dP}{dQ}(x) \right) dQ(x), \quad (6)$$

where the function $\frac{dP}{dQ}$ is the Radon-Nikodym derivative of $P$ with respect to $Q$.

The relative entropy exhibits a property often referred to as the information inequality [58, Theorem 2.6.3] in the case of probability measures on $(\Omega, \mathcal{F})$, with $\Omega$ a countable set. The following theorem explores this property in a more general scenario.

**Theorem 1:** If $P$ and $Q$ are both probability measures on a general measurable space $(\Omega, \mathcal{F})$, with $P$ absolutely continuous with respect to $Q$, then,

$$D(P||Q) \ge 0, \quad (7)$$

with equality if and only if $P$ and $Q$ are identical.

**Proof:** Consider the function $f: [0, \infty) \to \mathbb{R}$ such that for all $x \in (0, +\infty)$, $f(x) = x \log(x)$ and $f(0) = 0$. Note that $f$ is strictly convex. If $P$ and $Q$ are both probability measures on the measurable space $(\Omega, \mathcal{F})$, the following holds:

$$D(P||Q) = \int \frac{dP}{dQ}(x) \log \left( \frac{dP}{dQ}(x) \right) dQ(x) \quad (8)$$

$$= \int f \left( \frac{dP}{dQ}(x) \right) dQ(x) \quad (9)$$

$$\ge f \left( \int \frac{dP}{dQ}(x) dQ(x) \right) \quad (10)$$

$$= f(1) \quad (11)$$

$$= 0, \quad (12)$$

where the inequality (11) follows from Jensen's inequality [59, Section 6.3.5]. Equality in (11) holds if and only if for all $x \in \text{supp } Q$, $\frac{dP}{dQ}(x) = 1$, which implies that both $P$ and $Q$ are identical. This completes proof. ■

If $Q$ is not a probability measure, then it might be observed that $D(P||Q) < 0$. Consider for instance the case in which $P$ is a zero-mean Gaussian probability measure with variance $\sigma^2$ and $Q$ is the Lebesgue measure on $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$. Hence, the Radon-Nikodym derivative $\frac{dP}{dQ}$ is the Gaussian probability density function such that for all $x \in \mathbb{R}$,

$$\frac{dP}{dQ}(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left( -\frac{x^2}{2\sigma^2} \right). \quad (13)$$

Under this assumption, the relative entropy of $P$ with respect to $Q$ is the negative of the differential entropy of $P$. That is,

$$D(P||Q) = -\frac{1}{2} \log (2\pi\epsilon\sigma^2), \quad (14)$$

with $\epsilon$ being Néper's constant. See for instance [58, Example 8.1.2]. Hence, $D(P||Q)$ is negative for all $\sigma^2 \in (\frac{1}{2\pi\epsilon}, +\infty)$ and nonnegative for all $\sigma^2 \in (0, \frac{1}{2\pi\epsilon}]$. Finally, note also that

$$\lim_{\sigma^2 \to 0} D(P||Q) = +\infty, \quad \text{and} \quad (15)$$

$$\lim_{\sigma^2 \to +\infty} D(P||Q) = -\infty. \quad (16)$$

A central observation from (14) is that the equality $D(P||Q) = 0$ does not necessarily imply that $P$ and $Q$ are identical measures. For instance, when $\sigma^2 = \frac{1}{2\pi\epsilon}$ in (15), it holds that $D(P||Q) = 0$, while $P$ is a Gaussian probability measure and $Q$ is the Lebesgue measure.

The following property, known for the case of probability measures as the joint-convexity of the relative entropy, is extended by the following theorem.

**Theorem 2:** Let $P_1$ and $P_2$ be two probability measures and $Q_1$ and $Q_2$ be two $\sigma$-finite measures, all on the same measurable space. For all $i \in \{1, 2\}$, let $P_i$ be absolutely continuous with respect to $Q_i$. Then, for all $\lambda \in [0, 1]$,

$$D(\lambda P_1 + (1-\lambda)P_2 || \lambda Q_1 + (1-\lambda)Q_2) \\ \le \lambda D(P_1 || Q_1) + (1-\lambda) D(P_2 || Q_2). \quad (17)$$
The empirical risk induced by the model $\theta$, with respect to the dataset $z$ in (1) is determined by the function $L_z : \mathcal{M} \to [0, +\infty]$, which satisfies

$$L_z(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell(f(\theta, x_i), y_i). \quad (3)$$

Using this notation, the ERM consists of the following optimization problem:

$$\min_{\theta \in \mathcal{M}} L_z(\theta). \quad (4)$$

Let the set of solutions to the ERM problem in (4) be denoted by

$$\mathcal{T}(z) \triangleq \arg \min_{\theta \in \mathcal{M}} L_z(\theta). \quad (5)$$

Note that if the set $\mathcal{M}$ is finite, the ERM problem in (4) always possesses a solution, and thus, $|\mathcal{T}(z)| > 0$. Nonetheless, in general, the ERM problem might not necessarily possess a solution, i.e., $|\mathcal{T}(z)| = 0$.

### A. Notation and Main Assumptions

In the following, given a measurable space $(\Omega, \mathcal{F})$, the notation $\triangleq (\Omega, \mathcal{F})$ is used to represent the set of $\sigma$-finite measures that can be defined over $(\Omega, \mathcal{F})$. Given a measure $Q \in \triangleq (\Omega, \mathcal{F})$, the subset $\triangle_Q (\Omega, \mathcal{F})$ of $\triangle (\Omega, \mathcal{F})$ contains all $\sigma$-finite measures that are absolutely continuous with respect to the measure $Q$. Alternatively, the subset $\triangledown_Q (\Omega, \mathcal{F})$ of $\triangle (\Omega, \mathcal{F})$ contains all probability measures $P$ such that $Q$ is absolutely continuous with respect to $P$. Given a set $\mathcal{A} \subset \mathbb{R}^d$, the Borel $\sigma$-field over $\mathcal{A}$ is denoted by $\mathcal{B}(\mathcal{A})$.

The main assumption adopted in this work is that the function $L_z$ in (3) is measurable with respect to the Borel measurable spaces $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and $([0, +\infty], \mathcal{B}([0, +\infty]))$.

### B. Relative Entropy Extended to $\sigma$-Finite Measures

In this work, the relative entropy, which is usually defined for probability measures, is extended to $\sigma$-finite measures.

**Definition 1 (Generalized Relative Entropy):** Given two $\sigma$-finite measures $P$ and $Q$ on the same measurable space, such that $P$ is absolutely continuous with respect to $Q$, the relative entropy of $P$ with respect to $Q$ is

$$D(P||Q) = \int \frac{dP}{dQ}(x) \log \left( \frac{dP}{dQ}(x) \right) dQ(x), \quad (6)$$

where the function $\frac{dP}{dQ}$ is the Radon-Nikodym derivative of $P$ with respect to $Q$.

The relative entropy exhibits a property often referred to as the information inequality [58, Theorem 2.6.3] in the case of probability measures on $(\Omega, \mathcal{F})$, with $\Omega$ a countable set. The following theorem explores this property in a more general scenario.

**Theorem 1:** If $P$ and $Q$ are both probability measures on a general measurable space $(\Omega, \mathcal{F})$, with $P$ absolutely continuous with respect to $Q$, then,

$$D(P||Q) \ge 0, \quad (7)$$

with equality if and only if $P$ and $Q$ are identical.

**Proof:** Consider the function $f: [0, \infty) \to \mathbb{R}$ such that for all $x \in (0, +\infty)$, $f(x) = x \log(x)$ and $f(0) = 0$. Note that $f$ is strictly convex. If $P$ and $Q$ are both probability measures on the measurable space $(\Omega, \mathcal{F})$, the following holds:

$$D(P||Q) = \int \frac{dP}{dQ}(x) \log \left( \frac{dP}{dQ}(x) \right) dQ(x) \quad (8)$$

$$= \int f \left( \frac{dP}{dQ}(x) \right) dQ(x) \quad (9)$$

$$\ge f \left( \int \frac{dP}{dQ}(x) dQ(x) \right) \quad (10)$$

$$= f(1) \quad (11)$$

$$= 0, \quad (12)$$

where the inequality (11) follows from Jensen's inequality [59, Section 6.3.5]. Equality in (11) holds if and only if for all $x \in \text{supp } Q$, $\frac{dP}{dQ}(x) = 1$, which implies that both $P$ and $Q$ are identical. This completes proof. ■

If $Q$ is not a probability measure, then it might be observed that $D(P||Q) < 0$. Consider for instance the case in which $P$ is a zero-mean Gaussian probability measure with variance $\sigma^2$ and $Q$ is the Lebesgue measure on $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$. Hence, the Radon-Nikodym derivative $\frac{dP}{dQ}$ is the Gaussian probability density function such that for all $x \in \mathbb{R}$,

$$\frac{dP}{dQ}(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left( -\frac{x^2}{2\sigma^2} \right). \quad (13)$$

Under this assumption, the relative entropy of $P$ with respect to $Q$ is the negative of the differential entropy of $P$. That is,

$$D(P||Q) = -\frac{1}{2} \log (2\pi\epsilon\sigma^2), \quad (14)$$

with $\epsilon$ being Néper's constant. See for instance [58, Example 8.1.2]. Hence, $D(P||Q)$ is negative for all $\sigma^2 \in (\frac{1}{2\pi\epsilon}, +\infty)$ and nonnegative for all $\sigma^2 \in (0, \frac{1}{2\pi\epsilon}]$. Finally, note also that

$$\lim_{\sigma^2 \to 0} D(P||Q) = +\infty, \quad \text{and} \quad (15)$$

$$\lim_{\sigma^2 \to +\infty} D(P||Q) = -\infty. \quad (16)$$

A central observation from (14) is that the equality $D(P||Q) = 0$ does not necessarily imply that $P$ and $Q$ are identical measures. For instance, when $\sigma^2 = \frac{1}{2\pi\epsilon}$ in (15), it holds that $D(P||Q) = 0$, while $P$ is a Gaussian probability measure and $Q$ is the Lebesgue measure.

The following property, known for the case of probability measures as the joint-convexity of the relative entropy, is extended by the following theorem.

**Theorem 2:** Let $P_1$ and $P_2$ be two probability measures and $Q_1$ and $Q_2$ be two $\sigma$-finite measures, all on the same measurable space. For all $i \in \{1, 2\}$, let $P_i$ be absolutely continuous with respect to $Q_i$. Then, for all $\lambda \in [0, 1]$,

$$\begin{aligned} D(\lambda P_1 + (1-\lambda)P_2 || \lambda Q_1 + (1-\lambda)Q_2) \\ \le \lambda D(P_1 || Q_1) + (1-\lambda) D(P_2 || Q_2). \end{aligned} \quad (17)$$
Equality in (17) holds if and only if $P_1 = P_2$ and $Q_1 = Q_2$.

*Proof:* The proof is presented in Appendix A. ■

### C. ERM with Relative Entropy Regularization

Given a dataset, the *expected empirical risk* induced by a measure $P \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ is defined as follows.

**Definition 2 (Expected Empirical Risk):** Let $P$ be a probability measure in $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$. The expected empirical risk with respect to the dataset $z$ in (1) induced by the measure $P$ is

$$R_z(P) = \int L_z(\theta) dP(\theta), \quad (18)$$

where the function $L_z$ is defined in (3).

The ERM-RER problem is parametrized by a $\sigma$-finite measure in $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and a positive real, which are referred to as the *reference measure* and the *regularization factor*, respectively. Let $Q \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ be a $\sigma$-finite measure and let $\lambda$ be a positive real. The ERM-RER problem, with parameters $Q$ and $\lambda$, consists of the following optimization problem:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} R_z(P) + \lambda D(P \| Q), \quad (19a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (19b)$$

where the dataset $z$ is in (1), and the functional $R_z$ is defined in (18).

### D. Type-I and Type-II Relative Entropy Regularization

The optimization problem in (19) is coined Type-I ERM-RER in [60] in the aim of distinguishing it from the optimization problem

$$\min_{P \in \nabla_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} R_z(P) + \lambda D(Q \| P), \quad (20a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (20b)$$

which is coined Type-II ERM-RER.

The Type-II ERM-RER problem in (20), when $Q$ is a probability measure, exhibits a solution that is identical to the solution to the following Type-I ERM-RER problem [60, Theorem 1]:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \int \log(\beta + L_z(\nu)) dP(\nu) + D(P \| Q), \quad (21a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (21b)$$

where $\beta$ is a constant chosen to satisfy

$$\int \frac{\lambda}{\beta + L_z(\nu)} dQ(\nu) = 1. \quad (21c)$$

Essentially, by appropriately transforming the objective function, an equivalence can be established between Type-I and Type-II ERM-RER problems. Hence, without loss of generality, the remainder of this work focuses exclusively on Type-I ERM-RER, which is simply referred to as ERM-RER.

### III. THE SOLUTION TO THE ERM-RER PROBLEM

The solution to the ERM-RER problem in (19) is presented in terms of two objects. First, the function $K_{Q,z}: \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$ such that for all $t \in \mathbb{R}$,

$$K_{Q,z}(t) = \log \left( \int \exp(t L_z(\theta)) dQ(\theta) \right), \quad (22)$$

with $L_z$ in (3). Second, the set $K_{Q,z} \subset (0, +\infty)$, which is defined by

$$K_{Q,z} \triangleq \{s \in (0, +\infty) : K_{Q,z} \left(-\frac{1}{s}\right) < +\infty\}. \quad (23)$$

The notation for the function $K_{Q,z}$ and the set $K_{Q,z}$ are chosen such that their parametrization by (or dependence on) the dataset $z$ in (1) and the $\sigma$-finite measure $Q$ in (19) are highlighted.

The following lemma describes the set $K_{Q,z}$.

**Lemma 1:** The set $K_{Q,z}$ in (23) is a convex subset of $\mathbb{R}$. If the measure $Q$ in (19) is a probability measure, then, the set $K_{Q,z}$ in (23) satisfies

$$K_{Q,z} = (0, +\infty). \quad (24)$$

*Proof:* The proof is presented in Appendix B. ■

Using this notation, the solution to the ERM-RER problem in (19) is presented by the following theorem.

**Theorem 3:** If $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23), the solution to the optimization problem in (19) is a unique probability measure, denoted by $P_{\Theta|Z=z}^{(Q,\lambda)}$, which satisfies for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \exp \left( -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\theta) \right), \quad (25)$$

where the function $L_z$ is defined in (3) and the function $K_{Q,z}$ is defined in (22).

*Proof:* The proof is presented in Appendix C. ■

Contrary to the ERM problem in (4), which does not necessarily possess a solution, the ERM-RER problem in (19) always possess a solution when $Q$ is a probability measure. This is essentially because the set $K_{Q,z}$ is the set of all positive reals (Lemma 1), and thus, the condition $\lambda \in K_{Q,z}$ is always verified. On the contrary, when $Q$ is a $\sigma$-finite measure, the solution to the ERM-RER problem in (19) depends on whether $\lambda \in K_{Q,z}$. If the solution exists, it is $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), which is unique and corresponds to a Gibbs probability measure [61]. The function $K_{Q,z}$ is often referred to as the *log-partition function*, see for instance, [62, Section 7.3.1].

The following lemma shows that the Radon-Nikodym derivative in (25) is both nonnegative and finite.

**Lemma 2:** The Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies for all $\theta \in \text{supp } Q$ that

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) < +\infty. \quad (26)$$

Moreover, it is strictly positive almost surely with respect to $Q$.
Equality in (17) holds if and only if $P_1 = P_2$ and $Q_1 = Q_2$.

*Proof:* The proof is presented in Appendix A. ■

### C. ERM with Relative Entropy Regularization

Given a dataset, the *expected empirical risk* induced by a measure $P \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ is defined as follows.

**Definition 2 (Expected Empirical Risk):** Let $P$ be a probability measure in $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$. The expected empirical risk with respect to the dataset $z$ in (1) induced by the measure $P$ is

$$R_z(P) = \int L_z(\theta) dP(\theta), \quad (18)$$

where the function $L_z$ is defined in (3).

The ERM-RER problem is parametrized by a $\sigma$-finite measure in $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and a positive real, which are referred to as the *reference measure* and the *regularization factor*, respectively. Let $Q \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ be a $\sigma$-finite measure and let $\lambda$ be a positive real. The ERM-RER problem, with parameters $Q$ and $\lambda$, consists of the following optimization problem:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} R_z(P) + \lambda D(P \| Q), \quad (19a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (19b)$$

where the dataset $z$ is in (1), and the functional $R_z$ is defined in (18).

### D. Type-I and Type-II Relative Entropy Regularization

The optimization problem in (19) is coined Type-I ERM-RER in [60] in the aim of distinguishing it from the optimization problem

$$\min_{P \in \nabla_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} R_z(P) + \lambda D(Q \| P), \quad (20a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (20b)$$

which is coined Type-II ERM-RER.

The Type-II ERM-RER problem in (20), when $Q$ is a probability measure, exhibits a solution that is identical to the solution to the following Type-I ERM-RER problem [60, Theorem 1]:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \int \log(\beta + L_z(\nu)) dP(\nu) + D(P \| Q), \quad (21a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (21b)$$

where $\beta$ is a constant chosen to satisfy

$$\int \frac{\lambda}{\beta + L_z(\nu)} dQ(\nu) = 1. \quad (21c)$$

Essentially, by appropriately transforming the objective function, an equivalence can be established between Type-I and Type-II ERM-RER problems. Hence, without loss of generality, the remainder of this work focuses exclusively on Type-I ERM-RER, which is simply referred to as ERM-RER.

### III. THE SOLUTION TO THE ERM-RER PROBLEM

The solution to the ERM-RER problem in (19) is presented in terms of two objects. First, the function $K_{Q,z}: \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$ such that for all $t \in \mathbb{R}$,

$$K_{Q,z}(t) = \log \left( \int \exp(t L_z(\theta)) dQ(\theta) \right), \quad (22)$$

with $L_z$ in (3). Second, the set $K_{Q,z} \subset (0, +\infty)$, which is defined by

$$K_{Q,z} \triangleq \{s \in (0, +\infty) : K_{Q,z} \left(-\frac{1}{s}\right) < +\infty\}. \quad (23)$$

The notation for the function $K_{Q,z}$ and the set $K_{Q,z}$ are chosen such that their parametrization by (or dependence on) the dataset $z$ in (1) and the $\sigma$-finite measure $Q$ in (19) are highlighted.

The following lemma describes the set $K_{Q,z}$.

**Lemma 1:** The set $K_{Q,z}$ in (23) is a convex subset of $\mathbb{R}$. If the measure $Q$ in (19) is a probability measure, then, the set $K_{Q,z}$ in (23) satisfies

$$K_{Q,z} = (0, +\infty). \quad (24)$$

*Proof:* The proof is presented in Appendix B. ■

Using this notation, the solution to the ERM-RER problem in (19) is presented by the following theorem.

**Theorem 3:** If $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23), the solution to the optimization problem in (19) is a unique probability measure, denoted by $P_{\Theta|Z=z}^{(Q,\lambda)}$, which satisfies for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \exp \left( -K_{Q,z} \left( -\frac{1}{\lambda} \right) - \frac{1}{\lambda} L_z(\theta) \right), \quad (25)$$

where the function $L_z$ is defined in (3) and the function $K_{Q,z}$ is defined in (22).

*Proof:* The proof is presented in Appendix C. ■

Contrary to the ERM problem in (4), which does not necessarily possess a solution, the ERM-RER problem in (19) always possess a solution when $Q$ is a probability measure. This is essentially because the set $K_{Q,z}$ is the set of all positive reals (Lemma 1), and thus, the condition $\lambda \in K_{Q,z}$ is always verified. On the contrary, when $Q$ is a $\sigma$-finite measure, the solution to the ERM-RER problem in (19) depends on whether $\lambda \in K_{Q,z}$. If the solution exists, it is $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), which is unique and corresponds to a Gibbs probability measure [61]. The function $K_{Q,z}$ is often referred to as the *log-partition function*, see for instance, [62, Section 7.3.1].

The following lemma shows that the Radon-Nikodym derivative in (25) is both nonnegative and finite.

**Lemma 2:** The Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies for all $\theta \in \text{supp } Q$ that

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) < +\infty. \quad (26)$$

Moreover, it is strictly positive almost surely with respect to $Q$.
*Proof:* The proof is presented in Appendix D. ■

Theorem 3 has shown that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is absolutely continuous with respect to the measure $Q$. The following lemma shows that the converse is also true.

**Lemma 3:** The $\sigma$-finite measure $Q$ and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) are mutually absolutely continuous.

*Proof:* The proof is presented in Appendix E. ■

The relevance of Lemma 3 is that it shows that if $\lambda \in \mathcal{K}_{Q,z}$, the corresponding collections of negligible sets with respect to the measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ are identical. The following lemma shows that the negligible sets with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) are invariant with respect to $\lambda$.

**Lemma 4:** For all $(\alpha, \beta) \in \mathcal{K}_{Q,z} \times \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), assume that the probability measures $P_{\Theta|Z=z}^{(Q,\alpha)}$ and $P_{\Theta|Z=z}^{(Q,\beta)}$ satisfy (25) with $\lambda = \alpha$ and $\lambda = \beta$, respectively. Then, $P_{\Theta|Z=z}^{(Q,\alpha)}$ and $P_{\Theta|Z=z}^{(Q,\beta)}$ are mutually absolutely continuous.

*Proof:* The proof is presented in Appendix F. ■

Particular assumptions on the set $\mathcal{M}$ and the reference measure $Q$ lead to well-known instances of the ERM-RER problem in (19), as discussed hereunder.

### A. Examples

Three examples are of particular interest: (a) The set $\mathcal{M} \subset \mathbb{R}^d$ is countable and the measure $Q$ is the counting measure in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, which leads to the ERM-DisER problem; (b) The set $\mathcal{M}$ is an uncountable subset of $\mathbb{R}^d$, and $Q$ is the Lebesgue measure on $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, which leads to the ERM-DiffER problem; and (c) The set $\mathcal{M}$ and the measure $Q$ form a Borel probability measure space $(\mathcal{M}, \mathcal{B}(\mathcal{M}), Q)$, which leads to the information-risk minimization problem.

1) **ERM with Discrete Entropy Regularization:** When the set $\mathcal{M} \subset \mathbb{R}^d$ is countable and the $\sigma$-finite measure $Q$ in (19) is the counting measure in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, given a probability measure $P \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, the Radon-Nikodym derivative $\frac{dP}{dQ}$ is a probability mass function, denoted by $p$. Thus, the relative entropy $D(P||Q)$ is equivalent to the negative of the discrete entropy induced by $p$ [58, Chapter 2], denoted by $H(p)$. In this case, the ERM-RER in (19) can be re-written as the following ERM-DisER problem:

$$ \min_{p} \sum_{\theta \in \mathcal{M}} \mathcal{L}_{z}(\theta) p(\theta) - \lambda H(p), \quad (27) $$

where the optimization domain in (27) is the set of probability mass functions that can be defined over the measure space $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$. In this special case, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) whose probability mass function is the solution to the ERM-DisER problem in (27) satisfies

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \frac{\exp\left(-\frac{\mathcal{L}_z(\theta)}{\lambda}\right)}{\sum_{\nu \in \mathcal{M}} \exp\left(-\frac{\mathcal{L}_z(\nu)}{\lambda}\right)}, \quad (28) $$

which describes the discrete Gibbs probability measure on $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, with temperature parameter $\lambda$, and energy function $\mathcal{L}_z$ in (3).

2) **ERM with Differential Entropy Regularization:** When $\mathcal{M} \subset \mathbb{R}^d$ is uncountable and the $\sigma$-finite measure $Q$ in (19) is the Lebesgue measure in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, for all probability measures $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, the Radon-Nikodym derivative $\frac{dP}{dQ}$ is a probability density function, denoted by $g$. Thus, the relative entropy $D(P||Q)$ is equivalent to the negative of the differential entropy induced by $g$ [58, Chapter 8], denoted by $h(g)$. In this special case, the ERM-RER in (19) can be re-written as the following ERM-DiffER problem:

$$ \min_{g} \int_{\mathcal{M}} \mathcal{L}_{z}(\theta) g(\theta) d\theta - \lambda h(g), \quad (29) $$

where the optimization domain in (29) is the set of probability density functions that can be defined over the measure space $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$. The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) whose probability density function is the solution to the ERM-RER problem in (29) satisfies

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \frac{\exp\left(-\frac{\mathcal{L}_z(\theta)}{\lambda}\right)}{\int_{\mathcal{M}} \exp\left(-\frac{\mathcal{L}_z(\nu)}{\lambda}\right) d\nu}, \quad (30) $$

which describes the absolutely continuous Gibbs probability measure with temperature parameter $\lambda$ and energy function $\mathcal{L}_z$ in (3).

Both, the ERM-DiffER and ERM-DisER problems are closely related to those typically arising while using Jayne's maximum entropy principle [63], [64] for classification problems such as those in [24], [38], [39], and [65].

3) **Information-Risk Minimization:** When $Q$ is a probability measure, the ERM-RER in (19) is equivalent to the information-risk minimization (IRM) problem in [23]. The IRM problem in (19) is known to possess a unique solution equal to the Gibbs probability measure in (25), as independently shown in [23], [33], [61], [66], [67] and [68].

### B. Bounds on the Radon-Nikodym Derivative

The Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is greater for models inducing smaller empirical risks, as shown by the following corollary of Theorem 3.

**Corollary 1:** The Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies for all $(\theta_1, \theta_2) \in \text{supp} Q \times \text{supp} Q$, with $\mathcal{L}_z(\theta_2) \le \mathcal{L}_z(\theta_1)$, that

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta_1) \le \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta_2), \quad (31) $$

with equality if and only if $\mathcal{L}_z(\theta_1) = \mathcal{L}_z(\theta_2)$.

The intuition that follows from corollary 1 is that under the assumption that the ERM problem in (4) possesses a solution in the support of the reference measure, i.e., $\mathcal{T}(z) \cap \text{supp} Q$ is not empty, with $\mathcal{T}(z)$ in (5), the maximum of
*Proof:* The proof is presented in Appendix D. ■

Theorem 3 has shown that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is absolutely continuous with respect to the measure $Q$. The following lemma shows that the converse is also true.

**Lemma 3:** The $\sigma$-finite measure $Q$ and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) are mutually absolutely continuous.

*Proof:* The proof is presented in Appendix E. ■

The relevance of Lemma 3 is that it shows that if $\lambda \in \mathcal{K}_{Q,z}$, the corresponding collections of negligible sets with respect to the measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ are identical. The following lemma shows that the negligible sets with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) are invariant with respect to $\lambda$.

**Lemma 4:** For all $(\alpha, \beta) \in \mathcal{K}_{Q,z} \times \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), assume that the probability measures $P_{\Theta|Z=z}^{(Q,\alpha)}$ and $P_{\Theta|Z=z}^{(Q,\beta)}$ satisfy (25) with $\lambda = \alpha$ and $\lambda = \beta$, respectively. Then, $P_{\Theta|Z=z}^{(Q,\alpha)}$ and $P_{\Theta|Z=z}^{(Q,\beta)}$ are mutually absolutely continuous.

*Proof:* The proof is presented in Appendix F. ■

Particular assumptions on the set $\mathcal{M}$ and the reference measure $Q$ lead to well-known instances of the ERM-RER problem in (19), as discussed hereunder.

### A. Examples

Three examples are of particular interest: (a) The set $\mathcal{M} \subset \mathbb{R}^d$ is countable and the measure $Q$ is the counting measure in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, which leads to the ERM-DisER problem; (b) The set $\mathcal{M}$ is an uncountable subset of $\mathbb{R}^d$, and $Q$ is the Lebesgue measure on $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, which leads to the ERM-DiffER problem; and (c) The set $\mathcal{M}$ and the measure $Q$ form a Borel probability measure space $(\mathcal{M}, \mathcal{B}(\mathcal{M}), Q)$, which leads to the information-risk minimization problem.

1) **ERM with Discrete Entropy Regularization:** When the set $\mathcal{M} \subset \mathbb{R}^d$ is countable and the $\sigma$-finite measure $Q$ in (19) is the counting measure in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, given a probability measure $P \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, the Radon-Nikodym derivative $\frac{dP}{dQ}$ is a probability mass function, denoted by $p$. Thus, the relative entropy $D(P||Q)$ is equivalent to the negative of the discrete entropy induced by $p$ [58, Chapter 2], denoted by $H(p)$. In this case, the ERM-RER in (19) can be re-written as the following ERM-DisER problem:

$$ \min_{p} \sum_{\theta \in \mathcal{M}} \mathcal{L}_{z}(\theta) p(\theta) - \lambda H(p), \quad (27) $$

where the optimization domain in (27) is the set of probability mass functions that can be defined over the measure space $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$. In this special case, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) whose probability mass function is the solution to the ERM-DisER problem in (27) satisfies

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \frac{\exp\left(-\frac{\mathcal{L}_z(\theta)}{\lambda}\right)}{\sum_{\nu \in \mathcal{M}} \exp\left(-\frac{\mathcal{L}_z(\nu)}{\lambda}\right)}, \quad (28) $$

which describes the discrete Gibbs probability measure on $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, with temperature parameter $\lambda$, and energy function $\mathcal{L}_z$ in (3).

2) **ERM with Differential Entropy Regularization:** When $\mathcal{M} \subset \mathbb{R}^d$ is uncountable and the $\sigma$-finite measure $Q$ in (19) is the Lebesgue measure in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, for all probability measures $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, the Radon-Nikodym derivative $\frac{dP}{dQ}$ is a probability density function, denoted by $g$. Thus, the relative entropy $D(P||Q)$ is equivalent to the negative of the differential entropy induced by $g$ [58, Chapter 8], denoted by $h(g)$. In this special case, the ERM-RER in (19) can be re-written as the following ERM-DiffER problem:

$$ \min_{g} \int_{\mathcal{M}} \mathcal{L}_{z}(\theta) g(\theta) d\theta - \lambda h(g), \quad (29) $$

where the optimization domain in (29) is the set of probability density functions that can be defined over the measure space $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$. The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) whose probability density function is the solution to the ERM-RER problem in (29) satisfies

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \frac{\exp\left(-\frac{\mathcal{L}_z(\theta)}{\lambda}\right)}{\int_{\mathcal{M}} \exp\left(-\frac{\mathcal{L}_z(\nu)}{\lambda}\right) d\nu}, \quad (30) $$

which describes the absolutely continuous Gibbs probability measure with temperature parameter $\lambda$ and energy function $\mathcal{L}_z$ in (3).

Both, the ERM-DiffER and ERM-DisER problems are closely related to those typically arising while using Jayne's maximum entropy principle [63], [64] for classification problems such as those in [24], [38], [39], and [65].

3) **Information-Risk Minimization:** When $Q$ is a probability measure, the ERM-RER in (19) is equivalent to the information-risk minimization (IRM) problem in [23]. The IRM problem in (19) is known to possess a unique solution equal to the Gibbs probability measure in (25), as independently shown in [23], [33], [61], [66], [67] and [68].

### B. Bounds on the Radon-Nikodym Derivative

The Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is greater for models inducing smaller empirical risks, as shown by the following corollary of Theorem 3.

**Corollary 1:** The Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies for all $(\theta_1, \theta_2) \in \text{supp} Q \times \text{supp} Q$, with $\mathcal{L}_z(\theta_2) \le \mathcal{L}_z(\theta_1)$, that

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta_1) \le \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta_2), \quad (31) $$

with equality if and only if $\mathcal{L}_z(\theta_1) = \mathcal{L}_z(\theta_2)$.

The intuition that follows from corollary 1 is that under the assumption that the ERM problem in (4) possesses a solution in the support of the reference measure, i.e., $\mathcal{T}(z) \cap \text{supp} Q$ is not empty, with $\mathcal{T}(z)$ in (5), the maximum of
the function $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is achieved by the models in $\mathcal{T}(z) \cap \text{supp } Q$. When the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is either the probability mass function in (28) or the probability density function in (30), Corollary 1 shows that the elements of the set $\mathcal{T}(z) \cap \text{supp } Q$ are the modes of the corresponding probability density function or probability mass function.

### C. Asymptotes of the Radon-Nikodym Derivative

The following lemma describes the asymptotic behavior of the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) when the regularization factor increases, i.e., $\lambda \to +\infty$ and the reference measure $Q$ is a probability measure.

**Lemma 5:** Let the measure $Q$ in (19) be a probability measure. Then, for all $\theta \in \text{supp } Q$, the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies

$$ \lim_{\lambda \to +\infty} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = 1. \quad (32) $$

*Proof:* From Theorem 3, it follows that for all $\theta \in \text{supp } Q$,

$$ \begin{aligned} \lim_{\lambda \to +\infty} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) &= \lim_{\lambda \to +\infty} \frac{\exp\left(-\frac{L_z(\theta)}{\lambda}\right)}{\int \exp\left(-\frac{L_z(\nu)}{\lambda}\right) dQ(\nu)} \quad (33) \\ &= \frac{1}{\int dQ(\nu)} \quad (34) \\ &= 1, \end{aligned} $$

where the function $L_z$ is defined in (3). This completes the proof. ■

Lemma 5 unveils the fact that, when $Q$ is a probability measure, in the limit when $\lambda \to +\infty$, both probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ are identical. This is consistent with the fact that when $\lambda$ tends to infinity, the optimization problem in (19) boils down to exclusively minimizing the relative entropy. Such minimum is zero and is observed when both probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ are identical (Theorem 1). Such intuition breaks when the reference measure is a $\sigma$-finite measure, but not a probability measure. In such a case, the relative entropy term in (19) might be negative and a minimum might not exist. See for instance, the case of the relative entropy between a Gaussian measure and the Lebesgue measure in (14), which satisfies (16).

The limit of the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25), when $\lambda$ tends to zero from the right, can be studied using the following set

$$ \mathcal{L}_z(\delta) \triangleq \{\theta \in \mathcal{M} : L_z(\theta) \le \delta\}, \quad (36) $$

where the function $L_z$ is defined in (3) and $\delta \in [0, +\infty)$. In particular consider the nonnegative real

$$ \delta_{Q,z}^* \triangleq \inf \{\delta \in [0, +\infty) : Q(\mathcal{L}_z(\delta)) > 0\}. \quad (37) $$

Let also $\mathcal{L}_{Q,z}^*$ be the following level set of the empirical risk function $L_z$ in (3):

$$ \mathcal{L}_{Q,z}^* \triangleq \{\theta \in \text{supp } Q : L_z(\theta) = \delta_{Q,z}^*\}. \quad (38) $$

Using this notation, the limit of the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25), when $\lambda$ tends to zero from the right, is described by the following lemma.

**Lemma 6:** If $Q(\mathcal{L}_{Q,z}^*) > 0$, with the set $\mathcal{L}_{Q,z}^*$ in (38) and $Q$ the $\sigma$-finite measure in (19), then for all $\theta \in \text{supp } Q$, the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies

$$ \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \frac{1}{Q(\mathcal{L}_{Q,z}^*)} \mathbb{1}_{\{\theta \in \mathcal{L}_{Q,z}^*\}}. \quad (39) $$

Alternatively, if $Q(\mathcal{L}_{Q,z}^*) = 0$. Then, for all $\theta \in \text{supp } Q$,

$$ \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \begin{cases} +\infty & \text{if } \theta \in \mathcal{L}_{Q,z}^* \\ 0 & \text{otherwise.} \end{cases} \quad (40) $$

*Proof:* The proof is presented in Appendix G. ■

Consider that $Q(\mathcal{L}_{Q,z}^*) > 0$, with $\mathcal{L}_{Q,z}^*$ in (38). Under this assumption, from Lemma 6, it holds that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ asymptotically concentrates on the set $\mathcal{L}_{Q,z}^*$ when $\lambda$ tends to zero from the right. More specifically, note that for all measurable sets $A \subseteq \mathcal{L}_{Q,z}^* \cap \text{supp } Q$, it holds that

$$ \begin{aligned} \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A) &= \lim_{\lambda \to 0^+} \int_A \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) dQ(\theta) \quad (41) \\ &= \int \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \mathbb{1}_{\{\theta \in A\}} dQ(\theta) \quad (42) \\ &= \int \frac{1}{Q(\mathcal{L}_{Q,z}^*)} \mathbb{1}_{\{\theta \in \mathcal{L}_{Q,z}^*\}} \mathbb{1}_{\{\theta \in A\}} dQ(\theta) \quad (43) \\ &= \frac{1}{Q(\mathcal{L}_{Q,z}^*)} \int \mathbb{1}_{\{\theta \in A\}} dQ(\theta) \quad (44) \\ &= \frac{Q(A)}{Q(\mathcal{L}_{Q,z}^*)}, \end{aligned} $$

where the equality in (42) follows from Lemma 2 and the dominated convergence theorem [59, Theorem 2.6.9]. The equality in (43) follows from Lemma 6. In the particular case in which $A = \mathcal{L}_{Q,z}^*$ in (45), it holds that $\lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) = 1$, which verifies the asymptotic concentration of the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ on the set $\mathcal{L}_{Q,z}^*$.

Another interesting observation is that the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is a constant among the elements of the set $\mathcal{L}_{Q,z}^*$. This can be assimilated to a uniform distribution of the probability among the elements of the set $\mathcal{L}_{Q,z}^*$ in the limit when $\lambda$ tends to zero from the right, as previously highlighted in [26]–[28] and [29]. This becomes more evident in the case in which the set $\mathcal{M}$ is finite and $Q$ is the counting measure. In such a case, the asymptotic probability of each
the function $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is achieved by the models in $\mathcal{T}(z) \cap \text{supp } Q$. When the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is either the probability mass function in (28) or the probability density function in (30), Corollary 1 shows that the elements of the set $\mathcal{T}(z) \cap \text{supp } Q$ are the modes of the corresponding probability density function or probability mass function.

### C. Asymptotes of the Radon-Nikodym Derivative

The following lemma describes the asymptotic behavior of the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) when the regularization factor increases, i.e., $\lambda \to +\infty$ and the reference measure $Q$ is a probability measure.

**Lemma 5:** Let the measure $Q$ in (19) be a probability measure. Then, for all $\theta \in \text{supp } Q$, the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies

$$ \lim_{\lambda \to +\infty} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = 1. \quad (32) $$

*Proof:* From Theorem 3, it follows that for all $\theta \in \text{supp } Q$,

$$ \lim_{\lambda \to +\infty} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \lim_{\lambda \to +\infty} \frac{\exp\left(-\frac{L_z(\theta)}{\lambda}\right)}{\int \exp\left(-\frac{L_z(\nu)}{\lambda}\right) dQ(\nu)} \quad (33) $$

$$ \begin{aligned} &= \frac{1}{\int dQ(\nu)} \\ &= 1, \end{aligned} \quad (35) $$

where the function $L_z$ is defined in (3). This completes the proof. ■

Lemma 5 unveils the fact that, when $Q$ is a probability measure, in the limit when $\lambda \to +\infty$, both probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ are identical. This is consistent with the fact that when $\lambda$ tends to infinity, the optimization problem in (19) boils down to exclusively minimizing the relative entropy. Such minimum is zero and is observed when both probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $Q$ are identical (Theorem 1). Such intuition breaks when the reference measure is a $\sigma$-finite measure, but not a probability measure. In such a case, the relative entropy term in (19) might be negative and a minimum might not exist. See for instance, the case of the relative entropy between a Gaussian measure and the Lebesgue measure in (14), which satisfies (16).

The limit of the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25), when $\lambda$ tends to zero from the right, can be studied using the following set

$$ \mathcal{L}_z(\delta) \triangleq \{\theta \in \mathcal{M} : L_z(\theta) \le \delta\}, \quad (36) $$

where the function $L_z$ is defined in (3) and $\delta \in [0, +\infty)$. In particular consider the nonnegative real

$$ \delta_{Q,z}^* \triangleq \inf \{\delta \in [0, +\infty) : Q(\mathcal{L}_z(\delta)) > 0\}. \quad (37) $$

Let also $\mathcal{L}_{Q,z}^*$ be the following level set of the empirical risk function $L_z$ in (3):

$$ \mathcal{L}_{Q,z}^* \triangleq \{\theta \in \text{supp } Q : L_z(\theta) = \delta_{Q,z}^*\}. \quad (38) $$

Using this notation, the limit of the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25), when $\lambda$ tends to zero from the right, is described by the following lemma.

**Lemma 6:** If $Q(\mathcal{L}_{Q,z}^*) > 0$, with the set $\mathcal{L}_{Q,z}^*$ in (38) and $Q$ the $\sigma$-finite measure in (19), then for all $\theta \in \text{supp } Q$, the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) satisfies

$$ \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \frac{1}{Q(\mathcal{L}_{Q,z}^*)} \mathbb{1}_{\{\theta \in \mathcal{L}_{Q,z}^*\}}. \quad (39) $$

Alternatively, if $Q(\mathcal{L}_{Q,z}^*) = 0$. Then, for all $\theta \in \text{supp } Q$,

$$ \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = \begin{cases} +\infty & \text{if } \theta \in \mathcal{L}_{Q,z}^* \\ 0 & \text{otherwise.} \end{cases} \quad (40) $$

*Proof:* The proof is presented in Appendix G. ■

<figure><img src="image_2.png" /></figure>

Consider that $Q(\mathcal{L}_{Q,z}^*) > 0$, with $\mathcal{L}_{Q,z}^*$ in (38). Under this assumption, from Lemma 6, it holds that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ asymptotically concentrates on the set $\mathcal{L}_{Q,z}^*$ when $\lambda$ tends to zero from the right. More specifically, note that for all measurable sets $A \subseteq \mathcal{L}_{Q,z}^* \cap \text{supp } Q$, it holds that

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A) = \lim_{\lambda \to 0^+} \int_A \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) dQ(\theta) \quad (41) $$

$$ = \int \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \mathbb{1}_{\{\theta \in A\}} dQ(\theta) \quad (42) $$

$$ = \int \frac{1}{Q(\mathcal{L}_{Q,z}^*)} \mathbb{1}_{\{\theta \in \mathcal{L}_{Q,z}^*\}} \mathbb{1}_{\{\theta \in A\}} dQ(\theta) \quad (43) $$

$$ = \frac{1}{Q(\mathcal{L}_{Q,z}^*)} \int \mathbb{1}_{\{\theta \in A\}} dQ(\theta) \quad (44) $$

$$ = \frac{Q(A)}{Q(\mathcal{L}_{Q,z}^*)}, \quad (45) $$

where the equality in (42) follows from Lemma 2 and the dominated convergence theorem [59, Theorem 2.6.9]. The equality in (43) follows from Lemma 6. In the particular case in which $A = \mathcal{L}_{Q,z}^*$ in (45), it holds that $\lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) = 1$, which verifies the asymptotic concentration of the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ on the set $\mathcal{L}_{Q,z}^*$.

Another interesting observation is that the Radon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ in (25) is a constant among the elements of the set $\mathcal{L}_{Q,z}^*$. This can be assimilated to a uniform distribution of the probability among the elements of the set $\mathcal{L}_{Q,z}^*$ in the limit when $\lambda$ tends to zero from the right, as previously highlighted in [26]–[28] and [29]. This becomes more evident in the case in which the set $\mathcal{M}$ is finite and $Q$ is the counting measure. In such a case, the asymptotic probability of each
of the elements in $\mathcal{L}_{Q,z}^*$ when $\lambda$ tends to zero from the right is $\frac{1}{|\mathcal{L}_{Q,z}^*|}$.

Consider now that $Q(\mathcal{L}_{Q,z}^*) = 0$, with $\mathcal{L}_{Q,z}^*$ in (38). Under this assumption, in the asymptotic regime when $\lambda \to 0$, the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is not a probability measure but either the trivial measure or the infinite measure. This is typically the case in which $M = \mathbb{R}^d$, the measure $Q$ is absolutely continuous with respect to the Lebesgue measure, and the solution to the ERM problem in (4) has a unique solution on the support of $Q$, i.e., $\mathcal{L}_{Q,z}^* = \mathcal{T}(z)$ and $|\mathcal{T}(z)| = 1$, which implies $Q(\mathcal{L}_{Q,z}^*) = 0$.

An interesting question, which is left out of the scope of this paper, is the rate at which $P_{\Theta|Z=z}^{(Q,\lambda)}$ converges to such limiting measure. The interested reader is referred to [26], [29], and references therein.

The following lemma shows that independently of whether the set $\mathcal{L}_{Q,z}^*$ is negligible with respect to the measure $Q$, the limit when $\lambda$ tends to zero from the right of $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*)$ is equal to one.

**Lemma 7:** The measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the set $\mathcal{L}_{Q,z}^*$ in (38) satisfy,

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_{Q,z}^*) = 1. \quad (46) $$

*Proof:* The proof is presented in Appendix H. ■

Note that if the ERM problem in (4) possesses at least one solution and such solution is within the support of the measure $Q$, i.e., $\mathcal{T}(z) \cap \text{supp} Q \neq \emptyset$, then, when $\lambda$ tends to zero from the right, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ asymptotically concentrates on the solution (or the set of solutions within the support of $Q$) to the ERM problem in (4). Alternatively, in the case in which $\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z) = \emptyset$, when $\lambda$ tends to zero from the right, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ asymptotically concentrates on a set that does not contain the set of solutions to the ERM problem in (4). This observation leads to the introduction to two new classes of reference measures, namely, *coherent* and *consistent* measures, in the following section.

# IV. REFERENCE MEASURES

This section introduces two classes of reference measures, namely *coherent* and *consistent* measures, and discusses the special case of Gibbs reference measures.

## A. Coherent and Consistent Reference Measures

A class of reference measures of particular importance to establish connections between the set of solutions to the ERM problem in (4) and the solution to the ERM-RER problem in (19) is that of *coherent* measures. Let $\rho^* \ge 0$ be the infimum of the empirical risk $L_z$ in (3). That is,

$$ \rho^* \triangleq \inf\{L_z(\theta) : \theta \in \mathcal{M}\}. \quad (47) $$

Using this notation, coherent measures are defined as follows.

**Definition 3 (Coherent Measures):** The $\sigma$-finite measure $Q$ in (19) is said to be coherent if, for all $\delta \in (\rho^*, +\infty)$, with $\rho^*$ in (47), it holds that

$$ Q(\mathcal{L}_z(\delta)) > 0, \quad (48) $$

where the set $\mathcal{L}_z(\delta)$ is defined in (36).

When the reference measure $Q$ in the EMR-RER problem in (19) is a coherent measure, it holds that for all $\delta > \rho^*$, the set $\mathcal{L}_z(\delta)$ in (36) exhibits positive probability with respect to the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25). The following lemma highlights this observation.

**Lemma 8:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies for all $\delta \in (\rho^*, +\infty)$, with $\rho^*$ in (47), that

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) > 0, \quad (49) $$

with $\mathcal{L}_z(\delta)$ in (36), if and only if the $\sigma$-finite measure $Q$ in (19) is coherent.

*Proof:* The proof is presented in Appendix I. ■

Under the assumption that the ERM problem in (4) possesses a solution, it holds that

$$ \min_{\theta \in \mathcal{M}} \mathcal{L}_z(\theta) = \inf\{\mathcal{L}_z(\theta) : \theta \in \mathcal{M}\}. \quad (50) $$

Hence, when the $\sigma$-finite measure $Q$ in (19) is coherent, then

$$ \delta_{Q,z}^* = \rho^*, \quad (51) $$

with $\delta_{Q,z}^*$ in (37) and $\rho^*$ in (47), which implies that

$$ \mathcal{L}_{Q,z}^* \subseteq \mathcal{T}(z), \quad (52) $$

with $\mathcal{T}(z)$ in (5) and $\mathcal{L}_{Q,z}^*$ in (38). This observation, together with Lemma 7, leads to the following result.

**Lemma 9:** Assume that the ERM problem in (4) possesses a solution. Then, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the sets $\mathcal{T}(z)$ in (5) and $\mathcal{L}_{Q,z}^*$ in (38) satisfy

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z)) = 1, \quad (53) $$

if and only if the $\sigma$-finite measure $Q$ in (19) is coherent.

*Proof:* The proof follows by observing that if $Q$ is a coherent measure and the ERM problem in (4) possesses a solution, the inclusion in (52) holds. Thus, from Lemma 7, the equality in (53) holds. Alternatively, when the measure $Q$ in (19) is noncoherent, then $\delta_{Q,z}^* > \rho^*$, which implies that $\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z) = \emptyset$. Hence, from Lemma 7, it follows that

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z)) = 0, \quad (54) $$

and completes the proof. ■

The relevance of coherent measures in ERM-RER problems is well highlighted by Lemma 9. Essentially, when the ERM problem in (4) possesses at least one solution, the concentration of the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) on the set (or a subset) of solutions to the ERM problem in (4) occurs asymptotically when $\lambda$ tends to zero from the right, if only if
of the elements in $\mathcal{L}_{Q,z}^*$ when $\lambda$ tends to zero from the right is $\frac{1}{|\mathcal{L}_{Q,z}^*|}$.

Consider now that $Q(\mathcal{L}_{Q,z}^*) = 0$, with $\mathcal{L}_{Q,z}^*$ in (38). Under this assumption, in the asymptotic regime when $\lambda \to 0$, the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is not a probability measure but either the trivial measure or the infinite measure. This is typically the case in which $M = \mathbb{R}^d$, the measure $Q$ is absolutely continuous with respect to the Lebesgue measure, and the solution to the ERM problem in (4) has a unique solution on the support of $Q$, i.e., $\mathcal{L}_{Q,z}^* = \mathcal{T}(z)$ and $|\mathcal{T}(z)| = 1$, which implies $Q(\mathcal{L}_{Q,z}^*) = 0$.

An interesting question, which is left out of the scope of this paper, is the rate at which $P_{\Theta|Z=z}^{(Q,\lambda)}$ converges to such limiting measure. The interested reader is referred to [26], [29], and references therein.

The following lemma shows that independently of whether the set $\mathcal{L}_{Q,z}^*$ is negligible with respect to the measure $Q$, the limit when $\lambda$ tends to zero from the right of $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*)$ is equal to one.

**Lemma 7:** The measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the set $\mathcal{L}_{Q,z}^*$ in (38) satisfy,

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_{Q,z}^*) = 1. \quad (46) $$

*Proof:* The proof is presented in Appendix H. ■

Note that if the ERM problem in (4) possesses at least one solution and such solution is within the support of the measure $Q$, i.e., $\mathcal{T}(z) \cap \text{supp} Q \neq \emptyset$, then, when $\lambda$ tends to zero from the right, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ asymptotically concentrates on the solution (or the set of solutions within the support of $Q$) to the ERM problem in (4). Alternatively, in the case in which $\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z) = \emptyset$, when $\lambda$ tends to zero from the right, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ asymptotically concentrates on a set that does not contain the set of solutions to the ERM problem in (4). This observation leads to the introduction to two new classes of reference measures, namely, *coherent* and *consistent* measures, in the following section.

# IV. REFERENCE MEASURES

This section introduces two classes of reference measures, namely *coherent* and *consistent* measures, and discusses the special case of Gibbs reference measures.

## A. Coherent and Consistent Reference Measures

A class of reference measures of particular importance to establish connections between the set of solutions to the ERM problem in (4) and the solution to the ERM-RER problem in (19) is that of *coherent* measures. Let $\rho^* \ge 0$ be the infimum of the empirical risk $L_z$ in (3). That is,

$$ \rho^* \triangleq \inf\{L_z(\theta) : \theta \in \mathcal{M}\}. \quad (47) $$

Using this notation, coherent measures are defined as follows.

**Definition 3 (Coherent Measures):** The $\sigma$-finite measure $Q$ in (19) is said to be coherent if, for all $\delta \in (\rho^*, +\infty)$, with $\rho^*$ in (47), it holds that

$$ Q(\mathcal{L}_z(\delta)) > 0, \quad (48) $$

where the set $\mathcal{L}_z(\delta)$ is defined in (36).

When the reference measure $Q$ in the EMR-RER problem in (19) is a coherent measure, it holds that for all $\delta > \rho^*$, the set $\mathcal{L}_z(\delta)$ in (36) exhibits positive probability with respect to the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25). The following lemma highlights this observation.

**Lemma 8:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies for all $\delta \in (\rho^*, +\infty)$, with $\rho^*$ in (47), that

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) > 0, \quad (49) $$

with $\mathcal{L}_z(\delta)$ in (36), if and only if the $\sigma$-finite measure $Q$ in (19) is coherent.

*Proof:* The proof is presented in Appendix I. ■

Under the assumption that the ERM problem in (4) possesses a solution, it holds that

$$ \min_{\theta \in \mathcal{M}} \mathcal{L}_z(\theta) = \inf\{\mathcal{L}_z(\theta) : \theta \in \mathcal{M}\}. \quad (50) $$

Hence, when the $\sigma$-finite measure $Q$ in (19) is coherent, then

$$ \delta_{Q,z}^* = \rho^*, \quad (51) $$

with $\delta_{Q,z}^*$ in (37) and $\rho^*$ in (47), which implies that

$$ \mathcal{L}_{Q,z}^* \subseteq \mathcal{T}(z), \quad (52) $$

with $\mathcal{T}(z)$ in (5) and $\mathcal{L}_{Q,z}^*$ in (38). This observation, together with Lemma 7, leads to the following result.

**Lemma 9:** Assume that the ERM problem in (4) possesses a solution. Then, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the sets $\mathcal{T}(z)$ in (5) and $\mathcal{L}_{Q,z}^*$ in (38) satisfy

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z)) = 1, \quad (53) $$

if and only if the $\sigma$-finite measure $Q$ in (19) is coherent.

*Proof:* The proof follows by observing that if $Q$ is a coherent measure and the ERM problem in (4) possesses a solution, the inclusion in (52) holds. Thus, from Lemma 7, the equality in (53) holds. Alternatively, when the measure $Q$ in (19) is noncoherent, then $\delta_{Q,z}^* > \rho^*$, which implies that $\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z) = \emptyset$. Hence, from Lemma 7, it follows that

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z)) = 0, \quad (54) $$

and completes the proof. ■

The relevance of coherent measures in ERM-RER problems is well highlighted by Lemma 9. Essentially, when the ERM problem in (4) possesses at least one solution, the concentration of the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) on the set (or a subset) of solutions to the ERM problem in (4) occurs asymptotically when $\lambda$ tends to zero from the right, if only if
the reference measure $Q$ in (19) is coherent. Nonetheless, such asymptotic concentration is not a guarantee that for strictly positive values of $\lambda$ in (19), the set $\mathcal{T}(z)$ in (5) and the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfy $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{T}(z)) > 0$. In order to ensure this, another class of reference measures, known as consistent measures, is introduced.

**Definition 4 (Consistent Measure):** The $\sigma$-finite measure $Q$ in (19) is said to be consistent if $Q(\mathcal{L}_{Q,z}^*) > 0$, with $\mathcal{L}_{Q,z}^*$ in (38).

Note that every consistent measure is not necessarily coherent. For instance, if $Q$ is consistent but $\delta_{Q,z}^* > \rho^*$, with $\rho^*$ in (47) and $\delta_{Q,z}^*$ in (37), then, for all $\delta \in (\rho^*, \delta_{Q,z}^*)$, it follows that $Q(\mathcal{L}_z(\delta)) = 0$, and thus, $Q$ is not coherent. Alternatively, every coherent measure is not necessarily consistent. For instance, if $|\mathcal{L}_{Q,z}^*| = 1$ and $Q$ is coherent and absolutely continuous with respect to the Lebesgue measure, it follows that $Q(\mathcal{L}_{Q,z}^*) = 0$, and thus, $Q$ is not consistent.

The relevance of consistent measures is highlighted by the following lemma.

**Lemma 10:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the set $\mathcal{L}_{Q,z}^*$ in (38) satisfy

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) > 0, \quad (55)$$

if and only if the $\sigma$-finite measure $Q$ in (19) is consistent.

**Proof:** When $Q$ is nonconsistent, it holds that $Q(\mathcal{L}_{Q,z}^*) = 0$ and thus, from the fact that the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is absolutely continuous with respect to $Q$, it holds that $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) = 0$. When $Q$ is consistent, it holds that $Q(\mathcal{L}_{Q,z}^*) > 0$. Moreover, for all $\theta \in \mathcal{L}_{Q,z}^*$, it holds that $\mathcal{L}_z(\theta) < +\infty$ and thus, from Lemma 2, it follows that $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) > 0$. Hence,

$$ \begin{aligned} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) &= \int_{\mathcal{L}_{Q,z}^*} dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (56) \\ &= \int_{\mathcal{L}_{Q,z}^*} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) dQ(\theta) > 0, \quad (57) \end{aligned} $$

which completes the proof. ■

The following lemma highlights a central property of consistent measures when the ERM problem in (4) possesses a solution.

**Lemma 11:** Assume that the ERM problem in (4) possesses a solution in the support of $Q$. The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the sets $\mathcal{T}(z)$ in (5) and $\mathcal{L}_{Q,z}^*$ in (38) satisfy

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z)) > 0, \quad (58)$$

if and only if the $\sigma$-finite measure $Q$ in (19) is consistent.

**Proof:** The proof follows from Lemma 10 by noticing that when the ERM problem in (4) possesses a solution in the support of $Q$, the inclusion in (52) holds. ■

The distinction between coherent and consistent measures becomes more evident under certain conditions. Consider the case in which $\mathcal{M}$ is finite. In this case, if the solution to the ERM problem in (4) is in the support of the $\sigma$-finite measure $Q$, then $Q$ is both coherent and consistent. This is essentially because all measurable singletons (models) in supp $Q$ exhibit positive measure with respect to $Q$. Alternatively, if the solution to the ERM problem in (4) is not in the support of $Q$, then $Q$ is consistent but not coherent. Consider the case in which $\mathcal{M}$ is the set $\mathbb{R}^d$; the loss function $\ell$ in (2) is continuous; and the ERM problem in (4) admits a unique solution. In this case, any probability measure $Q$ absolutely continuous with respect to the Lebesgue measure is a coherent measure, but it is not a consistent measure. Alternatively, if the set of solutions to the ERM problem in (4) exhibits positive Lebesgue measure, then, the measure $Q$ is both coherent and consistent.

## B. Gibbs Reference Measures

In model selection, a natural idea is to proceed by successive approximations in the seek of lower computation complexity. From this perspective, one might wonder whether the solution to a current instance of an ERM-RER problem might serve as reference measure for the next instance. In this section, it is shown that this yields no benefit. Composing two successive ERM-REM problems boils down to a unique ERM-RER problem with the initial reference measure and a particular regularization factor. Under the assumption that $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the problem of interest is:

$$ \min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathrm{R}_z(P) + \alpha D(P \| P_{\Theta|Z=z}^{(Q,\lambda)}), \quad (59a) $$

$$ \text{s. t.} \quad \int dP(\theta) = 1, \quad (59b) $$

where $\alpha > 0$; the reference measure $P_{\Theta|Z=z}^{(Q,\lambda)}$, which satisfies (25), is the solution of the ERM-RER problem in (19); and the functional $\mathbb{R}_z$ is defined in (18). From Theorem 3, the solution to the ERM-RER problem in (59), which is denoted by $P_{\Theta|Z=z}^{(P_{\Theta|Z=z}^{(Q,\lambda)}, \alpha)}$, satisfies for all $\theta \in \text{supp } Q$ that

$$ \frac{dP_{\Theta|Z=z}^{(P_{\Theta|Z=z}^{(Q,\lambda)}, \alpha)}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) = \exp \left( -K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z} \left( -\frac{1}{\alpha} \right) - \frac{1}{\alpha} \mathcal{L}_z(\theta) \right). \quad (60) $$

The log-partition functions $K_{Q,z}$ in (22) and $K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}$ in (60) are strongly related, as shown by the following lemma.

**Lemma 12:** The functions $K_{Q,z}$ in (22) and $K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}$ in (60) satisfy for all $t \in \mathbb{R}$,

$$ K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}(t) = K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right). \quad (61) $$

Moreover, for all $t \le 0$,

$$ K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}(t) \le 0. \quad (62) $$
the reference measure $Q$ in (19) is coherent. Nonetheless, such asymptotic concentration is not a guarantee that for strictly positive values of $\lambda$ in (19), the set $\mathcal{T}(z)$ in (5) and the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfy $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{T}(z)) > 0$. In order to ensure this, another class of reference measures, known as *consistent measures*, is introduced.

**Definition 4 (Consistent Measure):** The $\sigma$-finite measure $Q$ in (19) is said to be consistent if $Q(\mathcal{L}_{Q,z}^*) > 0$, with $\mathcal{L}_{Q,z}^*$ in (38).

Note that every consistent measure is not necessarily coherent. For instance, if $Q$ is consistent but $\delta_{Q,z}^* > \rho^*$, with $\rho^*$ in (47) and $\delta_{Q,z}^*$ in (37), then, for all $\delta \in (\rho^*, \delta_{Q,z}^*)$, it follows that $Q(\mathcal{L}_z(\delta)) = 0$, and thus, $Q$ is not coherent. Alternatively, every coherent measure is not necessarily consistent. For instance, if $|\mathcal{L}_{Q,z}^*| = 1$ and $Q$ is coherent and absolutely continuous with respect to the Lebesgue measure, it follows that $Q(\mathcal{L}_{Q,z}^*) = 0$, and thus, $Q$ is not consistent.

The relevance of consistent measures is highlighted by the following lemma.

**Lemma 10:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the set $\mathcal{L}_{Q,z}^*$ in (38) satisfy

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) > 0, \quad (55)$$

if and only if the $\sigma$-finite measure $Q$ in (19) is consistent.

**Proof:** When $Q$ is nonconsistent, it holds that $Q(\mathcal{L}_{Q,z}^*) = 0$ and thus, from the fact that the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is absolutely continuous with respect to $Q$, it holds that $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) = 0$. When $Q$ is consistent, it holds that $Q(\mathcal{L}_{Q,z}^*) > 0$. Moreover, for all $\theta \in \mathcal{L}_{Q,z}^*$, it holds that $\mathcal{L}_z(\theta) < +\infty$ and thus, from Lemma 2, it follows that $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) > 0$. Hence,

$$\begin{aligned} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) &= \int_{\mathcal{L}_{Q,z}^*} dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (56) \\ &= \int_{\mathcal{L}_{Q,z}^*} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) dQ(\theta) > 0, \quad (57) \end{aligned}$$

which completes the proof. ■

The following lemma highlights a central property of consistent measures when the ERM problem in (4) possesses a solution.

**Lemma 11:** Assume that the ERM problem in (4) possesses a solution in the support of $Q$. The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the sets $\mathcal{T}(z)$ in (5) and $\mathcal{L}_{Q,z}^*$ in (38) satisfy

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^* \cap \mathcal{T}(z)) > 0, \quad (58)$$

if and only if the $\sigma$-finite measure $Q$ in (19) is consistent.

**Proof:** The proof follows from Lemma 10 by noticing that when the ERM problem in (4) possesses a solution in the support of $Q$, the inclusion in (52) holds. ■

The distinction between coherent and consistent measures becomes more evident under certain conditions. Consider the case in which $\mathcal{M}$ is finite. In this case, if the solution to the ERM problem in (4) is in the support of the $\sigma$-finite measure $Q$, then $Q$ is both coherent and consistent. This is essentially because all measurable singletons (models) in supp $Q$ exhibit positive measure with respect to $Q$. Alternatively, if the solution to the ERM problem in (4) is not in the support of $Q$, then $Q$ is consistent but not coherent. Consider the case in which $\mathcal{M}$ is the set $\mathbb{R}^d$; the loss function $\ell$ in (2) is continuous; and the ERM problem in (4) admits a unique solution. In this case, any probability measure $Q$ absolutely continuous with respect to the Lebesgue measure is a coherent measure, but it is not a consistent measure. Alternatively, if the set of solutions to the ERM problem in (4) exhibits positive Lebesgue measure, then, the measure $Q$ is both coherent and consistent.

## B. Gibbs Reference Measures

In model selection, a natural idea is to proceed by successive approximations in the seek of lower computation complexity. From this perspective, one might wonder whether the solution to a current instance of an ERM-RER problem might serve as reference measure for the next instance. In this section, it is shown that this yields no benefit. Composing two successive ERM-REM problems boils down to a unique ERM-RER problem with the initial reference measure and a particular regularization factor. Under the assumption that $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the problem of interest is:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathrm{R}_z(P) + \alpha D(P \| P_{\Theta|Z=z}^{(Q,\lambda)}), \quad (59a)$$

$$\text{s. t.} \quad \int dP(\theta) = 1, \quad (59b)$$

where $\alpha > 0$; the reference measure $P_{\Theta|Z=z}^{(Q,\lambda)}$, which satisfies (25), is the solution of the ERM-RER problem in (19); and the functional $\mathbb{R}_z$ is defined in (18). From Theorem 3, the solution to the ERM-RER problem in (59), which is denoted by $P_{\Theta|Z=z}^{(P_{\Theta|Z=z}^{(Q,\lambda)}, \alpha)}$, satisfies for all $\theta \in \text{supp } Q$ that

$$\frac{dP_{\Theta|Z=z}^{(P_{\Theta|Z=z}^{(Q,\lambda)}, \alpha)}}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) = \exp \left( -K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z} \left( -\frac{1}{\alpha} \right) - \frac{1}{\alpha} \mathcal{L}_z(\theta) \right). \quad (60)$$

The log-partition functions $K_{Q,z}$ in (22) and $K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}$ in (60) are strongly related, as shown by the following lemma.

**Lemma 12:** The functions $K_{Q,z}$ in (22) and $K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}$ in (60) satisfy for all $t \in \mathbb{R}$,

$$K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}(t) = K_{Q,z}\left(t - \frac{1}{\lambda}\right) - K_{Q,z}\left(-\frac{1}{\lambda}\right). \quad (61)$$

Moreover, for all $t \le 0$,

$$K_{P_{\Theta|Z=z}^{(Q,\lambda)}, z}(t) \le 0. \quad (62)$$
**Proof:** The proof of (61) relies on the fact that for all $t \in \{\nu \in \mathbb{R} : K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(\nu) < \infty\}$, the function $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}$ in (60) satisfies

$$K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) = \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta}) \right) \quad (63)$$

$$= \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \frac{dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \right) \quad (64)$$

$$= \log \left( \int \exp\left(\left(t - \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}\left(-\frac{1}{\lambda}\right)\right) dQ(\boldsymbol{\theta}) \right) \quad (65)$$

$$= \log \left( \int \exp\left(\left(t - \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta}) - K_{Q,z}\left(-\frac{1}{\lambda}\right) \right) \quad (66)$$

$$= K_{Q,z}\left(t - \frac{1}{\lambda}\right) - K_{Q,z}\left(-\frac{1}{\lambda}\right), \quad (67)$$

where the equality in (65) follows from (25). Moreover, from Lemma 15, it follows that the function $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}$ is continuous and nondecreasing. Let $s^* \in \mathbb{R} \cup \{+\infty\}$ be defined by

$$s^* \triangleq \sup \left\{ \nu \in \mathbb{R} : K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(\nu) < \infty \right\}. \quad (68)$$

If $s^* = +\infty$, then for all $t \in \mathbb{R}$, $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) < +\infty$, and the proof of (61) is completed.

Alternatively, if $s^* < +\infty$, it follows that for all $t > s^*$, $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) = +\infty$, which implies that $K_{Q,z}(t - \frac{1}{\lambda}) = +\infty$, as the function $K_{Q,z}$ is also continuous (Lemma 15) and $K_{Q,z}(\frac{1}{\lambda}) < +\infty$ (due to the choice of $\lambda$). Hence, in this case, the equality in (61) is of the form $+\infty = +\infty$. This completes the proof of (61).

The proof of (62) follows by noticing that for all $t \le 0$ and for all $\theta \in \text{supp } Q$, it holds that $\exp(t \mathbf{L}_z(\boldsymbol{\theta})) \le 1$. Hence,

$$K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) = \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta}) \right) \quad (69)$$

$$\le \log \left( \int dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta}) \right) \quad (70)$$

$$= 0, \quad (71)$$

which completes the proof.

<figure><img src="image_2.png" /></figure>

The following lemma establishes that the solution to the ERM-RER problem in (59) is identical to the solution to another ERM-RER problem of the form

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathbb{R}_z(P) + \left( \frac{1}{\frac{1}{\alpha} + \frac{1}{\lambda}} \right) D(P \| Q), \quad (72a)$$

$$\text{s. t.} \quad \int dP(\boldsymbol{\theta}) = 1, \quad (72b)$$

with $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), and whose solution, denoted by $P^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}$, satisfies for all $\boldsymbol{\theta} \in \text{supp } Q$,

$$\frac{dP^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ} = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda} - \frac{1}{\alpha}\right) - \left(\frac{1}{\lambda} + \frac{1}{\alpha}\right) \mathbf{L}_z(\boldsymbol{\theta})\right). \quad (73)$$

The formal statement is as follows.

**Lemma 13:** Let $\alpha \in (0, +\infty)$ and $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23). Then, the probability measures $P^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})_{\Theta|Z=z}$ in (60) and $P^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}$ in (73) are identical.

**Proof:** For all $\boldsymbol{\theta} \in \text{supp } Q$,

$$\frac{dP^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})}{dQ}(\boldsymbol{\theta}) = \frac{dP^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})}{dQ}(\boldsymbol{\theta}) \frac{dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ}(\boldsymbol{\theta}) \quad (74)$$

$$= \exp \left( - K_{P^{(Q,\lambda)}_{\Theta|Z=z},z} \left( -\frac{1}{\alpha} \right) - K_{Q,z} \left( -\frac{1}{\lambda} \right) - \left( \frac{1}{\alpha} + \frac{1}{\lambda} \right) \mathbf{L}_z(\boldsymbol{\theta}) \right) \quad (75)$$

$$= \exp \left( - K_{Q,z} \left( -\frac{1}{\alpha} - \frac{1}{\lambda} \right) - \left( \frac{1}{\alpha} + \frac{1}{\lambda} \right) \mathbf{L}_z(\boldsymbol{\theta}) \right) \quad (76)$$

$$= \frac{dP^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ}(\boldsymbol{\theta}), \quad (77)$$

where the equality in (74) follows from the fact that the measure $P^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})$ is absolutely continuous with respect to $P^{(Q,\lambda)}_{\Theta|Z=z}$ and $P^{(Q,\lambda)}_{\Theta|Z=z}$ is absolutely continuous with respect to the measure $Q$; the equality in (75) follows from Lemma 12; and the equality in (77) follows from Theorem 3.

For all measurable subsets $\mathcal{A}$ of $\mathcal{M}$, the following holds:

$$P^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha)}(\mathcal{A}) = \int_{\mathcal{A}} \frac{dP^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \quad (78)$$

$$= \int_{\mathcal{A}} \frac{dP^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \quad (79)$$

$$= \int_{\mathcal{A}} dP^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}(\boldsymbol{\theta}) \quad (80)$$

$$= P^{(Q, \frac{1}{\lambda}+\frac{1}{\alpha})}_{\Theta|Z=z}(\mathcal{A}), \quad (81)$$

where the equality in (79) follows from (77). This completes the proof.

<figure><img src="image_3.png" /></figure>
**Proof:** The proof of (61) relies on the fact that for all $t \in \{\nu \in \mathbb{R} : K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(\nu) < \infty\}$, the function $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}$ in (60) satisfies

$$K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) = \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta}) \right) \quad (63)$$

$$= \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \frac{dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \right) \quad (64)$$

$$= \log \left( \int \exp\left(\left(t - \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}\left(-\frac{1}{\lambda}\right)\right) dQ(\boldsymbol{\theta}) \right) \quad (65)$$

$$= \log \left( \int \exp\left(\left(t - \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta}) - K_{Q,z}\left(-\frac{1}{\lambda}\right) \right) \quad (66)$$

$$= K_{Q,z}\left(t - \frac{1}{\lambda}\right) - K_{Q,z}\left(-\frac{1}{\lambda}\right), \quad (67)$$

where the equality in (65) follows from (25). Moreover, from Lemma 15, it follows that the function $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}$ is continuous and nondecreasing. Let $s^* \in \mathbb{R} \cup \{+\infty\}$ be defined by

$$s^* \triangleq \sup \left\{ \nu \in \mathbb{R} : K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(\nu) < \infty \right\}. \quad (68)$$

If $s^* = +\infty$, then for all $t \in \mathbb{R}$, $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) < +\infty$, and the proof of (61) is completed.

Alternatively, if $s^* < +\infty$, it follows that for all $t > s^*$, $K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) = +\infty$, which implies that $K_{Q,z}(t - \frac{1}{\lambda}) = +\infty$, as the function $K_{Q,z}$ is also continuous (Lemma 15) and $K_{Q,z}(t - \frac{1}{\lambda}) < +\infty$ (due to the choice of $\lambda$). Hence, in this case, the equality in (61) is of the form $+\infty = +\infty$. This completes the proof of (61).

The proof of (62) follows by noticing that for all $t \le 0$ and for all $\theta \in \text{supp} Q$, it holds that $\exp(t \mathbf{L}_z(\boldsymbol{\theta})) \le 1$. Hence,

$$K_{P^{(Q,\lambda)}_{\Theta|Z=z},z}(t) = \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta}) \right) \quad (69)$$

$$\le \log \left( \int dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta}) \right) \quad (70)$$

$$= 0, \quad (71)$$

which completes the proof.

<figure><img src="image_3.png" /></figure>

The following lemma establishes that the solution to the ERM-RER problem in (59) is identical to the solution to another ERM-RER problem of the form

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathbb{R}_z(P) + \left( \frac{1}{\frac{1}{\alpha} + \frac{1}{\lambda}} \right) D(P \| Q), \quad (72a)$$

$$\text{s. t.} \quad \int dP(\boldsymbol{\theta}) = 1, \quad (72b)$$

with $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), and whose solution, denoted by $P^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}$, satisfies for all $\boldsymbol{\theta} \in \text{supp} Q$,

$$\frac{dP^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ} = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda} - \frac{1}{\alpha}\right) - \left(\frac{1}{\lambda} + \frac{1}{\alpha}\right) \mathbf{L}_z(\boldsymbol{\theta})\right). \quad (73)$$

The formal statement is as follows.

**Lemma 13:** Let $\alpha \in (0, +\infty)$ and $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23). Then, the probability measures $P^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})_{\Theta|Z=z}$ in (60) and $P^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}$ in (73) are identical.

**Proof:** For all $\boldsymbol{\theta} \in \text{supp} Q$,

$$\frac{dP^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})}{dQ}(\boldsymbol{\theta}) = \frac{dP^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})}{dQ}(\boldsymbol{\theta}) \frac{dP^{(Q,\lambda)}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ}(\boldsymbol{\theta}) \quad (74)$$

$$= \exp \left( - K_{P^{(Q,\lambda)}_{\Theta|Z=z},z} \left( -\frac{1}{\alpha} \right) - K_{Q,z} \left( -\frac{1}{\lambda} \right) - \left( \frac{1}{\alpha} + \frac{1}{\lambda} \right) \mathbf{L}_z(\boldsymbol{\theta}) \right) \quad (75)$$

$$= \exp \left( - K_{Q,z} \left( -\frac{1}{\alpha} - \frac{1}{\lambda} \right) - \left( \frac{1}{\alpha} + \frac{1}{\lambda} \right) \mathbf{L}_z(\boldsymbol{\theta}) \right) \quad (76)$$

$$= \frac{dP^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}(\boldsymbol{\theta})}{dQ}(\boldsymbol{\theta}), \quad (77)$$

where the equality in (74) follows from the fact that the measure $P^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha})$ is absolutely continuous with respect to $P^{(Q,\lambda)}_{\Theta|Z=z}$ and $P^{(Q,\lambda)}_{\Theta|Z=z}$ is absolutely continuous with respect to the measure $Q$; the equality in (75) follows from Lemma 12; and the equality in (77) follows from Theorem 3.

For all measurable subsets $\mathcal{A}$ of $\mathcal{M}$, the following holds:

$$P^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha)}(\mathcal{A}) = \int_{\mathcal{A}} \frac{dP^{(P^{(Q,\lambda)}_{\Theta|Z=z}, \alpha)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \quad (78)$$

$$= \int_{\mathcal{A}} \frac{dP^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \quad (79)$$

$$= \int_{\mathcal{A}} dP^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}(\boldsymbol{\theta}) \quad (80)$$

$$= P^{(Q, \frac{1}{\lambda} + \frac{1}{\alpha})}_{\Theta|Z=z}(\mathcal{A}), \quad (81)$$

where the equality in (79) follows from (77). This completes the proof.

<figure><img src="image_4.png" /></figure>
The following theorem establishes a relation between the solutions to the following optimization problems

$$
\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathrm{R}_z(P), \qquad (82\mathrm{a})
$$

$$
\text{s. t.} \quad D(P \| P_{\Theta|Z=z}^{(Q,\lambda)}) \le c, \text{ and} \quad (82\text{b})
$$

$$
\int dP(\boldsymbol{\theta}) = 1, \tag{82c}
$$

and

$$
\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathbf{R}_z(P) + \omega D(P \| Q), \quad (83\text{a})
$$

$$
\text{s. t.} \quad \int dP(\boldsymbol{\theta}) = 1, \qquad (83\text{b})
$$

with $c > 0$ and $\omega \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), two constants; $P_{\Theta|Z=z}^{(Q,\lambda)}$ the probability measure in (25); and $\mathbf{R}_z$ the functional in (18).

From Theorem 3, the solution to the ERM-RER problem in (83), which is denoted by $P_{\Theta|Z=z}^{(Q,\omega)}$, satisfies for all $\theta \in \text{supp } Q$ that

$$
\frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\omega)}}{\mathrm{d}Q}(\boldsymbol{\theta}) = \exp\left(-K_{Q,z}\left(-\frac{1}{\omega}\right) - \frac{1}{\omega}\mathrm{L}_z(\boldsymbol{\theta})\right), \quad (84)
$$

where the function $K_{Q,z}$ is in (22).

The following theorem formalizes the relation between both
optimization problems.

Theorem 4: Assume that $c$ and $\omega$ in (82) and (83) satisfy

$$
D \left( P_{\Theta|Z=z}^{(Q,\omega)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) = c, \quad (85)
$$

with $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $P_{\Theta|Z=z}^{(Q,\omega)}$ being the probability measures
in (25) and (84), respectively. Then, the solution to the opti-
mization problem in (82) is the probability measure $P_{\Theta|Z=z}^{(Q,\omega)}$.

Proof: The proof is presented in Appendix J.

V. THE LOG-PARTITION FUNCTION

This section introduces some properties of the log-partition
function $K_{Q,z}$ in (22) using the notion of separable empirical
risk functions.

A. Separable Empirical Risk Functions

Separable empirical risk functions are defined with respect to a measure $P \in \Delta(\mathcal{M})$.

Definition 5 (Separable Empirical Risk Function): The em-
pirical risk function L<sub>z</sub> in (3) is said to be separable with
respect to a &sigma;-finite measure P ∈ &Delta;(M), if there exist a
positive real c > 0 and two subsets A and B of M that are
nonnegligible with respect to P, and for all (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A &times; B,

$$
L_z(\theta_1) < c < L_z(\theta_2) < +\infty. \quad (86)
$$

In a nutshell, a nonseparable empirical risk function with
respect to the measure *Q* is a constant almost surely. More
specifically, there exists a real *a* ≥ 0, such that

$$
Q(\{\theta \in \mathcal{M} : L_z(\theta) = a\}) = 1. \quad (87)
$$

From this perspective, nonseparable empirical risk functions
exhibit little practical interest for model selection.

The definition of separability in Definition 5 and Lemma 3
lead to the following lemma.

*Lemma 14:* The empirical risk function L<sub>z</sub> in (3) is separable with respect to the &sigma;-finite measure Q in (19) if and only if it is separable with respect to the probability measure P<sub>&Theta;|Z=z</sub><sup>(Q,&lambda;)</sup> in (25).

*Proof:* Consider first that the function L<sub>z</sub> is separable with respect to the σ-finite measure Q. Hence, there exist a positive real c > 0 and two subsets A and B of M that are nonnegligible with respect to Q, such that for all (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A × B the inequality in (86) holds. Hence, from (86) the following inequalities hold:

$$
-\frac{1}{\lambda} \mathrm{L}_z(\theta_1) > -\frac{c}{\lambda} > -\frac{1}{\lambda} \mathrm{L}_z(\theta_2) > -\infty, \text{ and} \quad (88)
$$

$$
\exp\left(-\frac{1}{\lambda}\mathrm{L}_z(\theta_1)\right) > \exp\left(-\frac{c}{\lambda}\right) > \exp\left(-\frac{1}{\lambda}\mathrm{L}_z(\theta_2)\right) > 0. \quad (89)
$$

This implies that

$$
\frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\theta_1) > \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{c}{\lambda}\right) \quad (90)
$$

$$
> \frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\theta_2) \qquad (91)
$$

$$
> 0. \tag{92}
$$

Using the inequality in (90) and the facts that $Q(\mathcal{A}) > 0$ and $Q(\mathcal{B}) > 0$, the following holds

$$
P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{A}) = \int_{\mathcal{A}} \frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\boldsymbol{\theta}) \mathrm{d}Q(\boldsymbol{\theta}) > 0, \quad (93)
$$

and

$$
P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{B}) = \int_{\mathcal{B}} \frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\boldsymbol{\theta}) \mathrm{d}Q(\boldsymbol{\theta}) > 0. \quad (94)
$$

which implies that the function L<sub>z</sub> is separable with respect
to the probability measure P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup>.

Consider now that the function L<sub>z</sub> is separable with respect to the probability measure P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup>. Hence, there exist a positive real c > 0 and two subsets A and B of M that are nonnegligible with respect to P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup>, such that for all (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A × B the inequality in (86) holds. More specifically, P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (A) > 0 and P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (B) > 0. From Lemma 2 and the inequality in (86), it follows that for all pairs (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A × B, dP<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> / dQ (θ<sub>1</sub>) > 0 and dP<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> / dQ (θ<sub>2</sub>) > 0. Hence, from the fact that P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (A) > 0 and P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (B) > 0, it follows that Q(A) > 0 and Q(B) > 0, which implies that the function L<sub>z</sub> is separable with respect to the σ-finite measure Q. This completes the proof.

Lemma 14 shows that separable empirical risk functions, and
only these functions, lead to ERM-RER-optimal probability
measures from which models are sampled with different
probabilities. For the case of nonseparable empirical risk
The following theorem establishes a relation between the solutions to the following optimization problems

$$
\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathrm{R}_z(P), \qquad (82\mathrm{a})
$$

$$
\text{s. t.} \quad D(P \| P_{\Theta|Z=z}^{(Q,\lambda)}) \le c, \text{ and} \quad (82\text{b})
$$

$$
\int dP(\boldsymbol{\theta}) = 1, \qquad (82c)
$$

and

$$
\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \mathbf{R}_z (P) + \omega D(P \| Q), \quad (83\text{a})
$$

$$
\text{s. t.} \quad \int dP(\boldsymbol{\theta}) = 1, \qquad (83\text{b})
$$

with $c > 0$ and $\omega \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), two constants; $P_{\Theta|Z=z}^{(Q,\lambda)}$ the probability measure in (25); and $\mathbf{R}_z$ the functional in (18).

From Theorem 3, the solution to the ERM-RER problem in (83), which is denoted by $P_{\Theta|Z=z}^{(Q,\omega)}$, satisfies for all $\theta \in \text{supp } Q$ that

$$
\frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\omega)}}{\mathrm{d}Q}(\boldsymbol{\theta}) = \exp\left(-K_{Q,z}\left(-\frac{1}{\omega}\right) - \frac{1}{\omega}\mathrm{L}_z(\boldsymbol{\theta})\right), \quad (84)
$$

where the function $K_{Q,z}$ is in (22).

The following theorem formalizes the relation between both
optimization problems.

Theorem 4: Assume that $c$ and $\omega$ in (82) and (83) satisfy

$$
D \left( P_{\Theta|Z=z}^{(Q,\omega)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) = c, \quad (85)
$$

with $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $P_{\Theta|Z=z}^{(Q,\omega)}$ being the probability measures
in (25) and (84), respectively. Then, the solution to the opti-
mization problem in (82) is the probability measure $P_{\Theta|Z=z}^{(Q,\omega)}$.

Proof: The proof is presented in Appendix J.

V. THE LOG-PARTITION FUNCTION

This section introduces some properties of the log-partition
function $K_{Q,z}$ in (22) using the notion of separable empirical
risk functions.

A. Separable Empirical Risk Functions

Separable empirical risk functions are defined with respect to a measure $P \in \Delta(\mathcal{M})$.

Definition 5 (Separable Empirical Risk Function): The em-
pirical risk function L<sub>z</sub> in (3) is said to be separable with
respect to a &sigma;-finite measure P ∈ &Delta;(M), if there exist a
positive real c > 0 and two subsets A and B of M that are
nonnegligible with respect to P, and for all (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A &times; B,

$$
\mathbf{L}_z(\theta_1) < c < \mathbf{L}_z(\theta_2) < +\infty. \quad (86)
$$

In a nutshell, a nonseparable empirical risk function with
respect to the measure *Q* is a constant almost surely. More
specifically, there exists a real *a* ≥ 0, such that

$$
Q(\{\theta \in \mathcal{M} : \mathbf{L}_z(\theta) = a\}) = 1. \quad (87)
$$

From this perspective, nonseparable empirical risk functions
exhibit little practical interest for model selection.

The definition of separability in Definition 5 and Lemma 3
lead to the following lemma.

*Lemma 14:* The empirical risk function L<sub>z</sub> in (3) is separable with respect to the σ-finite measure Q in (19) if and only if it is separable with respect to the probability measure P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> in (25).

*Proof:* Consider first that the function L<sub>z</sub> is separable with respect to the σ-finite measure Q. Hence, there exist a positive real c > 0 and two subsets A and B of M that are nonnegligible with respect to Q, such that for all (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A × B the inequality in (86) holds. Hence, from (86) the following inequalities hold:

$$
-\frac{1}{\lambda}\mathbf{L}_z(\theta_1) > -\frac{c}{\lambda} > -\frac{1}{\lambda}\mathbf{L}_z(\theta_2) > -\infty, \text{ and} \quad (88)
$$

$$
\exp\left(-\frac{1}{\lambda}\mathbf{L}_z(\theta_1)\right) > \exp\left(-\frac{c}{\lambda}\right) > \exp\left(-\frac{1}{\lambda}\mathbf{L}_z(\theta_2)\right) > 0. (89)
$$

This implies that

$$
\frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\theta_1) > \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{c}{\lambda}\right) \quad (90)
$$

$$
> \frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\theta_2) \qquad (91)
$$

$$
> 0. \tag{92}
$$

Using the inequality in (90) and the facts that $Q(\mathcal{A}) > 0$ and $Q(\mathcal{B}) > 0$, the following holds

$$
P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{A}) = \int_{\mathcal{A}} \frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\boldsymbol{\theta}) \mathrm{d}Q(\boldsymbol{\theta}) > 0, \quad (93)
$$

and

$$
P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{B}) = \int_{\mathcal{B}} \frac{\mathrm{d}P_{\Theta|Z=z}^{(Q,\lambda)}}{\mathrm{d}Q}(\boldsymbol{\theta}) \mathrm{d}Q(\boldsymbol{\theta}) > 0. \quad (94)
$$

which implies that the function L<sub>z</sub> is separable with respect
to the probability measure P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup>.

Consider now that the function L<sub>z</sub> is separable with respect to the probability measure P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup>. Hence, there exist a positive real c > 0 and two subsets A and B of M that are nonnegligible with respect to P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup>, such that for all (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A × B the inequality in (86) holds. More specifically, P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (A) > 0 and P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (B) > 0. From Lemma 2 and the inequality in (86), it follows that for all pairs (θ<sub>1</sub>, θ<sub>2</sub>) ∈ A × B, dP<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> / dQ (θ<sub>1</sub>) > 0 and dP<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> / dQ (θ<sub>2</sub>) > 0. Hence, from the fact that P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (A) > 0 and P<sub>Θ|Z=z</sub><sup>(Q,λ)</sup> (B) > 0, it follows that Q(A) > 0 and Q(B) > 0, which implies that the function L<sub>z</sub> is separable with respect to the σ-finite measure Q. This completes the proof.

Lemma 14 shows that separable empirical risk functions, and
only these functions, lead to ERM-RER-optimal probability
measures from which models are sampled with different
probabilities. For the case of nonseparable empirical risk
functions, all models are sampled from the ERM-RER-optimal probability measure with the same probability.

## B. Properties of the Log-Partition Function

The log-partition function $K_{Q,z}$ in (22) is a nondecreasing continuous convex function as shown by the following lemmas.

**Lemma 15:** The function $K_{Q,z}$ in (22) is nondecreasing and differentiable infinitely many times in the interior of $\{t \in \mathbb{R} : K_{Q,z}(t) < +\infty\}$.

*Proof:* The proof is presented in Appendix K. ■

**Lemma 16:** The function $K_{Q,z}$ in (22) is convex in $\{t \in \mathbb{R} : K_{Q,z}(t) < +\infty\}$. Moreover, it is strictly convex if and only if the empirical risk function $L_z$ in (3) is separable with respect to the $\sigma$-finite measure $Q$ in (19).

*Proof:* The proof is presented in Appendix L. ■

In Lemma 15, it has been established that the log-partition function $K_{Q,z}$ in (22) is differentiable infinitely many times in the interval $\{t \in \mathbb{R} : K_{Q,z}(t) < +\infty\}$. Let the $m$-th derivative of the function $K_{Q,z}$ in (22) be denoted by $K_{Q,z}^{(m)} : \mathbb{R} \to \mathbb{R}$, with $m \in \mathbb{N}$. Hence, for all $s \in K_{Q,z}$,

$$K_{Q,z}^{(m)}\left(-\frac{1}{s}\right) \triangleq \frac{d^m}{dt^m} K_{Q,z}(t) \Big|_{t=-\frac{1}{s}}. \quad (95)$$

The following lemma provides explicit expressions for the first, second and third derivatives of the function $K_{Q,z}$ in (22).

**Lemma 17:** The first, second and third derivatives of the function $K_{Q,z}$ in (22), denoted respectively by $K_{Q,z}^{(1)}$, $K_{Q,z}^{(2)}$, and $K_{Q,z}^{(3)}$, satisfy for all $\lambda \in \text{int} K_{Q,z}$, with $K_{Q,z}$ in (23),

$$K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) = \int L_z(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta), \quad (96)$$

$$K_{Q,z}^{(2)}\left(-\frac{1}{\lambda}\right) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) \right)^2 dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta), \quad (97)$$

$$K_{Q,z}^{(3)}\left(-\frac{1}{\lambda}\right) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) \right)^3 dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta), \quad (98)$$

where the function $L_z$ is defined in (3) and the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ satisfies (25).

*Proof:* The proof is presented in Appendix M. ■

From Lemma 17, it follows that if $\Theta \sim P_{\Theta|Z=z}^{(Q,\lambda)}$, with $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), the random variable

$$W \triangleq L_z(\Theta), \quad (99)$$

with the function $L_z$ in (3), possesses a mean, variance, and third cumulant that are equivalent to $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ in (96), $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ in (97), and $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ in (98), respectively.

Note that if there exists a $\delta > 0$ such that the log-partition function $K_{Q,z}$ is differentiable within the open interval $(-\delta, \delta)$ and $Q$ in (19) is a probability measure, the function $K_{Q,z}$

is the cumulant generating function of the random variable

$$V \triangleq L_z(\Theta), \text{ with } \Theta \sim Q. \quad (100)$$

The following lemma leverages this observation.

**Lemma 18:** Assume that $Q$ in (19) is a probability measure and that there exists real $\delta > 0$ such that the log-partition function $K_{Q,z}$ in (22) is differentiable within $(-\delta, \delta)$. Then, the first, second and third derivatives of $K_{Q,z}$, denoted respectively by $K_{Q,z}^{(1)}$, $K_{Q,z}^{(2)}$, and $K_{Q,z}^{(3)}$, satisfy

$$K_{Q,z}^{(1)}(0) = \int L_z(\theta) dQ(\theta), \quad (101)$$

$$K_{Q,z}^{(2)}(0) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}(0) \right)^2 dQ(\theta), \quad (102)$$

$$K_{Q,z}^{(3)}(0) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}(0) \right)^3 dQ(\theta), \quad (103)$$

where the function $L_z$ is defined in (3).

*Proof:* The proof follows along the same arguments of the proof of Lemma 17. ■

The mean, variance, and third cumulant of the random variable $V$ in (100) are $K_{Q,z}^{(1)}(0)$ in (101), $K_{Q,z}^{(2)}(0)$ in (102), and $K_{Q,z}^{(3)}(0)$ in (103), respectively.

## VI. EXPECTATION OF THE EMPIRICAL RISK

The mean of the random variable $W$ in (99) is equivalent to the expectation of the empirical risk function $L_z$ with respect to the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), which is equal to $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$, with the functional $R_z$ in (18). Often, $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is referred to as the ERM-RER-optimal expected empirical risk to emphasize that this is the expected value of the empirical risk when models are sampled from the solution of the ERM-RER problem in (19). The following corollary of Lemma 17 formalizes this observation.

**Corollary 2:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) verifies that

$$R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right), \quad (104)$$

where the functional $R_z$ and the function $K_{Q,z}^{(1)}$ are defined in (18) and (96), respectively.

The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) exhibits the following property.

**Theorem 5:** The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) is nondecreasing with $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23). Moreover, $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is strictly increasing with $\lambda \in K_{Q,z}$ if and only if the function $L_z$ in (3) is separable with respect to the measure $Q$.

*Proof:* The proof is presented in Appendix N. ■

The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) has been shown to be nondecreasing with $\lambda$ in [9, Appendix E.4] for the special case in which $Q$ is a probability measure.
functions, all models are sampled from the ERM-RER-optimal probability measure with the same probability.

## B. Properties of the Log-Partition Function

The log-partition function $K_{Q,z}$ in (22) is a nondecreasing continuous convex function as shown by the following lemmas.

**Lemma 15:** The function $K_{Q,z}$ in (22) is nondecreasing and differentiable infinitely many times in the interior of $\{t \in \mathbb{R} : K_{Q,z}(t) < +\infty\}$.

*Proof:* The proof is presented in Appendix K.

**Lemma 16:** The function $K_{Q,z}$ in (22) is convex in $\{t \in \mathbb{R} : K_{Q,z}(t) < +\infty\}$. Moreover, it is strictly convex if and only if the empirical risk function $L_z$ in (3) is separable with respect to the $\sigma$-finite measure $Q$ in (19).

*Proof:* The proof is presented in Appendix L.

In Lemma 15, it has been established that the log-partition function $K_{Q,z}$ in (22) is differentiable infinitely many times in the interval $\{t \in \mathbb{R} : K_{Q,z}(t) < +\infty\}$. Let the $m$-th derivative of the function $K_{Q,z}$ in (22) be denoted by $K_{Q,z}^{(m)} : \mathbb{R} \to \mathbb{R}$, with $m \in \mathbb{N}$. Hence, for all $s \in K_{Q,z}$,

$$K_{Q,z}^{(m)}\left(-\frac{1}{s}\right) \triangleq \frac{d^m}{dt^m} K_{Q,z}(t)\bigg|_{t=-\frac{1}{s}} \quad (95)$$

The following lemma provides explicit expressions for the first, second and third derivatives of the function $K_{Q,z}$ in (22).

**Lemma 17:** The first, second and third derivatives of the function $K_{Q,z}$ in (22), denoted respectively by $K_{Q,z}^{(1)}$, $K_{Q,z}^{(2)}$, and $K_{Q,z}^{(3)}$, satisfy for all $\lambda \in \text{int} K_{Q,z}$, with $K_{Q,z}$ in (23),

$$K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) = \int L_z(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta), \quad (96)$$

$$K_{Q,z}^{(2)}\left(-\frac{1}{\lambda}\right) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) \right)^2 dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta), \quad (97)$$

$$K_{Q,z}^{(3)}\left(-\frac{1}{\lambda}\right) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) \right)^3 dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta), \quad (98)$$

where the function $L_z$ is defined in (3) and the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ satisfies (25).

*Proof:* The proof is presented in Appendix M.

From Lemma 17, it follows that if $\Theta \sim P_{\Theta|Z=z}^{(Q,\lambda)}$, with $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), the random variable

$$W \triangleq L_z(\Theta), \quad (99)$$

with the function $L_z$ in (3), possesses a mean, variance, and third cumulant that are equivalent to $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ in (96), $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ in (97), and $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ in (98), respectively.

Note that if there exists a $\delta > 0$ such that the log-partition function $K_{Q,z}$ is differentiable within the open interval $(-\delta, \delta)$ and $Q$ in (19) is a probability measure, the function $K_{Q,z}$

is the cumulant generating function of the random variable

$$V \triangleq L_z(\Theta), \text{ with } \Theta \sim Q. \quad (100)$$

The following lemma leverages this observation.

**Lemma 18:** Assume that $Q$ in (19) is a probability measure and that there exists real $\delta > 0$ such that the log-partition function $K_{Q,z}$ in (22) is differentiable within $(-\delta, \delta)$. Then, the first, second and third derivatives of $K_{Q,z}$, denoted respectively by $K_{Q,z}^{(1)}$, $K_{Q,z}^{(2)}$, and $K_{Q,z}^{(3)}$, satisfy

$$K_{Q,z}^{(1)}(0) = \int L_z(\theta) dQ(\theta), \quad (101)$$

$$K_{Q,z}^{(2)}(0) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}(0) \right)^2 dQ(\theta), \quad (102)$$

$$K_{Q,z}^{(3)}(0) = \int \left( L_z(\theta) - K_{Q,z}^{(1)}(0) \right)^3 dQ(\theta), \quad (103)$$

where the function $L_z$ is defined in (3).

*Proof:* The proof follows along the same arguments of the proof of Lemma 17.

The mean, variance, and third cumulant of the random variable $V$ in (100) are $K_{Q,z}^{(1)}(0)$ in (101), $K_{Q,z}^{(2)}(0)$ in (102), and $K_{Q,z}^{(3)}(0)$ in (103), respectively.

## VI. EXPECTATION OF THE EMPIRICAL RISK

The mean of the random variable $W$ in (99) is equivalent to the expectation of the empirical risk function $L_z$ with respect to the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), which is equal to $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$, with the functional $R_z$ in (18). Often, $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is referred to as the ERM-RER-optimal expected empirical risk to emphasize that this is the expected value of the empirical risk when models are sampled from the solution of the ERM-RER problem in (19). The following corollary of Lemma 17 formalizes this observation.

**Corollary 2:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) verifies that

$$R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right), \quad (104)$$

where the functional $R_z$ and the function $K_{Q,z}^{(1)}$ are defined in (18) and (96), respectively.

The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) exhibits the following property.

**Theorem 5:** The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) is nondecreasing with $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23). Moreover, $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is strictly increasing with $\lambda \in K_{Q,z}$ if and only if the function $L_z$ in (3) is separable with respect to the measure $Q$.

*Proof:* The proof is presented in Appendix N.

The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) has been shown to be nondecreasing with $\lambda$ in [9, Appendix E.4] for the special case in which $Q$ is a probability measure.
A question that arises from Theorem 5 is whether the value $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) can be made arbitrarily close to $\delta_{Q,z}^*$, with $\delta_{Q,z}^*$ in (37), by making $\lambda$ arbitrarily small. The following lemma shows that the value $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is often bounded away from $\delta_{Q,z}^*$, even for arbitrarily small values of $\lambda$.

**Lemma 19:** The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) satisfies,

$$R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \geq \delta_{Q,z}^*, \quad (105)$$

where $\delta_{Q,z}^*$ is defined in (37). Moreover, the inequality in (105) is strict if and only if the function $L_z$ in (3) is separable with respect to the measure $Q$ in (19).

*Proof:* The proof is presented in Appendix O. ■

In the asymptotic regime when $\lambda$ tends to zero, the expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) is equal to $\delta_{Q,z}^*$, as shown by the following lemma.

**Theorem 6:** The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) satisfies,

$$\lim_{\lambda \to 0^+} R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = \delta_{Q,z}^*, \quad (106)$$

where $\delta_{Q,z}^*$ is defined in (37).

*Proof:* The proof is presented in Appendix P. ■

The following lemma determines the value of the objective function of the ERM-RER problem in (19) when it is evaluated at its solution. This result appeared first in [11, Lemma 3].

**Lemma 20 (Lemma 3 in [11]):** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the $\sigma$-finite measure $Q$ in (19) satisfy

$$-\lambda K_{Q,z} \left(-\frac{1}{\lambda}\right) = R_z \left(P_{\Theta|Z=z}^{(Q,\lambda)}\right) + \lambda D \left(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q \right) \quad (107)$$

$$= R_z(Q) - \lambda D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) \quad (108)$$

where the functional $R_z$ is defined in (18); and the function $K_{Q,z}$ is defined in (22).

*Proof:* From Theorem 3, it follows that for all $\theta \in \text{supp} Q$,

$$\log \left( \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \right) = -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda} L_z(\theta) \quad (109)$$

where the function $L_z$ is defined in (3). Thus,

$$D(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) = \int \log \left( \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \right) d P_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (110)$$

$$= -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda} \int L_z(\theta) d P_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (111)$$

$$= -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda} R_z(P_{\Theta|Z=z}^{(Q,\lambda)}), \quad (112)$$

where the functional $R_z$ is defined in (18). This completes the proof of (107).

From Lemma 3 and (109), it follows that

$$D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) = - \int \log \left( \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \right) dQ(\theta) \quad (113)$$

$$= K_{Q,z} \left(-\frac{1}{\lambda}\right) + \frac{1}{\lambda} \int L_z(\theta) dQ(\theta) \quad (114)$$

$$= K_{Q,z} \left(-\frac{1}{\lambda}\right) + \frac{1}{\lambda} R_z(Q), \quad (115)$$

which completes the proof of (108). ■

The following corollary of Lemma 20 characterizes the difference between the expected values of the random variables $W$ and $V$ in (99) and (100), respectively.

**Corollary 3:** If measures $Q$ and $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) are both probability measures, then,

$$R_z(Q) - R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = \lambda \left( D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) + D(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) \right). \quad (116)$$

The right-hand side of (116) is a symmetrized Kullback-Liebler divergence, also known as Jeffrey's divergence [69], between the measures $Q$ and $P_{\Theta|Z=z}^{(Q,\lambda)}$. More importantly, when $Q$ is a probability measure, it follows that $D(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) \ge 0$ and $D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) \ge 0$, which leads to the following corollary from Lemma 20.

**Corollary 4:** If the $\sigma$-finite measure $Q$ in (19) is a probability measure, then, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies

$$R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \le R_z(Q), \quad (117)$$

where, the functional $R_z$ is defined in (18).

## VII. VARIANCE OF THE EMPIRICAL RISK

In Lemma 15, it has been established that if there exists a $\delta > 0$ such that the log-partition function $K_{Q,z}$ in (22) is finite within the open interval $(-\delta, \delta)$, the log-partition function $K_{Q,z}$ is differentiable infinitely many times within the interval $(-\infty, \delta)$. This together with the mean value theorem [70, Theorem 5.10] lead to the following characterization of the differences of the values $K_{Q,z}^{(2)}(-\frac{1}{t})$ and $K_{Q,z}^{(2)}(0)$, with $t > 0$.

**Lemma 21:** If the measure $Q$ in (19) is a probability measure and there exists a $\delta > 0$ such that the function $K_{Q,z}$ in (22) is differentiable within the open interval $(-\delta, \delta)$, then for all $t > 0$,

$$K_{Q,z}^{(2)} \left(-\frac{1}{t}\right) - K_{Q,z}^{(2)}(0) = -\frac{1}{t} K_{Q,z}^{(3)} \left(-\frac{1}{\beta}\right) < +\infty, \quad (118)$$

for some $\beta \in (t, +\infty)$, where the functions $K_{Q,z}^{(2)}$ and $K_{Q,z}^{(3)}$ are defined in (95).

*Proof:* The proof is an immediate consequence of Lemma 15 and the mean value theorem [70, Theorem 5.10]. ■
A question that arises from Theorem 5 is whether the value $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) can be made arbitrarily close to $\delta_{Q,z}^*$, with $\delta_{Q,z}^*$ in (37), by making $\lambda$ arbitrarily small. The following lemma shows that the value $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is often bounded away from $\delta_{Q,z}^*$, even for arbitrarily small values of $\lambda$.

**Lemma 19:** The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) satisfies,

$$R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \ge \delta_{Q,z}^*, \quad (105)$$

where $\delta_{Q,z}^*$ is defined in (37). Moreover, the inequality in (105) is strict if and only if the function $L_z$ in (3) is separable with respect to the measure $Q$ in (19).

*Proof:* The proof is presented in Appendix O. ■

In the asymptotic regime when $\lambda$ tends to zero, the expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) is equal to $\delta_{Q,z}^*$, as shown by the following lemma.

**Theorem 6:** The expected empirical risk $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ in (104) satisfies,

$$\lim_{\lambda \to 0^+} R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = \delta_{Q,z}^*, \quad (106)$$

where $\delta_{Q,z}^*$ is defined in (37).

*Proof:* The proof is presented in Appendix P. ■

The following lemma determines the value of the objective function of the ERM-RER problem in (19) when it is evaluated at its solution. This result appeared first in [11, Lemma 3].

**Lemma 20 (Lemma 3 in [11]):** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) and the $\sigma$-finite measure $Q$ in (19) satisfy

$$-\lambda K_{Q,z} \left(-\frac{1}{\lambda}\right) = R_z \left(P_{\Theta|Z=z}^{(Q,\lambda)}\right) + \lambda D \left(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q \right) \quad (107)$$

$$= R_z(Q) - \lambda D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) \quad (108)$$

where the functional $R_z$ is defined in (18); and the function $K_{Q,z}$ is defined in (22).

*Proof:* From Theorem 3, it follows that for all $\theta \in \text{supp} Q$,

$$\log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \right) = -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda} L_z(\theta) \quad (109)$$

where the function $L_z$ is defined in (3). Thus,

$$D(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) = \int \log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \right) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (110)$$

$$= -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda} \int L_z(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (111)$$

$$= -K_{Q,z} \left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda} R_z(P_{\Theta|Z=z}^{(Q,\lambda)}), \quad (112)$$

where the functional $R_z$ is defined in (18). This completes the proof of (107).

From Lemma 3 and (109), it follows that

$$D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) = - \int \log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \right) dQ(\theta) \quad (113)$$

$$= K_{Q,z} \left(-\frac{1}{\lambda}\right) + \frac{1}{\lambda} \int L_z(\theta) dQ(\theta) \quad (114)$$

$$= K_{Q,z} \left(-\frac{1}{\lambda}\right) + \frac{1}{\lambda} R_z(Q), \quad (115)$$

which completes the proof of (108). ■

The following corollary of Lemma 20 characterizes the difference between the expected values of the random variables $W$ and $V$ in (99) and (100), respectively.

**Corollary 3:** If measures $Q$ and $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) are both probability measures, then,

$$R_z(Q) - R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = \lambda \left( D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) + D(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) \right). \quad (116)$$

The right-hand side of (116) is a symmetrized Kullback-Liebler divergence, also known as Jeffrey's divergence [69], between the measures $Q$ and $P_{\Theta|Z=z}^{(Q,\lambda)}$. More importantly, when $Q$ is a probability measure, it follows that $D(P_{\Theta|Z=z}^{(Q,\lambda)} \| Q) \ge 0$ and $D(Q \| P_{\Theta|Z=z}^{(Q,\lambda)}) \ge 0$, which leads to the following corollary from Lemma 20.

**Corollary 4:** If the $\sigma$-finite measure $Q$ in (19) is a probability measure, then, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies

$$R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \le R_z(Q), \quad (117)$$

where, the functional $R_z$ is defined in (18).

## VII. VARIANCE OF THE EMPIRICAL RISK

In Lemma 15, it has been established that if there exists a $\delta > 0$ such that the log-partition function $K_{Q,z}$ in (22) is finite within the open interval $(-\delta, \delta)$, the log-partition function $K_{Q,z}$ is differentiable infinitely many times within the interval $(-\infty, \delta)$. This together with the mean value theorem [70, Theorem 5.10] lead to the following characterization of the differences of the values $K_{Q,z}^{(2)}(-\frac{1}{t})$ and $K_{Q,z}^{(2)}(0)$, with $t > 0$.

**Lemma 21:** If the measure $Q$ in (19) is a probability measure and there exists a $\delta > 0$ such that the function $K_{Q,z}$ in (22) is differentiable within the open interval $(-\delta, \delta)$, then for all $t > 0$,

$$K_{Q,z}^{(2)} \left(-\frac{1}{t}\right) - K_{Q,z}^{(2)}(0) = -\frac{1}{t} K_{Q,z}^{(3)} \left(-\frac{1}{\beta}\right) < +\infty, \quad (118)$$

for some $\beta \in (t, +\infty)$, where the functions $K_{Q,z}^{(2)}$ and $K_{Q,z}^{(3)}$ are defined in (95).

*Proof:* The proof is an immediate consequence of Lemma 15 and the mean value theorem [70, Theorem 5.10]. ■
The relevance of Lemma 21 lies on the fact that $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ and $K_{Q,z}^{(2)}(0)$ are the variances of the random variables $W$ in (99) and $V$ in (100). See Lemma 17 and Lemma 18. Under the assumptions of Lemma 21, it follows that the function $K_{Q,z}^{(3)}$ is continuous in $(-\infty, \delta)$, where $\delta > 0$. Hence, for all $t > 0$, the function $K_{Q,z}^{(3)}$ achieves a maximum and a minimum within the interval $[-\frac{1}{t}, 0]$. Such extrema allow providing lower and upper bounds on the variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ of the random variable $W$ in terms of the variance $K_{Q,z}^{(2)}(0)$ of the random variable $V$, as shown hereunder.

**Corollary 5:** If the measure $Q$ in (19) is a probability measure and there exists a $\delta > 0$ such that the function $K_{Q,z}$ in (22) is differentiable within the open interval $(-\delta, \delta)$, then for all $t > 0$,

$$K_{Q,z}^{(2)}(0) - \frac{1}{t}c_2 \le K_{Q,z}^{(2)}\left(-\frac{1}{t}\right) \le K_{Q,z}^{(2)}(0) - \frac{1}{t}c_1, (119)$$

where,

$$c_1 = \min_{s \in [-\frac{1}{t}, 0]} K_{Q,z}^{(3)}(s) \quad \text{and} \quad (120)$$

$$c_2 = \max_{s \in [-\frac{1}{t}, 0]} K_{Q,z}^{(3)}(s), \quad (121)$$

and the functions $K_{Q,z}^{(2)}$ and $K_{Q,z}^{(3)}$ are defined in (95).

The inequality in (119) reveals that under the assumptions of Corollary 5, in the asymptotic regime when $t \to +\infty$, the variances of the random variables $W$ in (99) and $V$ in (100) are identical. Additionally, unlike the means $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ and $K_{Q,z}^{(1)}(0)$ of the random variables $W$ and $V$, which satisfy $K_{Q,z}^{(1)}(-\frac{1}{\lambda}) \le K_{Q,z}^{(1)}(0)$ (Corollary 4), their variances $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ and $K_{Q,z}^{(2)}(0)$ might satisfy $K_{Q,z}^{(2)}(-\frac{1}{\lambda}) < K_{Q,z}^{(2)}(0)$ or $K_{Q,z}^{(2)}(-\frac{1}{\lambda}) \ge K_{Q,z}^{(2)}(0)$ depending on whether the function $K_{Q,z}^{(3)}$ is positive or negative within the interval $[-\frac{1}{\lambda}, 0]$.

Lemma 21 shows that the monotonicity of the expectation of the random variable $W$ in (99) with respect to $\lambda$, stated by Theorem 5, is not a property exhibited by the variance nor the third cumulant. The following example highlights this observation.

**Example 1:** Consider the ERM-RER problem in (19), under the assumption that $Q$ is a probability measure and the empirical risk function $L_z$ in (3) is such that for all $\theta \in \mathcal{M}$,

$$L_z(\theta) = \begin{cases} 0 & \text{if } \theta \in \mathcal{A} \\ 1 & \text{if } \theta \in \mathcal{M} \setminus \mathcal{A}, \end{cases} \quad (122)$$

where the sets $\mathcal{A} \subset \mathcal{M}$ and $\mathcal{M} \setminus \mathcal{A}$ are nonnegligible with respect to the reference probability measure $Q$. In this case, the function $K_{Q,z}$ in (22) satisfies for all $\lambda > 0$,

$$K_{Q,z}\left(-\frac{1}{\lambda}\right) = \log\left(Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))\right). \quad (123)$$

The derivatives $K_{Q,z}^{(1)}$, $K_{Q,z}^{(2)}$, and $K_{Q,z}^{(3)}$ in (95) of the function $K_{Q,z}$ in (123) satisfy for all $\lambda > 0$,

$$K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) = \frac{\exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))}{Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))}; \quad (124)$$

$$K_{Q,z}^{(2)}\left(-\frac{1}{\lambda}\right) = \frac{Q(\mathcal{A})(1-Q(\mathcal{A}))\exp\left(-\frac{1}{\lambda}\right)}{(Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A})))^2}; \quad \text{and} \quad (125)$$

$$K_{Q,z}^{(3)}\left(-\frac{1}{\lambda}\right) = K_{Q,z}^{(2)}\left(-\frac{1}{\lambda}\right) \left( \frac{Q(\mathcal{A}) - (1-Q(\mathcal{A}))\exp\left(-\frac{1}{\lambda}\right)}{Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))} \right). \quad (126)$$

Note that $K_{Q,z}^{(3)}(-\frac{1}{\lambda}) > 0$ if and only if

$$Q(\mathcal{A}) - (1 - Q(\mathcal{A})) \exp\left(-\frac{1}{\lambda}\right) > 0. \quad (127)$$

Assume that $Q(\mathcal{A}) \ge \frac{1}{2}$. Thus, it holds that for all $\lambda > 0$, the inequality in (127) is always satisfied. This follows from observing that for all $\lambda > 0$,

$$\exp\left(-\frac{1}{\lambda}\right) < 1 \le \frac{Q(\mathcal{A})}{1 - Q(\mathcal{A})}. \quad (128)$$

Hence, if $Q(\mathcal{A}) \ge \frac{1}{2}$, for all decreasing sequences of positive reals $\lambda_1 > \lambda_2 > \dots > 0$, it holds that

$$\frac{1}{4} \ge K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_1}\right) > K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_2}\right) > \dots > 0. \quad (129)$$

Alternatively, assume that $Q(\mathcal{A}) < \frac{1}{2}$. In this case, the inequality in (127) is satisfied if and only if

$$\lambda < \left( \log \left( \frac{1 - Q(\mathcal{A})}{Q(\mathcal{A})} \right) \right)^{-1}. \quad (130)$$

Hence, if $Q(\mathcal{A}) < \frac{1}{2}$, then for all decreasing sequences of positive reals

$$\left(\log\left(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}\right)\right)^{-1} > \lambda_1 > \lambda_2 > \dots > 0,$$

it holds that

$$\frac{1}{4} > K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_1}\right) > K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_2}\right) > \dots > 0. \quad (131)$$

Moreover, for all decreasing sequences of positive reals

$$\lambda_1 > \lambda_2 > \dots > \left(\log\left(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}\right)\right)^{-1},$$

it holds that

$$K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_1}\right) < K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_2}\right) < \dots < \frac{1}{4}. \quad (132)$$

The upperbound by $\frac{1}{4}$ in (129), (131) and (132) follows by noticing that the value $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ is maximized when $\lambda = \left(\log\left(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}\right)\right)^{-1}$ and $K_{Q,z}^{(2)}(-\frac{1}{\lambda}) = \frac{1}{4}$.

Example 1 provides important insights on the choice of the reference measure $Q$. Note for instance that when the reference measure assigns a probability to the set of models $\mathcal{T}(z)$ in (5) that is greater than or equal to the probability of suboptimal models $\mathcal{M} \setminus \mathcal{T}(z)$, i.e., $Q(\mathcal{T}(z)) \ge \frac{1}{2}$, the variance is
The relevance of Lemma 21 lies on the fact that $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ and $K_{Q,z}^{(2)}(0)$ are the variances of the random variables $W$ in (99) and $V$ in (100). See Lemma 17 and Lemma 18. Under the assumptions of Lemma 21, it follows that the function $K_{Q,z}^{(3)}$ is continuous in $(-\infty, \delta)$, where $\delta > 0$. Hence, for all $t > 0$, the function $K_{Q,z}^{(3)}$ achieves a maximum and a minimum within the interval $[-\frac{1}{t}, 0]$. Such extrema allow providing lower and upper bounds on the variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ of the random variable $W$ in terms of the variance $K_{Q,z}^{(2)}(0)$ of the random variable $V$, as shown hereunder.

**Corollary 5:** If the measure $Q$ in (19) is a probability measure and there exists a $\delta > 0$ such that the function $K_{Q,z}$ in (22) is differentiable within the open interval $(-\delta, \delta)$, then for all $t > 0$,

$$K_{Q,z}^{(2)}(0) - \frac{1}{t}c_2 \le K_{Q,z}^{(2)}\left(-\frac{1}{t}\right) \le K_{Q,z}^{(2)}(0) - \frac{1}{t}c_1, (119)$$

where,

$$c_1 = \min_{s \in [-\frac{1}{t}, 0]} K_{Q,z}^{(3)}(s) \quad \text{and} \quad (120)$$

$$c_2 = \max_{s \in [-\frac{1}{t}, 0]} K_{Q,z}^{(3)}(s), \quad (121)$$

and the functions $K_{Q,z}^{(2)}$ and $K_{Q,z}^{(3)}$ are defined in (95).

The inequality in (119) reveals that under the assumptions of Corollary 5, in the asymptotic regime when $t \to +\infty$, the variances of the random variables $W$ in (99) and $V$ in (100) are identical. Additionally, unlike the means $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ and $K_{Q,z}^{(1)}(0)$ of the random variables $W$ and $V$, which satisfy $K_{Q,z}^{(1)}(-\frac{1}{\lambda}) \le K_{Q,z}^{(1)}(0)$ (Corollary 4), their variances $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ and $K_{Q,z}^{(2)}(0)$ might satisfy $K_{Q,z}^{(2)}(-\frac{1}{\lambda}) < K_{Q,z}^{(2)}(0)$ or $K_{Q,z}^{(2)}(-\frac{1}{\lambda}) \ge K_{Q,z}^{(2)}(0)$ depending on whether the function $K_{Q,z}^{(3)}$ is positive or negative within the interval $[-\frac{1}{\lambda}, 0]$.

Lemma 21 shows that the monotonicity of the expectation of the random variable $W$ in (99) with respect to $\lambda$, stated by Theorem 5, is not a property exhibited by the variance nor the third cumulant. The following example highlights this observation.

**Example 1:** Consider the ERM-RER problem in (19), under the assumption that $Q$ is a probability measure and the empirical risk function $L_z$ in (3) is such that for all $\theta \in \mathcal{M}$,

$$L_z(\theta) = \begin{cases} 0 & \text{if } \theta \in \mathcal{A} \\ 1 & \text{if } \theta \in \mathcal{M} \setminus \mathcal{A}, \end{cases} \quad (122)$$

where the sets $\mathcal{A} \subset \mathcal{M}$ and $\mathcal{M} \setminus \mathcal{A}$ are nonnegligible with respect to the reference probability measure $Q$. In this case, the function $K_{Q,z}$ in (22) satisfies for all $\lambda > 0$,

$$K_{Q,z}\left(-\frac{1}{\lambda}\right) = \log\left(Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))\right). \quad (123)$$

The derivatives $K_{Q,z}^{(1)}$, $K_{Q,z}^{(2)}$, and $K_{Q,z}^{(3)}$ in (95) of the function $K_{Q,z}$ in (123) satisfy for all $\lambda > 0$,

$$K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) = \frac{\exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))}{Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))}; \quad (124)$$

$$K_{Q,z}^{(2)}\left(-\frac{1}{\lambda}\right) = \frac{Q(\mathcal{A})(1-Q(\mathcal{A}))\exp\left(-\frac{1}{\lambda}\right)}{(Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A})))^2}; \quad \text{and} \quad (125)$$

$$K_{Q,z}^{(3)}\left(-\frac{1}{\lambda}\right) = K_{Q,z}^{(2)}\left(-\frac{1}{\lambda}\right) \left( \frac{Q(\mathcal{A}) - (1-Q(\mathcal{A}))\exp\left(-\frac{1}{\lambda}\right)}{Q(\mathcal{A}) + \exp\left(-\frac{1}{\lambda}\right)(1-Q(\mathcal{A}))} \right). \quad (126)$$

Note that $K_{Q,z}^{(3)}(-\frac{1}{\lambda}) > 0$ if and only if

$$Q(\mathcal{A}) - (1 - Q(\mathcal{A})) \exp\left(-\frac{1}{\lambda}\right) > 0. \quad (127)$$

Assume that $Q(\mathcal{A}) \ge \frac{1}{2}$. Thus, it holds that for all $\lambda > 0$, the inequality in (127) is always satisfied. This follows from observing that for all $\lambda > 0$,

$$\exp\left(-\frac{1}{\lambda}\right) < 1 \le \frac{Q(\mathcal{A})}{1 - Q(\mathcal{A})}. \quad (128)$$

Hence, if $Q(\mathcal{A}) \ge \frac{1}{2}$, for all decreasing sequences of positive reals $\lambda_1 > \lambda_2 > \dots > 0$, it holds that

$$\frac{1}{4} \ge K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_1}\right) > K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_2}\right) > \dots > 0. \quad (129)$$

Alternatively, assume that $Q(\mathcal{A}) < \frac{1}{2}$. In this case, the inequality in (127) is satisfied if and only if

$$\lambda < \left( \log \left( \frac{1 - Q(\mathcal{A})}{Q(\mathcal{A})} \right) \right)^{-1}. \quad (130)$$

Hence, if $Q(\mathcal{A}) < \frac{1}{2}$, then for all decreasing sequences of positive reals

$$\left(\log\left(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}\right)\right)^{-1} > \lambda_1 > \lambda_2 > \dots > 0,$$

it holds that

$$\frac{1}{4} > K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_1}\right) > K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_2}\right) > \dots > 0. \quad (131)$$

Moreover, for all decreasing sequences of positive reals

$$\lambda_1 > \lambda_2 > \dots > \left(\log\left(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}\right)\right)^{-1},$$

it holds that

$$K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_1}\right) < K_{Q,z}^{(2)}\left(-\frac{1}{\lambda_2}\right) < \dots < \frac{1}{4}. \quad (132)$$

The upperbound by $\frac{1}{4}$ in (129), (131) and (132) follows by noticing that the value $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$ is maximized when $\lambda = \left(\log\left(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}\right)\right)^{-1}$ and $K_{Q,z}^{(2)}(-\frac{1}{\lambda}) = \frac{1}{4}$.

Example 1 provides important insights on the choice of the reference measure $Q$. Note for instance that when the reference measure assigns a probability to the set of models $\mathcal{T}(z)$ in (5) that is greater than or equal to the probability of suboptimal models $\mathcal{M} \setminus \mathcal{T}(z)$, i.e., $Q(\mathcal{T}(z)) \ge \frac{1}{2}$, the variance is
strictly decreasing to zero when $\lambda$ decreases. See for instance, Figure 1 and Figure 2. That is, when the reference measure assigns higher probability to the set of solutions to the ERM problem in (4), the variance is monotone with respect to the parameter $\lambda$.

Alternatively, when the reference measure assigns a probability to the set $\mathcal{T}(z)$ that is smaller than the probability of the set $M \setminus \mathcal{T}(z)$, i.e., $Q(\mathcal{T}(z)) < \frac{1}{2}$, there exists a critical point for $\lambda$ at $(\log(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}))^{-1}$. See for instance, Figure 3. More importantly, such a critical point can be arbitrarily close to zero depending on the value $Q(\mathcal{A})$. The variance strictly decreases when $\lambda$ decreases beyond the value $(\log(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}))^{-1}$. Otherwise, reducing $\lambda$ above the value $(\log(\frac{1-Q(\mathcal{A})}{Q(\mathcal{A})}))^{-1}$ increases the variance.

In general, these observations suggest that reference measures $Q$ that allocate small measures to the sets containing the set $\mathcal{T}(z)$ might require reducing the value $\lambda$ beyond a small threshold in order to observe small values of $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, which is the variance of the random variable $W$, in (99). These observations are central to understanding the concentration of probability that occurs when $\lambda$ decreases to zero, as discussed in Section IX.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Mean $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$, variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, and third central moment $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ of the empirical risk in Example 1, with $Q(\mathcal{A}) = \frac{3}{4}$.</figcaption></figure>

## VIII. CUMULANT GENERATING FUNCTION OF THE EMPIRICAL RISK

Consider the transport of the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) from $(M, \mathcal{B}(M))$ to $([0, +\infty], \mathcal{B}([0, +\infty]))$ through the function $L_z$ in (3). Denote the resulting probability measure in $([0, +\infty], \mathcal{B}([0, +\infty]))$ by $P_{W|Z=z}^{(Q,\lambda)}$. That is, for all $\mathcal{A} \in \mathcal{B}([0, +\infty])$,

$$P_{W|Z=z}^{(Q,\lambda)}(\mathcal{A}) = P_{\Theta|Z=z}^{(Q,\lambda)}(L_z^{-1}(\mathcal{A})), \quad (133)$$

where the term $L_z^{-1}(\mathcal{A})$ represents the set

$$L_z^{-1}(\mathcal{A}) \triangleq \{\nu \in \mathcal{M} : L_z(\nu) \in \mathcal{A}\}. \quad (134)$$

<figure><img src="image_5.png" /><figcaption>Fig. 2. Mean $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$, variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, and third central moment $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ of the empirical risk in Example 1, with $Q(\mathcal{A}) = \frac{1}{2}$.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 3. Mean $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$, variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, and third central moment $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ of the empirical risk in Example 1, with $Q(\mathcal{A}) = \frac{1}{4}$.</figcaption></figure>

Note that the random variable $W$ in (99) induces the probability measure $P_{W|Z=z}^{(Q,\lambda)}$ in $([0, +\infty], \mathcal{B}([0, +\infty]))$. The objective of this section is to study the properties of the cumulant generating function of the probability measure $P_{W|Z=z}^{(Q,\lambda)}$, denoted by $J_{z,Q,\lambda}: \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$, which satisfies for all $t \in \mathbb{R}$,

$$J_{z,Q,\lambda}(t) = \log \left( \int \exp(tw) dP_{W|Z=z}^{(Q,\lambda)}(w) \right) \quad (135)$$

$$= \log \left( \int \exp(t L_z(\theta)) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \right), \quad (136)$$

where the equality in (136) follows from [59, Theorem 1.6.12].
strictly decreasing to zero when $\lambda$ decreases. See for instance, Figure 1 and Figure 2. That is, when the reference measure assigns higher probability to the set of solutions to the ERM problem in (4), the variance is monotone with respect to the parameter $\lambda$.

Alternatively, when the reference measure assigns a probability to the set $T(z)$ that is smaller than the probability of the set $M \setminus T(z)$, i.e., $Q(T(z)) < \frac{1}{2}$, there exists a critical point for $\lambda$ at $(\log(\frac{1-Q(A)}{Q(A)}))^{-1}$. See for instance, Figure 3. More importantly, such a critical point can be arbitrarily close to zero depending on the value $Q(A)$. The variance strictly decreases when $\lambda$ decreases beyond the value $(\log(\frac{1-Q(A)}{Q(A)}))^{-1}$. Otherwise, reducing $\lambda$ above the value $(\log(\frac{1-Q(A)}{Q(A)}))^{-1}$ increases the variance.

In general, these observations suggest that reference measures $Q$ that allocate small measures to the sets containing the set $T(z)$ might require reducing the value $\lambda$ beyond a small threshold in order to observe small values of $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, which is the variance of the random variable $W$, in (99). These observations are central to understanding the concentration of probability that occurs when $\lambda$ decreases to zero, as discussed in Section IX.

<figure><img src="image_5.png" /><figcaption>Fig. 1. Mean $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$, variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, and third central moment $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ of the empirical risk in Example 1, with $Q(A) = \frac{3}{4}$</figcaption></figure>

## VIII. CUMULANT GENERATING FUNCTION OF THE EMPIRICAL RISK

Consider the transport of the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) from $(M, \mathcal{B}(M))$ to $([0, +\infty], \mathcal{B}([0, +\infty]))$ through the function $L_z$ in (3). Denote the resulting probability measure in $([0, +\infty], \mathcal{B}([0, +\infty]))$ by $P_{W|Z=z}^{(Q,\lambda)}$. That is, for all $A \in \mathcal{B}([0, +\infty])$,

$$P_{W|Z=z}^{(Q,\lambda)}(A) = P_{\Theta|Z=z}^{(Q,\lambda)}(L_z^{-1}(A)), \quad (133)$$

where the term $L_z^{-1}(A)$ represents the set

$$L_z^{-1}(A) \triangleq \{\nu \in \mathcal{M} : L_z(\nu) \in A\}. \quad (134)$$

<figure><img src="image_6.png" /><figcaption>Fig. 2. Mean $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$, variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, and third central moment $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ of the empirical risk in Example 1, with $Q(A) = \frac{1}{2}$</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 3. Mean $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$, variance $K_{Q,z}^{(2)}(-\frac{1}{\lambda})$, and third central moment $K_{Q,z}^{(3)}(-\frac{1}{\lambda})$ of the empirical risk in Example 1, with $Q(A) = \frac{1}{4}$</figcaption></figure>

Note that the random variable $W$ in (99) induces the probability measure $P_{W|Z=z}^{(Q,\lambda)}$ in $([0, +\infty], \mathcal{B}([0, +\infty]))$. The objective of this section is to study the properties of the cumulant generating function of the probability measure $P_{W|Z=z}^{(Q,\lambda)}$, denoted by $J_{z,Q,\lambda}: \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$, which satisfies for all $t \in \mathbb{R}$,

$$J_{z,Q,\lambda}(t) = \log \left( \int \exp(tw) dP_{W|Z=z}^{(Q,\lambda)}(w) \right) \quad (135)$$

$$= \log \left( \int \exp(t L_z(\theta)) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \right), \quad (136)$$

where the equality in (136) follows from [59, Theorem 1.6.12].
The following lemma provides an expression for $J_{z,Q,\lambda}$ in terms of the log-partition function $K_{Q,z}$ in (22).

**Lemma 22:** If $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23), then, the function $J_{z,Q,\lambda}$ in (135), verifies for all $t \in \mathbb{R}$,

$$J_{z,Q,\lambda}(t) = K_{P^{(Q,\lambda)}|Z=z,z}(t) \quad (137)$$

$$= K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right) \quad (138)$$

$$= \sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)} \left(-\frac{1}{\lambda}\right), \quad (139)$$

with the function $K_{Q,z}$ in (22) and the function $K_{Q,z}^{(m)}$ in (95).

*Proof:* The proof of (137) follows immediately from (22) and (136). The proof of (138) follows from Lemma 12. Finally, the proof of (139) follows by observing that a Taylor expansion of the function $K_{Q,z}$ in (22) at the point $-\frac{1}{\lambda}$, yields for all $t \in \{\nu \in \mathbb{R} : K_{Q,z}(\nu) < +\infty\}$,

$$K_{Q,z}(t) = K_{Q,z} \left(-\frac{1}{\lambda}\right) + \sum_{s=1}^{+\infty} \frac{K_{Q,z}^{(s)} \left(-\frac{1}{\lambda}\right)}{s!} \left(t + \frac{1}{\lambda}\right)^s . \quad (140)$$

Choosing $\alpha \in \{\nu \in \mathbb{R} : K_{Q,z}(\nu - \frac{1}{\lambda}) < +\infty\}$ such that $t = \alpha - \frac{1}{\lambda}$ in (140) yields

$$K_{Q,z} \left( \alpha - \frac{1}{\lambda} \right) = K_{Q,z} \left( -\frac{1}{\lambda} \right) + \sum_{s=1}^{+\infty} \frac{\alpha^s}{s!} K_{Q,z}^{(s)} \left( -\frac{1}{\lambda} \right), \quad (141)$$

which implies that for all $t \in \{\nu \in \mathbb{R} : K_{Q,z}(\nu - \frac{1}{\lambda}) < +\infty\}$,

$$K_{Q,z} \left( t - \frac{1}{\lambda} \right) - K_{Q,z} \left( -\frac{1}{\lambda} \right) = \sum_{s=1}^{+\infty} \frac{t^s}{s!} K_{Q,z}^{(s)} \left( -\frac{1}{\lambda} \right). \quad (142)$$

Let $s^* \in \mathbb{R} \cup \{+\infty\}$ be defined by

$$s^* \triangleq \sup \left\{ \nu \in \mathbb{R} : K_{Q,z} \left( \nu - \frac{1}{\lambda} \right) < \infty \right\}. \quad (143)$$

If $s^* = +\infty$, then for all $t \in \mathbb{R}$, $K_{Q,z}(t - \frac{1}{\lambda}) - K_{Q,z}(-\frac{1}{\lambda}) < +\infty$, and thus,

$$+\infty > J_{z,Q,\lambda}(t) = K_{P^{(Q,\lambda)}|Z=z,z}(t) \quad (144)$$

$$= K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right) \quad (145)$$

$$= \sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)} \left(-\frac{1}{\lambda}\right). \quad (146)$$

Alternatively, if $s^* < +\infty$, it follows that for all $t > s^*$, $K_{Q,z}(t - \frac{1}{\lambda}) = +\infty$. From the fact that the function $K_{Q,z}$ is continuous (Lemma 15) and $K_{Q,z}(-\frac{1}{\lambda}) < +\infty$ (due to the fact that $\lambda \in K_{Q,z}$ in (23)), it follows that

$$+\infty = J_{z,Q,\lambda}(t) = K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right) \quad (147)$$

$$= \sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)} \left(-\frac{1}{\lambda}\right), \quad (148)$$

which implies that $\sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)}(-\frac{1}{\lambda}) = +\infty$. Hence, in this case, the equality in (139) is of the form $+\infty = +\infty$. This completes the proof. ■

Alternative expressions for $J_{z,Q,\lambda}$ in (135) are provided hereunder.

**Lemma 23:** If $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23), then, the function $J_{z,Q,\lambda}$ in (135), verifies for all $t \in (0, +\infty)$,

$$J_{z,Q,\lambda} \left(-\frac{1}{t}\right) = -\frac{1}{t} \mathrm{R}_z \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right) - D \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) \quad (149)$$

$$= -\frac{1}{t} \mathrm{R}_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) + D \left( P_{\Theta|Z=z}^{(Q,\lambda)} \| P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right) \quad (150)$$

$$\le 0, \quad (151)$$

where the functional $\mathrm{R}_z$ is in (18); the function $K_{P^{(Q,\lambda)}|Z=z,z}$ is in (60); and the probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)}$ are respectively in (25) and (73).

*Proof:* The proof of (149) follows from (107) in Lemma 20 by observing that for all $t \in (0, +\infty)$,

$$-t K_{P^{(Q,\lambda)}|Z=z,z} \left(-\frac{1}{t}\right) = \mathrm{R}_z \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \right) + t D \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) \quad (152)$$

$$= \mathrm{R}_z \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right) + t D \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right), \quad (153)$$

where the equality in (153) follows from Lemma 13. The proof of (150) follows from (108) in Lemma 20 by observing that for all $t \in (0, +\infty)$,

$$-t K_{P^{(Q,\lambda)}|Z=z,z} \left(-\frac{1}{t}\right) = \mathrm{R}_z \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \right) - t D \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) \quad (154)$$

$$= \mathrm{R}_z \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z)} \right) - t D \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z)} \| P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right), \quad (155)$$

where the equality in (155) follows from Lemma 13, which completes the proof. ■

From Lemma 15 and Lemma 22, it follows that the function $J_{z,Q,\lambda}$ in (135) is increasing and differentiable infinitely many times in the interior of $\{t \in \mathbb{R} : K_{Q,z}(t - \frac{1}{\lambda}) < +\infty\}$. Moreover, note that $(-\infty, \frac{1}{\lambda}] \subset \{t \in \mathbb{R} : K_{Q,z}(t - \frac{1}{\lambda}) < +\infty\}$. Denote by $J_{z,Q,\lambda}^{(m)} : \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$, with $m \in \mathbb{N}$, the $m$-th derivative of the function $J_{z,Q,\lambda}$ in (135). That is, for all $s \in \mathbb{R}$,

$$J_{z,Q,\lambda}^{(m)}(s) = \left. \frac{d^m}{dt^m} J_{z,Q,\lambda}(t) \right|_{t=s}. \quad (156)$$
The following lemma provides an expression for $J_{z,Q,\lambda}$ in terms of the log-partition function $K_{Q,z}$ in (22).

**Lemma 22:** If $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23), then, the function $J_{z,Q,\lambda}$ in (135), verifies for all $t \in \mathbb{R}$,

$$J_{z,Q,\lambda}(t) = K_{P^{(Q,\lambda)}|Z=z,z}(t) \quad (137)$$

$$= K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right) \quad (138)$$

$$= \sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)} \left(-\frac{1}{\lambda}\right), \quad (139)$$

with the function $K_{Q,z}$ in (22) and the function $K_{Q,z}^{(m)}$ in (95).

*Proof:* The proof of (137) follows immediately from (22) and (136). The proof of (138) follows from Lemma 12. Finally, the proof of (139) follows by observing that a Taylor expansion of the function $K_{Q,z}$ in (22) at the point $-\frac{1}{\lambda}$, yields for all $t \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu) < +\infty\}$,

$$K_{Q,z}(t) = K_{Q,z} \left(-\frac{1}{\lambda}\right) + \sum_{s=1}^{+\infty} \frac{K_{Q,z}^{(s)} \left(-\frac{1}{\lambda}\right)}{s!} \left(t + \frac{1}{\lambda}\right)^s . \quad (140)$$

Choosing $\alpha \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu - \frac{1}{\lambda}) < +\infty\}$ such that $t = \alpha - \frac{1}{\lambda}$ in (140) yields

$$K_{Q,z} \left( \alpha - \frac{1}{\lambda} \right) = K_{Q,z} \left( -\frac{1}{\lambda} \right) + \sum_{s=1}^{+\infty} \frac{\alpha^s}{s!} K_{Q,z}^{(s)} \left( -\frac{1}{\lambda} \right), \quad (141)$$

which implies that for all $t \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu - \frac{1}{\lambda}) < +\infty\}$,

$$K_{Q,z} \left( t - \frac{1}{\lambda} \right) - K_{Q,z} \left( -\frac{1}{\lambda} \right) = \sum_{s=1}^{+\infty} \frac{t^s}{s!} K_{Q,z}^{(s)} \left( -\frac{1}{\lambda} \right). \quad (142)$$

Let $s^* \in \mathbb{R} \cup \{+\infty\}$ be defined by

$$s^* \triangleq \sup \left\{ \nu \in \mathbb{R} : K_{Q,z} \left( \nu - \frac{1}{\lambda} \right) < \infty \right\}. \quad (143)$$

If $s^* = +\infty$, then for all $t \in \mathbb{R}$, $K_{Q,z}(t - \frac{1}{\lambda}) - K_{Q,z}(-\frac{1}{\lambda}) < +\infty$, and thus,

$$+\infty > J_{z,Q,\lambda}(t) = K_{P^{(Q,\lambda)}|Z=z,z}(t) \quad (144)$$

$$= K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right) \quad (145)$$

$$= \sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)} \left(-\frac{1}{\lambda}\right). \quad (146)$$

Alternatively, if $s^* < +\infty$, it follows that for all $t > s^*$, $K_{Q,z}(t - \frac{1}{\lambda}) = +\infty$. From the fact that the function $K_{Q,z}$ is continuous (Lemma 15) and $K_{Q,z}(-\frac{1}{\lambda}) < +\infty$ (due to the fact that $\lambda \in K_{Q,z}$ in (23)), it follows that

$$+\infty = J_{z,Q,\lambda}(t) = K_{Q,z} \left(t - \frac{1}{\lambda}\right) - K_{Q,z} \left(-\frac{1}{\lambda}\right) \quad (147)$$

$$= \sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)} \left(-\frac{1}{\lambda}\right), \quad (148)$$

which implies that $\sum_{m=1}^{+\infty} \frac{t^m}{m!} K_{Q,z}^{(m)}(-\frac{1}{\lambda}) = +\infty$. Hence, in this case, the equality in (139) is of the form $+\infty = +\infty$. This completes the proof. ■

Alternative expressions for $J_{z,Q,\lambda}$ in (135) are provided hereunder.

**Lemma 23:** If $\lambda \in K_{Q,z}$, with $K_{Q,z}$ in (23), then, the function $J_{z,Q,\lambda}$ in (135), verifies for all $t \in (0, +\infty)$,

$$J_{z,Q,\lambda} \left(-\frac{1}{t}\right) = -\frac{1}{t} \mathrm{R}_z \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right) - D \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) \quad (149)$$

$$= -\frac{1}{t} \mathrm{R}_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) + D \left( P_{\Theta|Z=z}^{(Q,\lambda)} \| P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right) \quad (150)$$

$$\le 0, \quad (151)$$

where the functional $\mathrm{R}_z$ is in (18); the function $K_{P^{(Q,\lambda)}|Z=z,z}$ is in (60); and the probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)}$ are respectively in (25) and (73).

*Proof:* The proof of (149) follows from (107) in Lemma 20 by observing that for all $t \in (0, +\infty)$,

$$-t K_{P^{(Q,\lambda)}|Z=z,z} \left(-\frac{1}{t}\right) = \mathrm{R}_z \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \right) + t D \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) \quad (152)$$

$$= \mathrm{R}_z \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right) + t D \left( P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right), \quad (153)$$

where the equality in (153) follows from Lemma 13. The proof of (150) follows from (108) in Lemma 20 by observing that for all $t \in (0, +\infty)$,

$$-t K_{P^{(Q,\lambda)}|Z=z,z} \left(-\frac{1}{t}\right) = \mathrm{R}_z \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \right) - t D \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z,t)} \| P_{\Theta|Z=z}^{(Q,\lambda)} \right) \quad (154)$$

$$= \mathrm{R}_z \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z)} \right) - t D \left( P_{\Theta|Z=z}^{(P^{(Q,\lambda)}|Z=z)} \| P_{\Theta|Z=z}^{\left(\frac{Q}{\lambda+\frac{1}{t}}, \frac{1}{\lambda+\frac{1}{t}}\right)} \right), \quad (155)$$

where the equality in (155) follows from Lemma 13, which completes the proof. ■

From Lemma 15 and Lemma 22, it follows that the function $J_{z,Q,\lambda}$ in (135) is increasing and differentiable infinitely many times in the interior of $\{t \in \mathbb{R}: K_{Q,z}(t - \frac{1}{\lambda}) < +\infty\}$. Moreover, note that $(-\infty, \frac{1}{\lambda}] \subset \{t \in \mathbb{R}: K_{Q,z}(t - \frac{1}{\lambda}) < +\infty\}$. Denote by $J_{z,Q,\lambda}^{(m)}: \mathbb{R} \to \mathbb{R} \cup \{+\infty\}$, with $m \in \mathbb{N}$, the $m$-th derivative of the function $J_{z,Q,\lambda}$ in (135). That is, for all $s \in \mathbb{R}$,

$$J_{z,Q,\lambda}^{(m)}(s) = \left. \frac{d^m}{dt^m} J_{z,Q,\lambda}(t) \right|_{t=s}. \quad (156)$$
From Lemma 22, it follows that for all $m \in \mathbb{R}$, and for all $\alpha \in \mathbb{R}$, the following holds,

$$J_{z,Q,\lambda}^{(m)}(\alpha) = K_{Q,z}^{(m)}\left(\alpha - \frac{1}{\lambda}\right), \quad (157)$$

where the function $K_{Q,z}^{(m)}$ denotes the m-th derivative of the function $K_{Q,z}$ in (22). See for instance, Lemma 17. The equality in (157) establishes a relation between the cumulant generating function $J_{z,Q,\lambda}$ and the function $K_{Q,z}$. This observation becomes an alternative proof to Lemma 17.

The following theorem presents the relation between the cumulant generating function $J_{z,Q,\lambda}$ and the functions $K_{Q,z}^{(1)}$ and $K_{Q,z}^{(2)}$ in (96) and (97).

**Theorem 7:** For all $\alpha \in \mathbb{R}$, the function $J_{z,Q,\lambda}$ in (135) verifies the following equality

$$J_{z,Q,\lambda}(\alpha) = \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 K_{Q,z}^{(2)}(\xi_{\alpha}) \quad (158)$$

with

$$\xi_{\alpha} \in \left( \min \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\}, \max \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\} \right), (159)$$

where the functions $K_{Q,z}^{(1)}$ and $K_{Q,z}^{(2)}$ are defined in (96) and (97), respectively.

**Proof:** From Lemma 15, it follows that the function $K_{Q,z}$ is differentiable infinitely many times in the interior of $\{t \in \mathbb{R}: K_{Q,z}(t) < +\infty\}$. Then, a Taylor expansion of the function $K_{Q,z}$ in (22) at the point $-\frac{1}{\lambda}$ yields for all $t \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu) < +\infty\}$,

$$K_{Q,z}(t) = K_{Q,z}\left(-\frac{1}{\lambda}\right) + \sum_{s=1}^{+\infty} \frac{1}{s!} \left(t + \frac{1}{\lambda}\right)^s K_{Q,z}^{(s)}\left(-\frac{1}{\lambda}\right). \quad (160)$$

Choosing $t = \alpha - \frac{1}{\lambda}$, with $\alpha \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu - \frac{1}{\lambda}) < +\infty\}$ in (160), it holds from the Taylor-Lagrange theorem [71, Theorem 2.5.4] that

$$K_{Q,z}\left(\alpha - \frac{1}{\lambda}\right) = K_{Q,z}\left(-\frac{1}{\lambda}\right) + \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 K_{Q,z}^{(2)}(\xi), \quad (161)$$

where $\xi \in (\min\{-\frac{1}{\lambda}, \alpha - \frac{1}{\lambda}\}, \max\{-\frac{1}{\lambda}, \alpha - \frac{1}{\lambda}\})$. The proof is completed by noticing that from Lemma 22, it holds that $J_{z,Q,\lambda}(\alpha) = K_{Q,z}(\alpha - \frac{1}{\lambda}) - K_{Q,z}(-\frac{1}{\lambda})$. ■

In (158), the term $\xi_{\alpha}$ depends on $\alpha$ via (159). The focus is now on the term $K_{Q,z}^{(2)}(\xi_{\alpha})$, when $\alpha \in \{t \in \mathbb{R}: J_{z,Q,\lambda}(t) < +\infty\}$.

**Theorem 8:** The function $J_{z,Q,\lambda}$ in (135) verifies the following inequality for all $\alpha \in \{t \in \mathbb{R}: J_{z,Q,\lambda}(t) < +\infty\}$,

$$J_{z,Q,\lambda}(\alpha) \le \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 \beta_{Q,z}^2 \quad (162)$$

where $\beta_{Q,z}$ satisfies

$$\beta_{Q,z} = \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\alpha)} : \alpha \in (-\infty, b - \frac{1}{\lambda}) \right\}, \quad (163)$$

with

$$b = \sup \{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\}, \quad (164)$$

and the functions $K_{Q,z}^{(1)}$ and $K_{Q,z}^{(2)}$ are defined in (96) and (97), respectively.

**Proof:** From Lemma 22, it holds that

$$\{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\} = \{t \in \mathbb{R} : K_{Q,z}\left(t - \frac{1}{\lambda}\right) < +\infty\},$$

which implies that the set $\{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\}$ contains an interval of the form $(-\infty, b)$ and is contained within an interval of the form $(-\infty, b]$, with $b$ in (164). This follows from the fact that the function $K_{Q,z}$ is continuous and nondecreasing (Lemma 15). Note also that $K_{Q,z}(0) = 0$, and thus, $\frac{1}{\lambda} \in \{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\}$, which implies that $b > \frac{1}{\lambda}$.

From Theorem 7, it follows that for all $\alpha \in (-\infty, b)$,

$$J_{z,Q,\lambda}(\alpha) \le \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 \sigma_{\alpha}^2, \quad (165)$$

where $\sigma_{\alpha}$ is

$$\sigma_{\alpha} \triangleq \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\xi)} : \xi \in \left( \min \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\}, \max \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\} \right) \right\}.$$

In the asymptotic regime, when $\alpha \to -\infty$, it holds that

$$\lim_{\alpha \to -\infty} \sigma_{\alpha} \triangleq \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\xi)} : \xi \in \left(-\infty, -\frac{1}{\lambda}\right) \right\},$$

and when $\alpha \to b$, it holds that

$$\lim_{\alpha \to b} \sigma_{\alpha} \triangleq \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\xi)} : \xi \in \left(-\frac{1}{\lambda}, b - \frac{1}{\lambda}\right) \right\}.$$

Thus, for all $\alpha \in (-\infty, b)$,

$$\sigma_{\alpha} \le \beta_{Q,z},$$

with $\beta_{Q,z}$ defined in (163), which completes the proof. ■

The main implication of Theorem 8 is that if $\beta_{Q,z}$ in (163) is finite, the random variable $W$ in (99) is a sub-Gaussian random variable with sub-Gaussianity parameter $\beta_{Q,z}$ [57, Section 2.3]. This follows by noticing that the function $J_{z,Q,\lambda}$ in (136) is the cumulant generating function of the random variable $W$. Hence, whenever $\beta_{Q,z}$ is finite, when the models are sampled from the ERM-RER optimal measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), the empirical risk with respect to the dataset $z$ is a sub-Gaussian random variable with sub-Gaussianity parameter $\beta_{Q,z}$. The following corollary of Theorem 8 highlights this observation.

**Corollary 6:** If $\beta_{Q,z}$ in (163) is finite, the random variable $W$ in (99) is a sub-Gaussian random variable with sub-Gaussianity parameter $\beta_{Q,z}$ in (163).
From Lemma 22, it follows that for all $m \in \mathbb{R}$, and for all $\alpha \in \mathbb{R}$, the following holds,

$$J_{z,Q,\lambda}^{(m)}(\alpha) = K_{Q,z}^{(m)}\left(\alpha - \frac{1}{\lambda}\right), \quad (157)$$

where the function $K_{Q,z}^{(m)}$ denotes the m-th derivative of the function $K_{Q,z}$ in (22). See for instance, Lemma 17. The equality in (157) establishes a relation between the cumulant generating function $J_{z,Q,\lambda}$ and the function $K_{Q,z}$. This observation becomes an alternative proof to Lemma 17.

The following theorem presents the relation between the cumulant generating function $J_{z,Q,\lambda}$ and the functions $K_{Q,z}^{(1)}$ and $K_{Q,z}^{(2)}$ in (96) and (97).

**Theorem 7:** For all $\alpha \in \mathbb{R}$, the function $J_{z,Q,\lambda}$ in (135) verifies the following equality

$$J_{z,Q,\lambda}(\alpha) = \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 K_{Q,z}^{(2)}(\xi_{\alpha}) \quad (158)$$

with

$$\xi_{\alpha} \in \left( \min \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\}, \max \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\} \right), (159)$$

where the functions $K_{Q,z}^{(1)}$ and $K_{Q,z}^{(2)}$ are defined in (96) and (97), respectively.

**Proof:** From Lemma 15, it follows that the function $K_{Q,z}$ is differentiable infinitely many times in the interior of $\{t \in \mathbb{R}: K_{Q,z}(t) < +\infty\}$. Then, a Taylor expansion of the function $K_{Q,z}$ in (22) at the point $-\frac{1}{\lambda}$ yields for all $t \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu) < +\infty\}$,

$$K_{Q,z}(t) = K_{Q,z}\left(-\frac{1}{\lambda}\right) + \sum_{s=1}^{+\infty} \frac{1}{s!} \left(t + \frac{1}{\lambda}\right)^s K_{Q,z}^{(s)}\left(-\frac{1}{\lambda}\right). \quad (160)$$

Choosing $t = \alpha - \frac{1}{\lambda}$, with $\alpha \in \{\nu \in \mathbb{R}: K_{Q,z}(\nu - \frac{1}{\lambda}) < +\infty\}$ in (160), it holds from the Taylor-Lagrange theorem [71, Theorem 2.5.4] that

$$K_{Q,z}\left(\alpha - \frac{1}{\lambda}\right) = K_{Q,z}\left(-\frac{1}{\lambda}\right) + \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 K_{Q,z}^{(2)}(\xi), \quad (161)$$

where $\xi \in (\min\{-\frac{1}{\lambda}, \alpha - \frac{1}{\lambda}\}, \max\{-\frac{1}{\lambda}, \alpha - \frac{1}{\lambda}\})$. The proof is completed by noticing that from Lemma 22, it holds that $J_{z,Q,\lambda}(\alpha) = K_{Q,z}(\alpha - \frac{1}{\lambda}) - K_{Q,z}(-\frac{1}{\lambda})$. ■

In (158), the term $\xi_{\alpha}$ depends on $\alpha$ via (159). The focus is now on the term $K_{Q,z}^{(2)}(\xi_{\alpha})$, when $\alpha \in \{t \in \mathbb{R}: J_{z,Q,\lambda}(t) < +\infty\}$.

**Theorem 8:** The function $J_{z,Q,\lambda}$ in (135) verifies the following inequality for all $\alpha \in \{t \in \mathbb{R}: J_{z,Q,\lambda}(t) < +\infty\}$,

$$J_{z,Q,\lambda}(\alpha) \le \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 \beta_{Q,z}^2 \quad (162)$$

where $\beta_{Q,z}$ satisfies

$$\beta_{Q,z} = \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\alpha)} : \alpha \in (-\infty, b - \frac{1}{\lambda}) \right\}, \quad (163)$$

with

$$b = \sup \{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\}, \quad (164)$$

and the functions $K_{Q,z}^{(1)}$ and $K_{Q,z}^{(2)}$ are defined in (96) and (97), respectively.

**Proof:** From Lemma 22, it holds that

$$\{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\} = \{t \in \mathbb{R} : K_{Q,z}\left(t - \frac{1}{\lambda}\right) < +\infty\},$$

which implies that the set $\{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\}$ contains an interval of the form $(-\infty, b)$ and is contained within an interval of the form $(-\infty, b]$, with $b$ in (164). This follows from the fact that the function $K_{Q,z}$ is continuous and nondecreasing (Lemma 15). Note also that $K_{Q,z}(0) = 0$, and thus, $\frac{1}{\lambda} \in \{t \in \mathbb{R} : J_{z,Q,\lambda}(t) < +\infty\}$, which implies that $b > \frac{1}{\lambda}$.

From Theorem 7, it follows that for all $\alpha \in (-\infty, b)$,

$$J_{z,Q,\lambda}(\alpha) \le \alpha K_{Q,z}^{(1)}\left(-\frac{1}{\lambda}\right) + \frac{1}{2}\alpha^2 \sigma_{\alpha}^2, \quad (165)$$

where $\sigma_{\alpha}$ is

$$\sigma_{\alpha} \triangleq \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\xi)} : \xi \in \left( \min \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\}, \max \left\{ -\frac{1}{\lambda}, \alpha - \frac{1}{\lambda} \right\} \right) \right\}.$$

In the asymptotic regime, when $\alpha \to -\infty$, it holds that

$$\lim_{\alpha \to -\infty} \sigma_{\alpha} \triangleq \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\xi)} : \xi \in \left(-\infty, -\frac{1}{\lambda}\right) \right\},$$

and when $\alpha \to b$, it holds that

$$\lim_{\alpha \to b} \sigma_{\alpha} \triangleq \sup \left\{ \sqrt{K_{Q,z}^{(2)}(\xi)} : \xi \in \left(-\frac{1}{\lambda}, b - \frac{1}{\lambda}\right) \right\}.$$

Thus, for all $\alpha \in (-\infty, b)$,

$$\sigma_{\alpha} \le \beta_{Q,z},$$

with $\beta_{Q,z}$ defined in (163), which completes the proof. ■

The main implication of Theorem 8 is that if $\beta_{Q,z}$ in (163) is finite, the random variable $W$ in (99) is a sub-Gaussian random variable with sub-Gaussianity parameter $\beta_{Q,z}$ [57, Section 2.3]. This follows by noticing that the function $J_{z,Q,\lambda}$ in (136) is the cumulant generating function of the random variable $W$. Hence, whenever $\beta_{Q,z}$ is finite, when the models are sampled from the ERM-RER optimal measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), the empirical risk with respect to the dataset $z$ is a sub-Gaussian random variable with sub-Gaussianity parameter $\beta_{Q,z}$. The following corollary of Theorem 8 highlights this observation.

**Corollary 6:** If $\beta_{Q,z}$ in (163) is finite, the random variable $W$ in (99) is a sub-Gaussian random variable with sub-Gaussianity parameter $\beta_{Q,z}$ in (163).
## IX. CONCENTRATION OF PROBABILITY

Consider the following set,

$$ \mathcal{N}_{Q,z}(\lambda) \triangleq \{\theta \in \mathcal{M} : L_z(\theta) \le R_z \left(P_{\Theta|Z=z}^{(Q,\lambda)}\right)\}, \quad (166) $$

where the function $L_z$ is defined by (3); the functional $R_z$ is defined by (18); and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25). This section introduces two results. First, in Theorem 9, it is shown that when $\lambda$ tends to zero, the set $\mathcal{N}_{Q,z}(\lambda)$ forms an indexed family of sets that is monotonic and decreases to the set

$$ \mathcal{N}_{Q,z}^{*} \triangleq \mathcal{L}_z (\delta_{Q,z}^{*}), \quad (167) $$

where $\delta_{Q,z}^*$ is defined in (37); and the set $\mathcal{L}_z(\delta_{Q,z}^*)$ is defined in (36). Second, in Theorem 10, it is shown that the probability $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda))$ strictly increases when $\lambda$ tends to zero. More importantly, in Theorem 11, it is shown that the limit of the probability $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda))$, when $\lambda \to 0$, is equal to one. These observations justify referring to the set $\mathcal{N}_{Q,z}^*$ as the limit set. These observations are complementary to those stated in Section III-B and Section III-C. This section ends by showing that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ concentrates on a specific subset $\mathcal{L}_{Q,z}^*$ in (38) of the set $\mathcal{N}_{Q,z}$. At the light of this observation, the set $\mathcal{L}_{Q,z}^*$ is referred to as the nonnegligible limit set. Finally, it is shown that when the $\sigma$-finite measure $Q$ in (19) is coherent, the sets $\mathcal{N}_{Q,z}^*$ and $\mathcal{L}_{Q,z}^*$ are identical.

### A. The Limit Set

The set $\mathcal{N}_{Q,z}(\lambda)$ in (166), with $\lambda \in \mathcal{K}_{Q,z}$ and $\mathcal{K}_{Q,z}$ in (23), contains all the models that induce an empirical risk that is smaller than or equal to $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$, i.e., the ERM-RER-optimal expected empirical risk in (104). This observation unveils the existence of a relation between the set $\mathcal{N}_{Q,z}^*$ in (167) and the set $\mathcal{T}(z)$ in (5), as shown by the following lemma.

**Lemma 24:** The set $\mathcal{N}_{Q,z}^*$ in (167) satisfies

$$ \mathcal{T}(z) \subseteq \mathcal{N}_{Q,z}^*, \quad (168) $$

where the set $\mathcal{T}(z)$ is in (5). Moreover,

$$ \mathcal{T}(z) = \mathcal{N}_{Q,z}^*, \quad (169) $$

if and only if (a) the ERM problem in (4) possesses a solution; and (b) the reference measure $Q$ in (19) is coherent.

**Proof:** If the set $\mathcal{T}(z)$ in (5) is empty, the inclusion in (168) is trivially true. Assume that $|\mathcal{T}(z)| > 0$. Hence, the proof of the inclusion in (168) follows from observing that for all $\theta \in \mathcal{T}(z)$, it holds that $L_z(\theta) = \rho^* \le \delta_{Q,z}^*$, with $\delta_{Q,z}^*$ in (37) and $\rho^*$ in (47). Hence, $\theta \in \mathcal{N}_{Q,z}^*$. This completes the proof of the inclusion in (168).

The proof of the equality in (169) is presented in two parts. In the first part, it is proved that if (169) holds, then the ERM problem in (4) possesses a solution and the measure $Q$ is coherent. The second part proves the converse. The proof of the first part is as follows. Under the assumption that $\mathcal{T}(z) = \mathcal{N}_{Q,z}^*$ holds, it follows that $\delta_{Q,z}^* = \rho^*$, with $\rho^*$ in (47), which

implies that the ERM problem in (4) possesses a solution. Moreover, for all $\delta \in (\rho^*, +\infty)$, it holds that $Q(\mathcal{L}_z(\delta)) > 0$, which verifies that the measure $Q$ is coherent and completes the proof of the first part. The proof of the second part is as follows. Under the assumption that the ERM problem in (4) possesses a solution and the measure $Q$ is coherent, it follows that $\delta_{Q,z}^* = \rho^*$. Hence, $\mathcal{T}(z) = \mathcal{N}_{Q,z}^*$, which completes the proof of the second part. ■

The following theorem highlights that the set $\mathcal{N}_{Q,z}(\lambda)$ is decreasing with $\lambda$.

**Theorem 9:** For all $(\lambda_1, \lambda_2) \in \mathcal{K}_{Q,z} \times \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23) and $\lambda_1 > \lambda_2$, the sets $\mathcal{N}_{Q,z}(\lambda_1)$ and $\mathcal{N}_{Q,z}(\lambda_2)$ in (166) satisfy

$$ \mathcal{M} \supseteq \mathcal{N}_{Q,z}(\lambda_1) \supseteq \mathcal{N}_{Q,z}(\lambda_2) \supseteq \mathcal{N}_{Q,z}^*, \quad (170) $$

with $\mathcal{N}_{Q,z}^*$ being the set defined in (167). Moreover, if the empirical risk function $L_z$ in (3) is continuous on $\mathcal{M}$ and separable with respect to the measure $Q$ in (19), then,

$$ \mathcal{M} \supseteq \mathcal{N}_{Q,z}(\lambda_1) \supseteq \mathcal{N}_{Q,z}(\lambda_2) \supseteq \mathcal{N}_{Q,z}^*. \quad (171) $$

**Proof:** The proof is presented in Appendix Q. ■

<figure><img src="image_7.png" /></figure>

An interesting observation is that for all $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), only a subset of $\mathcal{N}_{Q,z}(\lambda)$ might exhibit nonzero probability with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25). Consider for instance that the measure $Q$ in (19) is non-coherent (Definition 3). That is, $\delta_{Q,z}^* > \rho^*$, with $\delta_{Q,z}^*$ in (37) and $\rho^*$ in (47). Thus, for all $\gamma \in (\rho^*, \delta_{Q,z}^*)$, it holds that $Q(\mathcal{L}_z(\gamma)) = 0$, with the set $\mathcal{L}_z(\cdot)$ in (36). From Lemma 3, this implies that for all $\gamma \in (\rho^*, \delta_{Q,z}^*)$, the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\gamma)) = 0$, while verifying that $\mathcal{L}_z(\gamma) \subseteq \mathcal{N}_{Q,z}(\lambda)$. These observations lead to the analysis of the asymptotic concentration of probability in the following section.

### B. The Nonnegligible Limit Set

The first step in the analysis of the asymptotic concentration of the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is to show that the probability $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda))$ increases when $\lambda$ tends to zero, as shown by the following theorem.

**Theorem 10:** For all $(\lambda_1, \lambda_2) \in \mathcal{K}_{Q,z} \times \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23) and $\lambda_1 > \lambda_2$, assume that the measures $P_{\Theta|Z=z}^{(Q,\lambda_1)}$ and $P_{\Theta|Z=z}^{(Q,\lambda_2)}$ satisfy (25) with $\lambda = \lambda_1$ and $\lambda = \lambda_2$, respectively. Then, the set $\mathcal{N}_{Q,z}(\lambda_2)$ in (166) satisfies

$$ 0 < P_{\Theta|Z=z}^{(Q,\lambda_1)}(\mathcal{N}_{Q,z}(\lambda_2)) \le P_{\Theta|Z=z}^{(Q,\lambda_2)}(\mathcal{N}_{Q,z}(\lambda_2)), \quad (172) $$

where strict inequality holds if and only if the function $L_z$ is separable with respect to the $\sigma$-finite measure $Q$.

**Proof:** The proof is presented in Appendix R. ■

<figure><img src="image_8.png" /></figure>

The following lemma highlights a case in which a stronger concentration of probability is observed.

**Lemma 25:** Let the function $L_z$ in (3) be separable with respect to the $\sigma$-finite measure $Q$ in (19). Let also $(\lambda_1, \lambda_2) \in \mathcal{K}_{Q,z} \times$