## IX. CONCENTRATION OF PROBABILITY

Consider the following set,

$$ \mathcal{N}_{Q,z}(\lambda) \triangleq \{\theta \in \mathcal{M} : L_z(\theta) \le R_z \left(P_{\Theta|Z=z}^{(Q,\lambda)}\right)\}, \quad (166) $$

where the function $L_z$ is defined by (3); the functional $R_z$ is defined by (18); and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25). This section introduces two results. First, in Theorem 9, it is shown that when $\lambda$ tends to zero, the set $\mathcal{N}_{Q,z}(\lambda)$ forms an indexed family of sets that is monotonic and decreases to the set

$$ \mathcal{N}_{Q,z}^* \triangleq \mathcal{L}_z (\delta_{Q,z}^*), \quad (167) $$

where $\delta_{Q,z}^*$ is defined in (37); and the set $\mathcal{L}_z(\delta_{Q,z}^*)$ is defined in (36). Second, in Theorem 10, it is shown that the probability $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda))$ strictly increases when $\lambda$ tends to zero. More importantly, in Theorem 11, it is shown that the limit of the probability $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda))$, when $\lambda \to 0$, is equal to one. These observations justify referring to the set $\mathcal{N}_{Q,z}^*$ as the limit set. These observations are complementary to those stated in Section III-B and Section III-C. This section ends by showing that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ concentrates on a specific subset $\mathcal{L}_{Q,z}^*$ in (38) of the set $\mathcal{N}_{Q,z}^*$. At the light of this observation, the set $\mathcal{L}_{Q,z}^*$ is referred to as the nonnegligible limit set. Finally, it is shown that when the $\sigma$-finite measure $Q$ in (19) is coherent, the sets $\mathcal{N}_{Q,z}^*$ and $\mathcal{L}_{Q,z}^*$ are identical.

### A. The Limit Set

The set $\mathcal{N}_{Q,z}(\lambda)$ in (166), with $\lambda \in \mathcal{K}_{Q,z}$ and $\mathcal{K}_{Q,z}$ in (23), contains all the models that induce an empirical risk that is smaller than or equal to $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$, i.e., the ERM-RER-optimal expected empirical risk in (104). This observation unveils the existence of a relation between the set $\mathcal{N}_{Q,z}^*$ in (167) and the set $\mathcal{T}(z)$ in (5), as shown by the following lemma.

**Lemma 24:** The set $\mathcal{N}_{Q,z}^*$ in (167) satisfies

$$ \mathcal{T}(z) \subseteq \mathcal{N}_{Q,z}^*, \quad (168) $$

where the set $\mathcal{T}(z)$ is in (5). Moreover,

$$ \mathcal{T}(z) = \mathcal{N}_{Q,z}^*, \quad (169) $$

if and only if (a) the ERM problem in (4) possesses a solution; and (b) the reference measure $Q$ in (19) is coherent.

**Proof:** If the set $\mathcal{T}(z)$ in (5) is empty, the inclusion in (168) is trivially true. Assume that $|\mathcal{T}(z)| > 0$. Hence, the proof of the inclusion in (168) follows from observing that for all $\theta \in \mathcal{T}(z)$, it holds that $L_z(\theta) = \rho^* \le \delta_{Q,z}^*$, with $\delta_{Q,z}^*$ in (37) and $\rho^*$ in (47). Hence, $\theta \in \mathcal{N}_{Q,z}^*$. This completes the proof of the inclusion in (168).

The proof of the equality in (169) is presented in two parts. In the first part, it is proved that if (169) holds, then the ERM problem in (4) possesses a solution and the measure $Q$ is coherent. The second part proves the converse. The proof of the first part is as follows. Under the assumption that $\mathcal{T}(z) = \mathcal{N}_{Q,z}^*$ holds, it follows that $\delta_{Q,z}^* = \rho^*$, with $\rho^*$ in (47), which

implies that the ERM problem in (4) possesses a solution. Moreover, for all $\delta \in (\rho^*, +\infty)$, it holds that $Q(\mathcal{L}_z(\delta)) > 0$, which verifies that the measure $Q$ is coherent and completes the proof of the first part. The proof of the second part is as follows. Under the assumption that the ERM problem in (4) possesses a solution and the measure $Q$ is coherent, it follows that $\delta_{Q,z}^* = \rho^*$. Hence, $\mathcal{T}(z) = \mathcal{N}_{Q,z}^*$, which completes the proof of the second part. ■

The following theorem highlights that the set $\mathcal{N}_{Q,z}(\lambda)$ is decreasing with $\lambda$.

**Theorem 9:** For all $(\lambda_1, \lambda_2) \in \mathcal{K}_{Q,z} \times \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23) and $\lambda_1 > \lambda_2$, the sets $\mathcal{N}_{Q,z}(\lambda_1)$ and $\mathcal{N}_{Q,z}(\lambda_2)$ in (166) satisfy

$$ \mathcal{M} \supseteq \mathcal{N}_{Q,z}(\lambda_1) \supseteq \mathcal{N}_{Q,z}(\lambda_2) \supseteq \mathcal{N}_{Q,z}^*, \quad (170) $$

with $\mathcal{N}_{Q,z}^*$ being the set defined in (167). Moreover, if the empirical risk function $L_z$ in (3) is continuous on $\mathcal{M}$ and separable with respect to the measure $Q$ in (19), then,

$$ \mathcal{M} \supseteq \mathcal{N}_{Q,z}(\lambda_1) \supseteq \mathcal{N}_{Q,z}(\lambda_2) \supseteq \mathcal{N}_{Q,z}^*. \quad (171) $$

**Proof:** The proof is presented in Appendix Q. ■

An interesting observation is that for all $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), only a subset of $\mathcal{N}_{Q,z}(\lambda)$ might exhibit nonzero probability with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25). Consider for instance that the measure $Q$ in (19) is non-coherent (Definition 3). That is, $\delta_{Q,z}^* > \rho^*$, with $\delta_{Q,z}^*$ in (37) and $\rho^*$ in (47). Thus, for all $\gamma \in (\rho^*, \delta_{Q,z}^*)$, it holds that $Q(\mathcal{L}_z(\gamma)) = 0$, with the set $\mathcal{L}_z(\cdot)$ in (36). From Lemma 3, this implies that for all $\gamma \in (\rho^*, \delta_{Q,z}^*)$, the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\gamma)) = 0$, while verifying that $\mathcal{L}_z(\gamma) \subseteq \mathcal{N}_{Q,z}(\lambda)$. These observations lead to the analysis of the asymptotic concentration of probability in the following section.

### B. The Nonnegligible Limit Set

The first step in the analysis of the asymptotic concentration of the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is to show that the probability $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda))$ increases when $\lambda$ tends to zero, as shown by the following theorem.

**Theorem 10:** For all $(\lambda_1, \lambda_2) \in \mathcal{K}_{Q,z} \times \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23) and $\lambda_1 > \lambda_2$, assume that the measures $P_{\Theta|Z=z}^{(Q,\lambda_1)}$ and $P_{\Theta|Z=z}^{(Q,\lambda_2)}$ satisfy (25) with $\lambda = \lambda_1$ and $\lambda = \lambda_2$, respectively. Then, the set $\mathcal{N}_{Q,z}(\lambda_2)$ in (166) satisfies

$$ 0 < P_{\Theta|Z=z}^{(Q,\lambda_1)}(\mathcal{N}_{Q,z}(\lambda_2)) \le P_{\Theta|Z=z}^{(Q,\lambda_2)}(\mathcal{N}_{Q,z}(\lambda_2)), \quad (172) $$

where strict inequality holds if and only if the function $L_z$ is separable with respect to the $\sigma$-finite measure $Q$.

**Proof:** The proof is presented in Appendix R. ■

The following lemma highlights a case in which a stronger concentration of probability is observed.

**Lemma 25:** Let the function $L_z$ in (3) be separable with respect to the $\sigma$-finite measure $Q$ in (19). Let also $(\lambda_1, \lambda_2) \in \mathcal{K}_{Q,z} \times$
$\mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), be two positive reals such that $\lambda_1 > \lambda_2$ and

$$Q(\mathcal{N}_{Q,z}(\lambda_1) \cap (\mathcal{N}_{Q,z}(\lambda_2))^c) = 0, \quad (173)$$

with the complement with respect to the set of models $\mathcal{M}$. Then, two measures $P_{\Theta|Z=z}^{(Q,\lambda_1)}$ and $P_{\Theta|Z=z}^{(Q,\lambda_2)}$ that respectively satisfy (25) with $\lambda = \lambda_1$ and $\lambda = \lambda_2$ verify that

$$P_{\Theta|Z=z}^{(Q,\lambda_1)}(\mathcal{N}_{Q,z}(\lambda_1)) < P_{\Theta|Z=z}^{(Q,\lambda_2)}(\mathcal{N}_{Q,z}(\lambda_2)), \quad (174)$$

where, the set $\mathcal{N}_{Q,z}(\cdot)$ is defined in (166).

*Proof:* The proof is presented in Appendix S. ■

The following example shows the relevance of Lemma 25 in the case in which the empirical risk function $L_z$ in (3) is a simple function and separable with respect to the $\sigma$-finite measure $Q$ in (19).

*Example 2:* Consider Example 1. Note that, for all $\lambda > 0$,

$$0 < R_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) < 1, \quad (175)$$

where $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is the ERM-RER-optimal expected empirical risk in (104). The equality in (175) implies that given two reals $\lambda_1$ and $\lambda_2$ such that $\lambda_1 > \lambda_2 > 0$, it holds that,

$$\begin{aligned} \mathcal{N}_{Q,z}(\lambda_1) \cap (\mathcal{N}_{Q,z}(\lambda_2))^c &= \{\nu \in \mathcal{M} : R_z(P_{\Theta|Z=z}^{(Q,\lambda_2)}) < L_z(\nu) \le R_z(P_{\Theta|Z=z}^{(Q,\lambda_1)})\} \quad (176) \\ &= \emptyset, \quad (177) \end{aligned}$$

and moreover, $\mathcal{N}_{Q,z}(\lambda_1) = \mathcal{N}_{Q,z}(\lambda_2)$. Finally, from Lemma 25,

$$P_{\Theta|Z=z}^{(Q,\lambda_1)}(\mathcal{N}_{Q,z}(\lambda_1)) < P_{\Theta|Z=z}^{(Q,\lambda_2)}(\mathcal{N}_{Q,z}(\lambda_2)). \quad (178)$$

The main result of this section is presented by the following theorem.

*Theorem 11:* The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies

$$\lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{N}_{Q,z}(\lambda)) = 1, \quad (179)$$

where, the set $\mathcal{N}_{Q,z}(\lambda)$ is defined in (166).

*Proof:* The proof follows immediately from Lemma 7 and by noticing that for all $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the sets $\mathcal{L}_{Q,z}^*$ in (38) and $\mathcal{N}_{Q,z}(\lambda)$ in (166) satisfies $\mathcal{L}_{Q,z}^* \subseteq \mathcal{N}_{Q,z}(\lambda)$. ■

Note that Theorem 11 and Lemma 7 lead to the following conclusion

$$\lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{N}_{Q,z}(\lambda) \setminus \mathcal{L}_{Q,z}^*) = 0, \quad (180)$$

which follows from the fact that $\mathcal{L}_{Q,z}^* \subset \mathcal{N}_{Q,z}(\lambda)$, with $\mathcal{L}_{Q,z}^*$ in (38). This justifies referring to the set $\mathcal{L}_{Q,z}^*$ as the nonnegligible limit set.

## X. $(\delta, \epsilon)$-OPTIMALITY

This section introduces a PAC guarantee of optimality for the models that are sampled from the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) with respect to the ERM problem in (4). Such guarantee is defined as follows.

*Definition 6* (($\delta, \epsilon$)-Optimality): Given a pair of positive reals $(\delta, \epsilon)$, with $\epsilon < 1$, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is said to be $(\delta, \epsilon)$-optimal, if the set $\mathcal{L}_z(\delta)$ in (36) satisfies

$$P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_z(\delta)) > 1 - \epsilon. \quad (181)$$

If the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is $(\delta, \epsilon)$-optimal, then it assigns a probability that is always greater than $1 - \epsilon$ to a set that contains models that induce an empirical risk that is smaller than $\delta$. From this perspective, particular interest is given to the smallest $\delta$ and $\epsilon$ for which $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal.

The main result of this section is presented by the following theorem.

*Theorem 12:* For all $(\delta, \epsilon) \in (\delta_{Q,z}^*, +\infty) \times (0, 1)$, with $\delta_{Q,z}^*$ in (37), there exists a real $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal.

*Proof:* Let $\delta$ be a real in $(\delta_{Q,z}^*, +\infty)$, with $\delta_{Q,z}^*$ in (37). Let also $\gamma \in \mathcal{K}_{Q,z}$ satisfy the following equality:

$$K_{Q,z}^{(1)} \left(-\frac{1}{\gamma}\right) \le \delta. \quad (182)$$

Note that from Lemma 15, it follows that the function $K_{Q,z}^{(1)}$ is continuous. Moreover, from Theorem 6, it follows that such a $\gamma$ in (182) always exists. From (36) and (166), it holds that

$$\mathcal{N}_{Q,z}(\gamma) \subseteq \mathcal{L}_z(\delta), \quad (183)$$

and thus,

$$P_{\Theta|Z=z}^{(Q,\gamma)} (\mathcal{L}_z(\delta)) \ge P_{\Theta|Z=z}^{(Q,\gamma)} (\mathcal{N}_{Q,z}(\gamma)). \quad (184)$$

Let $\lambda$ be a positive real such that $\lambda \le \gamma$ and

$$P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{N}_{Q,z}(\lambda)) > 1 - \epsilon. \quad (185)$$

The existence of such a positive real $\lambda$ follows from Theorem 11. Hence, from (185), it holds that,

$$1 - \epsilon < P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{N}_{Q,z}(\lambda)) \quad (186)$$

$$\le P_{\Theta|Z=z}^{(Q,\lambda)} (\mathcal{L}_z(\delta)), \quad (187)$$

where the inequality in (187) follows from the fact that $\mathcal{N}_{Q,z}(\lambda) \subseteq \mathcal{N}_{Q,z}(\gamma) \subseteq \mathcal{L}_z(\delta)$. Finally, the inequality in (187) implies that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal (Definition 6). This completes the proof. ■

A stronger optimality claim can be stated when the reference measure is coherent.

*Theorem 13:* For all $(\delta, \epsilon) \in (\rho^*, +\infty) \times (0, 1)$, with $\rho^*$ in (47), there always exists a $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that
$\mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), be two positive reals such that $\lambda_1 > \lambda_2$ and

$$Q(\mathcal{N}_{Q,z}(\lambda_1) \cap (\mathcal{N}_{Q,z}(\lambda_2))^c) = 0, \quad (173)$$

with the complement with respect to the set of models $\mathcal{M}$. Then, two measures $P_{\Theta|Z=z}^{(Q,\lambda_1)}$ and $P_{\Theta|Z=z}^{(Q,\lambda_2)}$ that respectively satisfy (25) with $\lambda = \lambda_1$ and $\lambda = \lambda_2$ verify that

$$P_{\Theta|Z=z}^{(Q,\lambda_1)}(\mathcal{N}_{Q,z}(\lambda_1)) < P_{\Theta|Z=z}^{(Q,\lambda_2)}(\mathcal{N}_{Q,z}(\lambda_2)), \quad (174)$$

where, the set $\mathcal{N}_{Q,z}(\cdot)$ is defined in (166).

*Proof:* The proof is presented in Appendix S. ■

The following example shows the relevance of Lemma 25 in the case in which the empirical risk function $L_z$ in (3) is a simple function and separable with respect to the $\sigma$-finite measure $Q$ in (19).

*Example 2:* Consider Example 1. Note that, for all $\lambda > 0$,

$$0 < R_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) < 1, \quad (175)$$

where $R_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ is the ERM-RER-optimal expected empirical risk in (104). The equality in (175) implies that given two reals $\lambda_1$ and $\lambda_2$ such that $\lambda_1 > \lambda_2 > 0$, it holds that,

$$\begin{align}
\mathcal{N}_{Q,z}(\lambda_1) \cap (\mathcal{N}_{Q,z}(\lambda_2))^c \\
&= \{\nu \in \mathcal{M} : R_z(P_{\Theta|Z=z}^{(Q,\lambda_2)}) < L_z(\nu) \le R_z(P_{\Theta|Z=z}^{(Q,\lambda_1)})\} \tag{176} \\
&= \emptyset, \tag{177}
\end{align}$$

and moreover, $\mathcal{N}_{Q,z}(\lambda_1) = \mathcal{N}_{Q,z}(\lambda_2)$. Finally, from Lemma 25,

$$P_{\Theta|Z=z}^{(Q,\lambda_1)}(\mathcal{N}_{Q,z}(\lambda_1)) < P_{\Theta|Z=z}^{(Q,\lambda_2)}(\mathcal{N}_{Q,z}(\lambda_2)). \quad (178)$$

The main result of this section is presented by the following theorem.

*Theorem 11:* The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies

$$\lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda)) = 1, \quad (179)$$

where, the set $\mathcal{N}_{Q,z}(\lambda)$ is defined in (166).

*Proof:* The proof follows immediately from Lemma 7 and by noticing that for all $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the sets $\mathcal{L}_{Q,z}^*$ in (38) and $\mathcal{N}_{Q,z}(\lambda)$ in (166) satisfies $\mathcal{L}_{Q,z}^* \subseteq \mathcal{N}_{Q,z}(\lambda)$. ■

Note that Theorem 11 and Lemma 7 lead to the following conclusion

$$\lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda) \setminus \mathcal{L}_{Q,z}^*) = 0, \quad (180)$$

which follows from the fact that $\mathcal{L}_{Q,z}^* \subset \mathcal{N}_{Q,z}(\lambda)$, with $\mathcal{L}_{Q,z}^*$ in (38). This justifies referring to the set $\mathcal{L}_{Q,z}^*$ as the nonnegligible limit set.

## X. $(\delta, \epsilon)$-OPTIMALITY

This section introduces a PAC guarantee of optimality for the models that are sampled from the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) with respect to the ERM problem in (4). Such guarantee is defined as follows.

*Definition 6* (($\delta, \epsilon$)-Optimality): Given a pair of positive reals $(\delta, \epsilon)$, with $\epsilon < 1$, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is said to be $(\delta, \epsilon)$-optimal, if the set $\mathcal{L}_z(\delta)$ in (36) satisfies

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) > 1 - \epsilon. \quad (181)$$

If the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is $(\delta, \epsilon)$-optimal, then it assigns a probability that is always greater than $1 - \epsilon$ to a set that contains models that induce an empirical risk that is smaller than $\delta$. From this perspective, particular interest is given to the smallest $\delta$ and $\epsilon$ for which $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal.

The main result of this section is presented by the following theorem.

*Theorem 12:* For all $(\delta, \epsilon) \in (\delta_{Q,z}^*, +\infty) \times (0, 1)$, with $\delta_{Q,z}^*$ in (37), there exists a real $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal.

*Proof:* Let $\delta$ be a real in $(\delta_{Q,z}^*, +\infty)$, with $\delta_{Q,z}^*$ in (37). Let also $\gamma \in \mathcal{K}_{Q,z}$ satisfy the following equality:

$$K_{Q,z}^{(1)} \left(-\frac{1}{\gamma}\right) \le \delta. \quad (182)$$

Note that from Lemma 15, it follows that the function $K_{Q,z}^{(1)}$ is continuous. Moreover, from Theorem 6, it follows that such a $\gamma$ in (182) always exists. From (36) and (166), it holds that

$$\mathcal{N}_{Q,z}(\gamma) \subseteq \mathcal{L}_z(\delta), \quad (183)$$

and thus,

$$P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{L}_z(\delta)) \ge P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{N}_{Q,z}(\gamma)). \quad (184)$$

Let $\lambda$ be a positive real such that $\lambda \le \gamma$ and

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda)) > 1 - \epsilon. \quad (185)$$

The existence of such a positive real $\lambda$ follows from Theorem 11. Hence, from (185), it holds that,

$$1 - \epsilon < P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{N}_{Q,z}(\lambda)) \quad (186)$$

$$\le P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)), \quad (187)$$

where the inequality in (187) follows from the fact that $\mathcal{N}_{Q,z}(\lambda) \subseteq \mathcal{N}_{Q,z}(\gamma) \subseteq \mathcal{L}_z(\delta)$. Finally, the inequality in (187) implies that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal (Definition 6). This completes the proof. ■

A stronger optimality claim can be stated when the reference measure is coherent.

*Theorem 13:* For all $(\delta, \epsilon) \in (\rho^*, +\infty) \times (0, 1)$, with $\rho^*$ in (47), there always exists a $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that
the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal if and only if the reference measure $Q$ is coherent.

*Proof:* The proof is divided into two parts. The first part shows that if for all $(\delta, \epsilon) \in (\rho^*, +\infty) \times (0, 1)$, there always exists a $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is $(\delta, \epsilon)$-optimal, then, the measure $Q$ is coherent. The second part deals with the converse.

The first part is as follows. Let $\gamma \in \mathcal{K}_{Q,z}$ be such that

$$P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{L}_z(\delta)) > 1 - \epsilon, \quad (188)$$

then, for all measurable subsets $A$ of $\mathcal{L}_z(\delta)$, it holds that

$$1 - \epsilon < P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{L}_z(\delta)) \quad (189) \\ = \int_A \frac{dP_{\Theta|Z=z}^{(Q,\gamma)}(\nu)dQ(\nu)}{dQ} + \int_{\mathcal{L}_z(\delta)\setminus A} \frac{dP_{\Theta|Z=z}^{(Q,\gamma)}(\nu)dQ(\nu)}{dQ},$$

which, together with Lemma 2, implies that there exists at least one measurable subset $A$ for which $Q(A) > 0$, and thus,

$$Q(\mathcal{L}_z(\delta)) > Q(A) > 0, \quad (190)$$

which implies that the measure $Q$ is coherent. This completes the first part of the proof.

The second part of the proof is as follows. Under the assumption that the measure $Q$ is coherent, it follows that $\delta_{Q,z}^* = \rho^*$. Then, from Theorem 12, it follows that for all $(\delta, \epsilon) \in (\delta_{Q,z}^*, +\infty) \times (0, 1)$, there always exists a $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal. This completes the second part of the proof. ■

# XI. SENSITIVITY AND GENERALIZATION

This section introduces the notion of sensitivity and establishes its connections with the notion of generalization error of the Gibbs algorithm, cf. [9].

## A. Sensitivity

The sensitivity of the expected empirical risk $R_z$ in (18) to deviations from the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) towards an alternative probability measure $P \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ is introduced as a novel metric to evaluate the generalization capabilities of the ERM-RER-optimal measure $P_{\Theta|Z=z}^{(Q,\lambda)}$. Deviations from the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ towards an alternative probability measure $P$ would allow comparing the ERM-RER-optimal measure with alternative measures (or algorithms). For instance, if new datasets become available, a new ERM-RER problem can be formulated using a larger dataset obtained by aggregating the old and the new datasets, cf. [11] and [72]. Intuitively, the ERM-RER-optimal measure obtained after the aggregation of datasets might exhibit better generalization capabilities, see for instance [11]. This analysis is the motivation of the sensitivity, which is defined as follows.

*Definition 7 (Sensitivity):* Given the $\sigma$-finite measure $Q$ and the positive real $\lambda > 0$ in (19), let $S_{Q,\lambda} : (\mathcal{X} \times \mathcal{Y})^n \times \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M})) \to (-\infty, +\infty]$ be a functional such that

$$S_{Q,\lambda}(z,P) = \begin{cases} R_z(P) - R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) & \text{if } \lambda \in \mathcal{K}_{Q,z} \\ +\infty & \text{otherwise,} \end{cases} \quad (191)$$

where the functional $R_z$ is defined in (18) and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25). The sensitivity of the expected empirical risk $R_z$ due to a deviation from $P_{\Theta|Z=z}^{(Q,\lambda)}$ to $P$ is $S_{Q,\lambda}(z, P)$.

Recently, the following exact expression for the sensitivity $S_{Q,\lambda}(z, P)$ in (191) was introduced in [11].

*Theorem 14 (Theorem 1 in [11]):* The sensitivity $S_{Q,\lambda}(z, P)$ in (191) satisfies

$$S_{Q,\lambda}(z,P) = \lambda(D(P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|) + D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|) - D(P\|Q\|)), \quad (192)$$

where the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25).

The following theorem introduces an upper bound on the absolute value of the sensitivity $S_{Q,\lambda}(z, P)$ in (191), which requires the calculation of only one of the relative entropies in Theorem 14.

*Theorem 15:* For all $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, the sensitivity $S_{Q,\lambda}(z, P)$ in (191) satisfies

$$|S_{Q,\lambda}(z, P)| \le \sqrt{2\beta_{Q,z}^2 D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|)}, \quad (193)$$

where the constant $\beta_{Q,z}$ is defined in (163).

*Proof:* The proof is presented in Appendix T. ■

Note that equality holds in (193) in the trivial case in which the empirical risk function is not separable with respect to $Q$ (Definition 5). In such case, for all $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, it holds that $S_{Q,\lambda}(z, P) = 0$ and $\beta_{Q,z} = 0$.

Theorem 15 establishes an upper and a lower bound on the increase and decrease of the expected empirical risk that can be obtained by deviating from the optimal solution of the ERM-RER problem in (19). More specifically, note that for all probability measures $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, it holds that,

$$R_z(P) \ge R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) - \sqrt{2\beta_{Q,z}^2 D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|)} \quad (194)$$

$$R_z(P) \le R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) + \sqrt{2\beta_{Q,z}^2 D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|)} \quad (195)$$

## B. Generalization Error

This section unveils the interesting connection between the notion of sensitivity and the notion of generalization error of the Gibbs algorithm, cf. [9]. The Generalization error is defined under the assumption that datasets are sampled from a probability measure

$$P_Z \in \Delta((\mathcal{X} \times \mathcal{Y})^n, \mathcal{F}), \quad (196)$$
the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal if and only if the reference measure $Q$ is coherent.

*Proof:* The proof is divided into two parts. The first part shows that if for all $(\delta, \epsilon) \in (\rho^*, +\infty) \times (0, 1)$, there always exists a $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) is $(\delta, \epsilon)$-optimal, then, the measure $Q$ is coherent. The second part deals with the converse.

The first part is as follows. Let $\gamma \in \mathcal{K}_{Q,z}$ be such that

$$P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{L}_z(\delta)) > 1 - \epsilon, \quad (188)$$

then, for all measurable subsets $\mathcal{A}$ of $\mathcal{L}_z(\delta)$, it holds that

$$1 - \epsilon < P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{L}_z(\delta)) \quad (189) \\ = \int_{\mathcal{A}} \frac{dP_{\Theta|Z=z}^{(Q,\gamma)}(\nu)dQ(\nu)}{dQ} + \int_{\mathcal{L}_z(\delta)\setminus\mathcal{A}} \frac{dP_{\Theta|Z=z}^{(Q,\gamma)}(\nu)dQ(\nu)}{dQ},$$

which, together with Lemma 2, implies that there exists at least one measurable subset $\mathcal{A}$ for which $Q(\mathcal{A}) > 0$, and thus,

$$Q(\mathcal{L}_z(\delta)) > Q(\mathcal{A}) > 0, \quad (190)$$

which implies that the measure $Q$ is coherent. This completes the first part of the proof.

The second part of the proof is as follows. Under the assumption that the measure $Q$ is coherent, it follows that $\delta_{Q,z}^* = \rho^*$. Then, from Theorem 12, it follows that for all $(\delta, \epsilon) \in (\delta_{Q,z}^*, +\infty) \times (0, 1)$, there always exists a $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), such that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is $(\delta, \epsilon)$-optimal. This completes the second part of the proof. ■

## XI. SENSITIVITY AND GENERALIZATION

This section introduces the notion of sensitivity and establishes its connections with the notion of generalization error of the Gibbs algorithm, cf. [9].

### A. Sensitivity

The sensitivity of the expected empirical risk $R_z$ in (18) to deviations from the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) towards an alternative probability measure $P \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ is introduced as a novel metric to evaluate the generalization capabilities of the ERM-RER-optimal measure $P_{\Theta|Z=z}^{(Q,\lambda)}$. Deviations from the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ towards an alternative probability measure $P$ would allow comparing the ERM-RER-optimal measure with alternative measures (or algorithms). For instance, if new datasets become available, a new ERM-RER problem can be formulated using a larger dataset obtained by aggregating the old and the new datasets, cf. [11] and [72]. Intuitively, the ERM-RER-optimal measure obtained after the aggregation of datasets might exhibit better generalization capabilities, see for instance [11]. This analysis is the motivation of the sensitivity, which is defined as follows.

*Definition 7 (Sensitivity):* Given the $\sigma$-finite measure $Q$ and the positive real $\lambda > 0$ in (19), let $S_{Q,\lambda} : (\mathcal{X} \times \mathcal{Y})^n \times \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M})) \to (-\infty, +\infty]$ be a functional such that

$$S_{Q,\lambda}(z,P) = \begin{cases} R_z(P) - R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) & \text{if } \lambda \in \mathcal{K}_{Q,z} \\ +\infty & \text{otherwise,} \end{cases} \quad (191)$$

where the functional $R_z$ is defined in (18) and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25). The sensitivity of the expected empirical risk $R_z$ due to a deviation from $P_{\Theta|Z=z}^{(Q,\lambda)}$ to $P$ is $S_{Q,\lambda}(z, P)$.

Recently, the following exact expression for the sensitivity $S_{Q,\lambda}(z, P)$ in (191) was introduced in [11].

*Theorem 14 (Theorem 1 in [11]):* The sensitivity $S_{Q,\lambda}(z, P)$ in (191) satisfies

$$S_{Q,\lambda}(z,P) = \lambda(D(P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|) + D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|) - D(P\|Q\|)), \quad (192)$$

where the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25).

The following theorem introduces an upper bound on the absolute value of the sensitivity $S_{Q,\lambda}(z, P)$ in (191), which requires the calculation of only one of the relative entropies in Theorem 14.

*Theorem 15:* For all $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, the sensitivity $S_{Q,\lambda}(z, P)$ in (191) satisfies

$$|S_{Q,\lambda}(z, P)| \le \sqrt{2\beta_{Q,z}^2 D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|)}, \quad (193)$$

where the constant $\beta_{Q,z}$ is defined in (163).

*Proof:* The proof is presented in Appendix T. ■

Note that equality holds in (193) in the trivial case in which the empirical risk function is not separable with respect to $Q$ (Definition 5). In such case, for all $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, it holds that $S_{Q,\lambda}(z, P) = 0$ and $\beta_{Q,z} = 0$.

Theorem 15 establishes an upper and a lower bound on the increase and decrease of the expected empirical risk that can be obtained by deviating from the optimal solution of the ERM-RER problem in (19). More specifically, note that for all probability measures $P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, it holds that,

$$R_z(P) \ge R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) - \sqrt{2\beta_{Q,z}^2 D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|)} \quad (194)$$

$$R_z(P) \le R_z(P_{\Theta|Z=z}^{(Q,\lambda)}) + \sqrt{2\beta_{Q,z}^2 D(P\|P_{\Theta|Z=z}^{(Q,\lambda)}\|Q\|)} \quad (195)$$

### B. Generalization Error

This section unveils the interesting connection between the notion of sensitivity and the notion of generalization error of the Gibbs algorithm, cf. [9]. The Generalization error is defined under the assumption that datasets are sampled from a probability measure

$$P_Z \in \Delta((\mathcal{X} \times \mathcal{Y})^n, \mathcal{F}), \quad (196)$$
where $\mathcal{F}$ denotes a given $\sigma$-field on the set $(\mathcal{X} \times \mathcal{Y})^n$. For such a probability measure $P_Z$ in (196), let the set $\mathcal{K}_{Q,P_Z} \subset \mathbb{R}$ be

$$ \mathcal{K}_{Q,P_Z} = \bigcap_{z \in \text{supp } P_Z} \mathcal{K}_{Q,z}, \quad (197) $$

where the $\sigma$-finite measure $Q$ is in (19). The set $\mathcal{K}_{Q,P_Z}$ in (197) can be empty for some choices of the $\sigma$-finite measure $Q$. Nonetheless, from Lemma 1, it follows that if $Q$ is a probability measure, then,

$$ \mathcal{K}_{Q,P_Z} = (0, +\infty). \quad (198) $$

Under the assumption that datasets are sampled from $P_Z$ in (196), the generalization error of the Gibbs algorithm with parameters $Q$ and $\lambda$, is defined as the expectation with respect to the product measure $P_{\Theta|Z}^{(Q,\lambda)} \cdot P_Z$, with $P_{\Theta|Z}^{(Q,\lambda)}$ in (25), of difference between: (a) the population risk due to a model $\theta \in \mathcal{M}$,

$$ \int L_z(\theta) dP_Z(z) \quad (199) $$

with the function $L_z$ defined in (3); and (b) The empirical risk induced by the model $\theta$ with respect to a training dataset $z$, that is, $L_z(\theta)$. More specifically, the generalization error of the Gibbs algorithm with parameters $Q$ and $\lambda$ is

$$ \begin{align} & \iint \left( \int L_z(\theta) dP_Z(z) - L_\nu(\theta) \right) dP_{\Theta|Z=\nu}^{(Q,\lambda)}(\theta) dP_Z(\nu) \nonumber \\ &= \iint \left( \int L_z(\theta) dP_Z(z) \right) dP_{\Theta}^{(Q,\lambda)}(\theta) \nonumber \\ &\quad - \int L_\nu(\theta) dP_{\Theta|Z=\nu}^{(Q,\lambda)}(\theta) dP_Z(\nu) \tag{200} \end{align} $$

$$ \begin{align} &= \int \left( \int L_z(\theta) dP_{\Theta}^{(Q,\lambda)}(\theta) \right) dP_Z(z) \nonumber \\ &\quad - \int L_\nu(\theta) dP_{\Theta|Z=\nu}^{(Q,\lambda)}(\theta) dP_Z(\nu) \tag{201} \end{align} $$

$$ = \int \left( \mathbb{R}_\nu (P_{\Theta}^{(Q,\lambda)}) - \mathbb{R}_\nu (P_{\Theta|Z=\nu}^{(Q,\lambda)}) \right) dP_Z(\nu), \quad (202) $$

where the probability measure $P_{\Theta}^{(Q,\lambda)}$ satisfies for all sets $A \in \mathcal{B}(\mathcal{M})$,

$$ P_{\Theta}^{(Q,\lambda)}(A) = \int P_{\Theta|Z=\nu}^{(Q,\lambda)}(A) dP_Z(\nu), \quad (203) $$

and the functional $\mathbb{R}_\nu$ is defined in (18).

The following theorem establishes a connection between sensitivity and generalization error in the particular case in which $Q$ in (19) is a probability measure.

**Theorem 16:** Under the assumption that datasets are sampled from $P_Z$ in (196), the generalization error of the Gibbs algorithm with parameters $Q$ (a probability measure) and $\lambda > 0$, is

$$ \int S_{Q,\lambda} (\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu), \quad (204) $$

where the functional $S_{Q,\lambda}$ is in (191); and the probability measure $P_{\Theta}^{(Q,\lambda)}$ is in (203).

*Proof:* The proof uses the fact that under the assumption that $Q$ is a probability measure, for all $\nu \in \text{supp } P_Z$, it follows from Lemma 1 that $\mathcal{K}_{Q,\nu} = (0, +\infty)$. This implies that for all $z \in \text{supp } P_Z$ and for all $\lambda > 0$, the ERM-RER problem in (19), always possesses as solution the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25). Thus, the measure $P_{\Theta}^{(Q,\lambda)}$ in (203) is well defined. Moreover, $S_{Q,\lambda}(z, P_{\Theta}^{(Q,\lambda)}) = \mathbb{R}_z(P_{\Theta}^{(Q,\lambda)}) - \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ and the integral in (202) is also well defined, which completes the proof. ■

**Theorem 16** provides an interesting viewpoint of the generalization error. For instance, the probability measure $P_{\Theta}^{(Q,\lambda)}$ in (196) can be understood as the barycenter of a subset of $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ containing the solutions to ERM-RER problems of the form in (19), with $z \in \text{supp } P_Z$ in (196). Hence, the generalization error of the Gibbs algorithm is the expectation (with respect to $P_Z$) of the sensitivity of the expected empirical risks $\mathbb{R}_z$ in (18) to variations from the ERM-RER-optimal measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ towards the barycenter, i.e., the measure $P_{\Theta}^{(Q,\lambda)}$.

The following definition extends the notion of generalization error to Gibbs algorithms obtained by assuming that the reference measure $Q$ in (19) is a $\sigma$-finite measure. This definition also exploits the relation between the notions of sensitivity and generalization error introduced by Theorem 16.

**Definition 8 (Generalization Error of the Gibbs Algorithm):** Given a $\sigma$-finite measure $Q \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and a real $\lambda > 0$, let the functional $G_{Q,\lambda} : \Delta((\mathcal{X} \times \mathcal{Y})^n, \mathcal{F}) \to (-\infty, +\infty]$ be such that

$$ G_{Q,\lambda}(P_Z) = \begin{cases} \int S_{Q,\lambda}(\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) & \text{if } \lambda \in \mathcal{K}_{Q,P_Z} \\ +\infty & \text{otherwise,} \end{cases} \quad (205) $$

where the functional $S_{Q,\lambda}$ is in (191); the set $\mathcal{K}_{Q,P_Z}$ is in (197); and the probability measure $P_{\Theta}^{(Q,\lambda)}$ is in (203). The generalization error induced by the Gibbs algorithm with parameters $Q$ and $\lambda$ under the assumption that datasets are sampled from the probability measure $P_Z$, is $G_{Q,\lambda}(P_Z)$.

The main difficulty for extending the notion of generalization error to Gibbs algorithms obtained under the assumption that the reference measure is not a probability measure, but a $\sigma$-finite measure, is that the integrals in (202) and (203) might not be well defined. This is essentially due to the fact that, while the ERM-RER problem in (19) always possesses a solution when $Q$ is a probability measure, the existence of a solution when $Q$ is not a probability measure is subject to the condition that for all $z \in \text{supp } P_Z$, $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23). This leads to the condition that $\lambda \in \mathcal{K}_{Q,P_Z}$, with the set $\mathcal{K}_{Q,P_Z}$ in (197). When such a condition is not met, the definition of sensitivity is void.

The following theorem provides a closed-form expression for the generalization error of the Gibbs algorithm in the general case in which the reference measure $Q$ in (19) is a $\sigma$-finite measure.
where $\mathcal{F}$ denotes a given $\sigma$-field on the set $(\mathcal{X} \times \mathcal{Y})^n$. For such a probability measure $P_Z$ in (196), let the set $\mathcal{K}_{Q,P_Z} \subset \mathbb{R}$ be

$$ \mathcal{K}_{Q,P_Z} = \bigcap_{z \in \text{supp } P_Z} \mathcal{K}_{Q,z}, \quad (197) $$

where the $\sigma$-finite measure $Q$ is in (19). The set $\mathcal{K}_{Q,P_Z}$ in (197) can be empty for some choices of the $\sigma$-finite measure $Q$. Nonetheless, from Lemma 1, it follows that if $Q$ is a probability measure, then,

$$ \mathcal{K}_{Q,P_Z} = (0, +\infty). \quad (198) $$

Under the assumption that datasets are sampled from $P_Z$ in (196), the generalization error of the Gibbs algorithm with parameters $Q$ and $\lambda$, is defined as the expectation with respect to the product measure $P_{\Theta|Z}^{(Q,\lambda)} \cdot P_Z$, with $P_{\Theta|Z}^{(Q,\lambda)}$ in (25), of difference between: (a) the population risk due to a model $\theta \in \mathcal{M}$,

$$ \int L_z(\theta) dP_Z(z) \quad (199) $$

with the function $L_z$ defined in (3); and (b) The empirical risk induced by the model $\theta$ with respect to a training dataset $z$, that is, $L_z(\theta)$. More specifically, the generalization error of the Gibbs algorithm with parameters $Q$ and $\lambda$ is

$$ \begin{aligned} & \int \int \left( \int L_z(\theta) dP_Z(z) - L_\nu(\theta) \right) dP_{\Theta|Z=\nu}^{(Q,\lambda)}(\theta) dP_Z(\nu) \\ &= \int \left( \int L_z(\theta) dP_Z(z) \right) dP_{\Theta}^{(Q,\lambda)}(\theta) \\ &\quad - \int L_\nu(\theta) dP_{\Theta|Z=\nu}^{(Q,\lambda)}(\theta) dP_Z(\nu) \end{aligned} \quad (200) $$

$$ \begin{aligned} &= \int \left( \int L_z(\theta) dP_{\Theta}^{(Q,\lambda)}(\theta) \right) dP_Z(z) \\ &\quad - \int L_\nu(\theta) dP_{\Theta|Z=\nu}^{(Q,\lambda)}(\theta) dP_Z(\nu) \end{aligned} \quad (201) $$

$$ = \int \left( \mathbb{R}_\nu (P_{\Theta}^{(Q,\lambda)}) - \mathbb{R}_\nu (P_{\Theta|Z=\nu}^{(Q,\lambda)}) \right) dP_Z(\nu), \quad (202) $$

where the probability measure $P_{\Theta}^{(Q,\lambda)}$ satisfies for all sets $A \in \mathcal{B}(\mathcal{M})$,

$$ P_{\Theta}^{(Q,\lambda)}(A) = \int P_{\Theta|Z=\nu}^{(Q,\lambda)}(A) dP_Z(\nu), \quad (203) $$

and the functional $\mathbb{R}_\nu$ is defined in (18).

The following theorem establishes a connection between sensitivity and generalization error in the particular case in which $Q$ in (19) is a probability measure.

**Theorem 16:** Under the assumption that datasets are sampled from $P_Z$ in (196), the generalization error of the Gibbs algorithm with parameters $Q$ (a probability measure) and $\lambda > 0$, is

$$ \int S_{Q,\lambda} (\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu), \quad (204) $$

where the functional $S_{Q,\lambda}$ is in (191); and the probability measure $P_{\Theta}^{(Q,\lambda)}$ is in (203).

*Proof:* The proof uses the fact that under the assumption that $Q$ is a probability measure, for all $\nu \in \text{supp } P_Z$, it follows from Lemma 1 that $\mathcal{K}_{Q,\nu} = (0, +\infty)$. This implies that for all $z \in \text{supp } P_Z$ and for all $\lambda > 0$, the ERM-RER problem in (19), always possesses as solution the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25). Thus, the measure $P_{\Theta}^{(Q,\lambda)}$ in (203) is well defined. Moreover, $S_{Q,\lambda}(z, P_{\Theta}^{(Q,\lambda)}) = \mathbb{R}_z(P_{\Theta}^{(Q,\lambda)}) - \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)})$ and the integral in (202) is also well defined, which completes the proof. ■

Theorem 16 provides an interesting viewpoint of the generalization error. For instance, the probability measure $P_{\Theta}^{(Q,\lambda)}$ in (196) can be understood as the barycenter of a subset of $\Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ containing the solutions to ERM-RER problems of the form in (19), with $z \in \text{supp } P_Z$ in (196). Hence, the generalization error of the Gibbs algorithm is the expectation (with respect to $P_Z$) of the sensitivity of the expected empirical risks $\mathbb{R}_z$ in (18) to variations from the ERM-RER-optimal measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ towards the barycenter, i.e., the measure $P_{\Theta}^{(Q,\lambda)}$.

The following definition extends the notion of generalization error to Gibbs algorithms obtained by assuming that the reference measure $Q$ in (19) is a $\sigma$-finite measure. This definition also exploits the relation between the notions of sensitivity and generalization error introduced by Theorem 16.

**Definition 8 (Generalization Error of the Gibbs Algorithm):** Given a $\sigma$-finite measure $Q \in \Delta(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and a real $\lambda > 0$, let the functional $G_{Q,\lambda} : \Delta((\mathcal{X} \times \mathcal{Y})^n, \mathcal{F}) \to (-\infty, +\infty]$ be such that

$$ G_{Q,\lambda}(P_Z) = \begin{cases} \int S_{Q,\lambda}(\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) & \text{if } \lambda \in \mathcal{K}_{Q,P_Z} \\ +\infty & \text{otherwise,} \end{cases} \quad (205) $$

where the functional $S_{Q,\lambda}$ is in (191); the set $\mathcal{K}_{Q,P_Z}$ is in (197); and the probability measure $P_{\Theta}^{(Q,\lambda)}$ is in (203). The generalization error induced by the Gibbs algorithm with parameters $Q$ and $\lambda$ under the assumption that datasets are sampled from the probability measure $P_Z$, is $G_{Q,\lambda}(P_Z)$.

The main difficulty for extending the notion of generalization error to Gibbs algorithms obtained under the assumption that the reference measure is not a probability measure, but a $\sigma$-finite measure, is that the integrals in (202) and (203) might not be well defined. This is essentially due to the fact that, while the ERM-RER problem in (19) always possesses a solution when $Q$ is a probability measure, the existence of a solution when $Q$ is not a probability measure is subject to the condition that for all $z \in \text{supp } P_Z$, $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23). This leads to the condition that $\lambda \in \mathcal{K}_{Q,P_Z}$, with the set $\mathcal{K}_{Q,P_Z}$ in (197). When such a condition is not met, the definition of sensitivity is void.

The following theorem provides a closed-form expression for the generalization error of the Gibbs algorithm in the general case in which the reference measure $Q$ in (19) is a $\sigma$-finite measure.
**Theorem 17:** If $\lambda \in K_{Q,P_Z}$, with $K_{Q,P_Z}$ in (197), the generalization error $G_{Q,\lambda}(P_Z)$ in (205) satisfies

$$ G_{Q,\lambda}(P_Z) = \lambda \left( \int D(P_{\Theta|Z=\nu}^{(Q,\lambda)} \| P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) + \int D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu) \right), \quad (206) $$

where for all $z \in \text{supp } P_Z$, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25); and the probability measure $P_{\Theta}^{(Q,\lambda)}$ is defined in (203).

*Proof:* The proof is presented in Appendix U. ■

The terms $\int D(P_{\Theta|Z=\nu}^{(Q,\lambda)} \| P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu)$ and $\int D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)$ in the right-hand side of (206) are respectively the mutual and the lautum information [56] induced by a joint probability measure $P_{\Theta,Z}$ whose marginals are $P_Z$ in (196) and $P_{\Theta}^{(Q,\lambda)}$ in (203). When the reference measure $Q$ in (19) is a probability measure, Theorem 17 reduces to [9, Theorem 1]. Interestingly, independently of whether the reference measure $Q$ in (19) is a probability measure, or whether the $n$ data points in the datasets are independent and identically distributed, the generalization error $G_{Q,\lambda}(P_Z)$ in (205) is always a factor of the sum of the mutual and lautum information induced by the joint probability measure $P_{\Theta,Z}$ mentioned above.

Theorem 17 also provides an alternative interpretation of the generalization error $G_{Q,\lambda}(P_Z)$ in (205). Note that by writing one of the factors in the right-hand side of (206) as

$$ \int \left( D(P_{\Theta|Z=\nu}^{(Q,\lambda)} \| P_{\Theta}^{(Q,\lambda)}) + D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) \right) dP_Z(\nu), $$

it becomes clear that $G_{Q,\lambda}(P_Z)$ is the expectation with respect to $P_Z$ of the symmetrized Kullback-Leibler divergence, also known as Jeffrey's divergence [69], of the probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $P_{\Theta}^{(Q,\lambda)}$. That is, the solution to the ERM-RER problem in (19) and the barycenter induced by $P_Z$.

The following theorem provides an upper-bound on the generalization error of the Gibbs algorithm only in terms of the lautum information induced by such a joint probability measure $P_{\Theta,Z}$.

**Theorem 18:** The generalization error $G_{Q,\lambda}(P_Z)$ in (205) satisfies for all $\lambda \in K_{Q,P_Z}$,

$$ 0 \le G_{Q,\lambda}(P_Z) \le \sqrt{2\sigma_Q^2 \int D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)}, \quad (207) $$

where for all $z \in \text{supp } P_Z$, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25); the probability measure $P_{\Theta}^{(Q,\lambda)}$ is defined in (203); and

$$ \sigma_Q = \sup \{ \beta_{Q,z} : z \in (\mathcal{X} \times \mathcal{Y})^n \}, \quad (208) $$

with $\beta_{Q,z}$ in (163).

*Proof:* The proof of the inequality $G_{Q,\lambda}(P_Z) \ge 0$ follows from observing that for all $\nu \in (\mathcal{X} \times \mathcal{Y})^n$, the terms $D(P_{\Theta|Z=\nu}^{(Q,\lambda)} \| P_{\Theta}^{(Q,\lambda)})$ and $D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)})$ in (206) are

nonnegative (Theorem 1). The proof of the remaining inequality follows from (205) and the following inequalities:

$$ G_{Q,\lambda}(P_Z) = \left| \int S_{Q,\lambda}(\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) \right| \quad (209) $$

$$ \le \left| \int S_{Q,\lambda}(\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) \right| \quad (210) $$

$$ \le \sqrt{2\beta_{Q,\nu} D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)}, \quad (211) $$

$$ \le \sqrt{2\sigma_Q^2 D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)} \quad (212) $$

$$ \le \sqrt{2\sigma_Q^2 \int D(P_{\Theta}^{(Q,\lambda)} \| P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)}, \quad (213) $$

where the equality in (209) follows from (205); the inequality in (210) follows from [59, Theorem 1.5.9(c)]; the inequality in (211) follows from Theorem 15; the inequality in (212) follows from (208); and the inequality in (213) follows from Jensen's inequality [59, Section 6.3.5]. This completes the proof. ■

In a nutshell, the generalization error $G_{Q,\lambda}(P_Z)$ in (205) is upper bounded up to a constant factor by the square root of the lautum information induced by the joint probability measure $P_{\Theta,Z}$ mentioned above. Theorem 18 is reminiscent of [33, Theorem 1], which provides a similar upper-bound on $G_{Q,\lambda}(P_Z)$ using the mutual information instead of the lautum information induced by the joint probability measure $P_{\Theta,Z}$. The interest in Theorem 18 for the specific case of the Gibbs algorithm, lies on the fact that it holds under milder conditions than those in [33, Theorem 1]. For instance, no additional conditions on the loss function $\ell$ in (2) concerning sub-Gaussianity are assumed. Moreover, the probability measure $P_Z$ from which datasets are sampled is not necessarily a product measure.

## XII. CONCLUSIONS AND FINAL REMARKS

The classical ERM-RER problem in (19) has been studied under the assumption that the reference measure $Q$ is a $\sigma$-finite measure, instead of a probability measure, which leads to a more general problem that includes the ERM problem with (discrete or differential) entropy regularization and the information-risk minimization problem. While in the case in which the reference measure is a probability measure the solution to the ERM-RER problem always exists, in this general case, the existence of a solution is subject to a condition that depends on the loss function, the reference measure, the regularization factor, and the training dataset. When a solution exists, it has been proved that it is unique. Additionally, if it exists, such a solution and the reference measure are mutually absolutely continuous. Interestingly, the empirical risk observed when models are sampled from the ERM-RER-optimal probability measure is a sub-Gaussian random variable that exhibits a PAC guarantee for the ERM problem. That is, for some positive $\delta$ and $\epsilon$, it is shown that there always exist some parameters for the ERM-RER problem such that the set of models that induce an empirical risk smaller than $\delta$ exhibits a probability that is not smaller than $1-\epsilon$. Interestingly,
**Theorem 17:** If $\lambda \in K_{Q,P_Z}$, with $K_{Q,P_Z}$ in (197), the generalization error $G_{Q,\lambda}(P_Z)$ in (205) satisfies

$$ G_{Q,\lambda}(P_Z) = \lambda \left( \int D(P_{\Theta|Z=\nu}^{(Q,\lambda)} || P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) + \int D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu) \right), \quad (206) $$

where for all $z \in \text{supp } P_Z$, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25); and the probability measure $P_{\Theta}^{(Q,\lambda)}$ is defined in (203).

*Proof:* The proof is presented in Appendix U. ■

<figure><img src="image_8.png" /></figure>

The terms $\int D(P_{\Theta|Z=\nu}^{(Q,\lambda)} || P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu)$ and $\int D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)$ in the right-hand side of (206) are respectively the mutual and the lautum information [56] induced by a joint probability measure $P_{\Theta,Z}$ whose marginals are $P_Z$ in (196) and $P_{\Theta}^{(Q,\lambda)}$ in (203). When the reference measure $Q$ in (19) is a probability measure, Theorem 17 reduces to [9, Theorem 1]. Interestingly, independently of whether the reference measure $Q$ in (19) is a probability measure, or whether the $n$ data points in the datasets are independent and identically distributed, the generalization error $G_{Q,\lambda}(P_Z)$ in (205) is always a factor of the sum of the mutual and lautum information induced by the joint probability measure $P_{\Theta,Z}$ mentioned above.

Theorem 17 also provides an alternative interpretation of the generalization error $G_{Q,\lambda}(P_Z)$ in (205). Note that by writing one of the factors in the right-hand side of (206) as

$$ \int \left( D(P_{\Theta|Z=\nu}^{(Q,\lambda)} || P_{\Theta}^{(Q,\lambda)}) + D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) \right) dP_Z(\nu), $$

it becomes clear that $G_{Q,\lambda}(P_Z)$ is the expectation with respect to $P_Z$ of the symmetrized Kullback-Leibler divergence, also known as Jeffrey's divergence [69], of the probability measures $P_{\Theta|Z=z}^{(Q,\lambda)}$ and $P_{\Theta}^{(Q,\lambda)}$. That is, the solution to the ERM-RER problem in (19) and the barycenter induced by $P_Z$.

The following theorem provides an upper-bound on the generalization error of the Gibbs algorithm only in terms of the lautum information induced by such a joint probability measure $P_{\Theta,Z}$.

**Theorem 18:** The generalization error $G_{Q,\lambda}(P_Z)$ in (205) satisfies for all $\lambda \in K_{Q,P_Z}$,

$$ 0 \le G_{Q,\lambda}(P_Z) \le \sqrt{2\sigma_Q^2 \int D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)}, \quad (207) $$

where for all $z \in \text{supp } P_Z$, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25); the probability measure $P_{\Theta}^{(Q,\lambda)}$ is defined in (203); and

$$ \sigma_Q = \sup \{ \beta_{Q,z} : z \in (\mathcal{X} \times \mathcal{Y})^n \}, \quad (208) $$

with $\beta_{Q,z}$ in (163).

*Proof:* The proof of the inequality $G_{Q,\lambda}(P_Z) \ge 0$ follows from observing that for all $\nu \in (\mathcal{X} \times \mathcal{Y})^n$, the terms $D(P_{\Theta|Z=\nu}^{(Q,\lambda)} || P_{\Theta}^{(Q,\lambda)})$ and $D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)})$ in (206) are

nonnegative (Theorem 1). The proof of the remaining inequality follows from (205) and the following inequalities:

$$ G_{Q,\lambda}(P_Z) = \left| \int S_{Q,\lambda}(\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) \right| \quad (209) $$

$$ \le \left| \int S_{Q,\lambda}(\nu, P_{\Theta}^{(Q,\lambda)}) dP_Z(\nu) \right| \quad (210) $$

$$ \le \sqrt{2\beta_{Q,\nu} D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)}, \quad (211) $$

$$ \le \sqrt{2\sigma_Q^2 D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)} \quad (212) $$

$$ \le \sqrt{2\sigma_Q^2 \int D(P_{\Theta}^{(Q,\lambda)} || P_{\Theta|Z=\nu}^{(Q,\lambda)}) dP_Z(\nu)}, \quad (213) $$

where the equality in (209) follows from (205); the inequality in (210) follows from [59, Theorem 1.5.9(c)]; the inequality in (211) follows from Theorem 15; the inequality in (212) follows from (208); and the inequality in (213) follows from Jensen's inequality [59, Section 6.3.5]. This completes the proof. ■

<figure><img src="image_9.png" /></figure>

In a nutshell, the generalization error $G_{Q,\lambda}(P_Z)$ in (205) is upper bounded up to a constant factor by the square root of the lautum information induced by the joint probability measure $P_{\Theta,Z}$ mentioned above. Theorem 18 is reminiscent of [33, Theorem 1], which provides a similar upper-bound on $G_{Q,\lambda}(P_Z)$ using the mutual information instead of the lautum information induced by the joint probability measure $P_{\Theta,Z}$. The interest in Theorem 18 for the specific case of the Gibbs algorithm, lies on the fact that it holds under milder conditions than those in [33, Theorem 1]. For instance, no additional conditions on the loss function $\ell$ in (2) concerning sub-Gaussianity are assumed. Moreover, the probability measure $P_Z$ from which datasets are sampled is not necessarily a product measure.

## XII. CONCLUSIONS AND FINAL REMARKS

The classical ERM-RER problem in (19) has been studied under the assumption that the reference measure $Q$ is a $\sigma$-finite measure, instead of a probability measure, which leads to a more general problem that includes the ERM problem with (discrete or differential) entropy regularization and the information-risk minimization problem. While in the case in which the reference measure is a probability measure the solution to the ERM-RER problem always exists, in this general case, the existence of a solution is subject to a condition that depends on the loss function, the reference measure, the regularization factor, and the training dataset. When a solution exists, it has been proved that it is unique. Additionally, if it exists, such a solution and the reference measure are mutually absolutely continuous. Interestingly, the empirical risk observed when models are sampled from the ERM-RER-optimal probability measure is a sub-Gaussian random variable that exhibits a PAC guarantee for the ERM problem. That is, for some positive $\delta$ and $\epsilon$, it is shown that there always exist some parameters for the ERM-RER problem such that the set of models that induce an empirical risk smaller than $\delta$ exhibits a probability that is not smaller than $1-\epsilon$. Interestingly,
none of these results relies on statistical assumptions on the datasets.

The sensitivity of the expected empirical risk to deviations from the ERM-RER-optimal measure to alternative measures is introduced as a new performance metric to evaluate the generalization capabilities of the Gibbs algorithm. In particular, an upper bound on the absolute value of the sensitivity, which depends on the training dataset, is presented. This bound is formed by a constant factor and the square root of the relative entropy of the alternative measure (the deviation) with respect to the ERM-RER solution. Finally, it is shown that the expectation of the sensitivity (with respect to the datasets) to deviations towards a particular measure is equivalent to the generalization error of the Gibbs algorithm. Equipped with this observation, the generalization error is shown to be in the most general case, up to a constant factor, the sum of the mutual and lautum information between the models and the datasets, which was a result known exclusively for the case in which the reference is a probability measure, cf. [9]. From this perspective, it is argued that the study of the generalization capabilities of the Gibbs algorithm based on generalization error is a significantly narrow view. This is essentially because it is looking at an expectation of the sensitivity to deviations to a particular measure, i.e., the barycenter of the set of ERM-RER solutions induced by a prior on the datasets. A broader view is offered by the study of the sensitivity to deviations towards other measures, i.e., ERM-RER-optimal measures obtained with different training data sets. This approach has lead already to a few initial results in [11] that highlight the connections to sensitivity, training error, and test error. Nonetheless, the study of the sensitivity in the aim of describing the generalization capabilities of learning algorithms remains by now as an open problem.

## APPENDIX A
PROOF OF THEOREM 2

Consider the function $f: [0, +\infty) \to \mathbb{R}$ such that

$$f(x) = \begin{cases} x \log(x) & \text{if } x > 0 \\ 0 & \text{if } x = 0, \end{cases} \quad (214)$$

and note that it is strictly convex. From the assumption that for all $i \in \{1, 2\}$, $P_i$ and $Q_i$ are both measures on the same measurable space $(\Omega, \mathcal{F})$, with $P_i$ absolutely continuous with respect to $Q_i$, let $g: \Omega \to [0, \infty)$ be the function

$$g(x) = \frac{d(\lambda P_1 + (1-\lambda)P_2)}{d(\lambda Q_1 + (1-\lambda)Q_2)}(x), \quad (215)$$

where $\frac{d(\lambda P_1 + (1-\lambda)P_2)}{d(\lambda Q_1 + (1-\lambda)Q_2)}$ is the Radon-Nikodym derivative of the measure $\lambda P_1 + (1-\lambda)P_2$ with respect to $\lambda Q_1 + (1-\lambda)Q_2$.

Using this notation, for all $\lambda \in (0, 1)$,

$$ \begin{align} & D(\lambda P_1 + (1-\lambda)P_2 \| \lambda Q_1 + (1-\lambda)Q_2) \nonumber \\ & \quad - \lambda D(P_1 \| Q_1) + (1-\lambda)D(P_2 \| Q_2) \tag{216} \\ &= \int \log(g(x)) d(\lambda P_1 + (1-\lambda)P_2)(x) \nonumber \\ & \quad - \lambda \int \log\left(\frac{dP_1}{dQ_1}(x)\right) dP_1(x) - (1-\lambda) \int \log\left(\frac{dP_2}{dQ_2}(x)\right) dP_2(x) \nonumber \\ &= \lambda \int \log(g(x)) dP_1(x) + (1-\lambda) \int \log(g(x)) dP_2(x) \nonumber \\ & \quad - \lambda \int \log\left(\frac{dP_1}{dQ_1}(x)\right) dP_1(x) - (1-\lambda) \int \log\left(\frac{dP_2}{dQ_2}(x)\right) dP_2(x) \nonumber \\ &= \lambda \int \log\left(\left(\frac{dP_1}{dQ_1}(x)\right)^{-1} g(x)\right) dP_1(x) \nonumber \\ & \quad + (1-\lambda) \int \log\left(\left(\frac{dP_2}{dQ_2}(x)\right)^{-1} g(x)\right) dP_2(x) \nonumber \\ &= \lambda \int \frac{dP_1}{dQ_1}(x) \log\left(\left(\frac{dP_1}{dQ_1}(x)\right)^{-1} g(x)\right) dQ_1(x) \nonumber \\ & \quad + (1-\lambda) \int \frac{dP_2}{dQ_2}(x) \log\left(\left(\frac{dP_2}{dQ_2}(x)\right)^{-1} g(x)\right) dQ_2(x) \nonumber \\ &= \lambda \int \frac{g(x) \frac{dP_1}{dQ_1}(x)}{g(x)} \log\left(\left(\frac{dP_1}{dQ_1}(x)\right)^{-1} g(x)\right) dQ_1(x) \nonumber \\ & \quad + (1-\lambda) \int \frac{g(x) \frac{dP_2}{dQ_2}(x)}{g(x)} \log\left(\left(\frac{dP_2}{dQ_2}(x)\right)^{-1} g(x)\right) dQ_2(x) \nonumber \\ &= -\lambda \int g(x) f\left(\frac{dP_1}{dQ_1}(x)(g(x))^{-1}\right) dQ_1(x) \nonumber \\ & \quad - (1-\lambda) \int g(x) f\left(\frac{dP_2}{dQ_2}(x)(g(x))^{-1}\right) dQ_2(x), \tag{217} \end{align} $$

where the function $f$ is defined in (214). Let $\beta_1$ and $\beta_2$ be the following constants:

$$\beta_1 \triangleq \int g(\nu) dQ_1(\nu) \text{ and } \beta_2 \triangleq \int g(\nu) dQ_2(\nu). \quad (218)$$

From (217) and (218), it follows that for all $\lambda \in (0, 1)$,

$$ \begin{align} & D(\lambda P_1 + (1-\lambda)P_2 \| \lambda Q_1 + (1-\lambda)Q_2) \nonumber \\ & \quad - \lambda D(P_1 \| Q_1) + (1-\lambda)D(P_2 \| Q_2) \nonumber \\ &= -\lambda \beta_1 \int \frac{g(x)}{\beta_1} f\left(\frac{dP_1}{dQ_1}(x)(g(x))^{-1}\right) dQ_1(x) \nonumber \\ & \quad - (1-\lambda)\beta_2 \int \frac{g(x)}{\beta_2} f\left(\frac{dP_2}{dQ_2}(x)(g(x))^{-1}\right) dQ_2(x) \nonumber \\ & \le -\lambda \beta_1 f\left(\int \frac{g(x)}{\beta_1} \frac{dP_1}{dQ_1}(x)(g(x))^{-1} dQ_1(x)\right) \tag{219} \\ & \quad - (1-\lambda)\beta_2 f\left(\int \frac{g(x)}{\beta_2} \frac{dP_2}{dQ_2}(x)(g(x))^{-1} dQ_2(x)\right) \nonumber \\ & = -\lambda \beta_1 f\left(\frac{1}{\beta_1} \int dP_1(x)\right) - (1-\lambda)\beta_2 f\left(\frac{1}{\beta_2} \int dP_2(x)\right) \tag{220} \end{align} $$
none of these results relies on statistical assumptions on the datasets.

The sensitivity of the expected empirical risk to deviations from the ERM-RER-optimal measure to alternative measures is introduced as a new performance metric to evaluate the generalization capabilities of the Gibbs algorithm. In particular, an upper bound on the absolute value of the sensitivity, which depends on the training dataset, is presented. This bound is formed by a constant factor and the square root of the relative entropy of the alternative measure (the deviation) with respect to the ERM-RER solution. Finally, it is shown that the expectation of the sensitivity (with respect to the datasets) to deviations towards a particular measure is equivalent to the generalization error of the Gibbs algorithm. Equipped with this observation, the generalization error is shown to be in the most general case, up to a constant factor, the sum of the mutual and lautum information between the models and the datasets, which was a result known exclusively for the case in which the reference is a probability measure, cf. [9]. From this perspective, it is argued that the study of the generalization capabilities of the Gibbs algorithm based on generalization error is a significantly narrow view. This is essentially because it is looking at an expectation of the sensitivity to deviations to a particular measure, i.e., the barycenter of the set of ERM-RER solutions induced by a prior on the datasets. A broader view is offered by the study of the sensitivity to deviations towards other measures, i.e., ERM-RER-optimal measures obtained with different training data sets. This approach has lead already to a few initial results in [11] that highlight the connections to sensitivity, training error, and test error. Nonetheless, the study of the sensitivity in the aim of describing the generalization capabilities of learning algorithms remains by now as an open problem.

## APPENDIX A
PROOF OF THEOREM 2

Consider the function $f : [0, +\infty) \to \mathbb{R}$ such that

$$f(x) = \begin{cases} x \log(x) & \text{if } x > 0 \\ 0 & \text{if } x = 0, \end{cases} \quad (214)$$

and note that it is strictly convex. From the assumption that for all $i \in \{1, 2\}$, $P_i$ and $Q_i$ are both measures on the same measurable space $(\Omega, \mathcal{F})$, with $P_i$ absolutely continuous with respect to $Q_i$, let $g : \Omega \to [0, \infty)$ be the function

$$g(x) = \frac{d(\lambda P_1 + (1-\lambda)P_2)}{d(\lambda Q_1 + (1-\lambda)Q_2)}(x), \quad (215)$$

where $\frac{d(\lambda P_1 + (1-\lambda)P_2)}{d(\lambda Q_1 + (1-\lambda)Q_2)}$ is the Radon-Nikodym derivative of the measure $\lambda P_1 + (1-\lambda)P_2$ with respect to $\lambda Q_1 + (1-\lambda)Q_2$.

Using this notation, for all $\lambda \in (0, 1)$,

$$ \begin{align} & D(\lambda P_1 + (1-\lambda)P_2 \| \lambda Q_1 + (1-\lambda)Q_2) \nonumber \\ & \quad - \lambda D(P_1 \| Q_1) + (1-\lambda)D(P_2 \| Q_2) \tag{216} \\ &= \int \log(g(x)) d(\lambda P_1 + (1-\lambda)P_2)(x) \nonumber \\ & \quad - \lambda \int \log\left(\frac{dP_1}{dQ_1}(x)\right) dP_1(x) - (1-\lambda) \int \log\left(\frac{dP_2}{dQ_2}(x)\right) dP_2(x) \nonumber \\ &= \lambda \int \log(g(x)) dP_1(x) + (1-\lambda) \int \log(g(x)) dP_2(x) \nonumber \\ & \quad - \lambda \int \log\left(\frac{dP_1}{dQ_1}(x)\right) dP_1(x) - (1-\lambda) \int \log\left(\frac{dP_2}{dQ_2}(x)\right) dP_2(x) \nonumber \\ &= \lambda \int \log\left(\left(\frac{dP_1}{dQ_1}(x)\right)^{-1} g(x)\right) dP_1(x) \nonumber \\ & \quad + (1-\lambda) \int \log\left(\left(\frac{dP_2}{dQ_2}(x)\right)^{-1} g(x)\right) dP_2(x) \nonumber \\ &= \lambda \int \frac{dP_1}{dQ_1}(x) \log\left(\left(\frac{dP_1}{dQ_1}(x)\right)^{-1} g(x)\right) dQ_1(x) \nonumber \\ & \quad + (1-\lambda) \int \frac{dP_2}{dQ_2}(x) \log\left(\left(\frac{dP_2}{dQ_2}(x)\right)^{-1} g(x)\right) dQ_2(x) \nonumber \\ &= \lambda \int \frac{g(x) \frac{dP_1}{dQ_1}(x)}{g(x)} \log\left(\left(\frac{dP_1}{dQ_1}(x)\right)^{-1} g(x)\right) dQ_1(x) \nonumber \\ & \quad + (1-\lambda) \int \frac{g(x) \frac{dP_2}{dQ_2}(x)}{g(x)} \log\left(\left(\frac{dP_2}{dQ_2}(x)\right)^{-1} g(x)\right) dQ_2(x) \nonumber \\ &= -\lambda \int g(x) f\left(\frac{dP_1}{dQ_1}(x)(g(x))^{-1}\right) dQ_1(x) \nonumber \\ & \quad - (1-\lambda) \int g(x) f\left(\frac{dP_2}{dQ_2}(x)(g(x))^{-1}\right) dQ_2(x), \tag{217} \end{align} $$

where the function $f$ is defined in (214). Let $\beta_1$ and $\beta_2$ be the following constants:

$$\beta_1 \triangleq \int g(\nu) dQ_1(\nu) \text{ and } \beta_2 \triangleq \int g(\nu) dQ_2(\nu). \quad (218)$$

From (217) and (218), it follows that for all $\lambda \in (0, 1)$,

$$ \begin{align} & D(\lambda P_1 + (1-\lambda)P_2 \| \lambda Q_1 + (1-\lambda)Q_2) \nonumber \\ & \quad - \lambda D(P_1 \| Q_1) + (1-\lambda)D(P_2 \| Q_2) \nonumber \\ &= -\lambda \beta_1 \int \frac{g(x)}{\beta_1} f\left(\frac{dP_1}{dQ_1}(x)(g(x))^{-1}\right) dQ_1(x) \nonumber \\ & \quad - (1-\lambda)\beta_2 \int \frac{g(x)}{\beta_2} f\left(\frac{dP_2}{dQ_2}(x)(g(x))^{-1}\right) dQ_2(x) \nonumber \\ & \le -\lambda \beta_1 f\left(\int \frac{g(x)}{\beta_1} \frac{dP_1}{dQ_1}(x)(g(x))^{-1} dQ_1(x)\right) \tag{219} \\ & \quad - (1-\lambda)\beta_2 f\left(\int \frac{g(x)}{\beta_2} \frac{dP_2}{dQ_2}(x)(g(x))^{-1} dQ_2(x)\right) \nonumber \\ & = -\lambda \beta_1 f\left(\frac{1}{\beta_1} \int dP_1(x)\right) - (1-\lambda)\beta_2 f\left(\frac{1}{\beta_2} \int dP_2(x)\right) \tag{220} \end{align} $$
$$=-\lambda\beta_1 f\left(\frac{1}{\beta_1}\right) - (1-\lambda)\beta_2 f\left(\frac{1}{\beta_2}\right) \quad (221)$$

$$\leq -f\left(\lambda\beta_1 \frac{1}{\beta_1} + (1-\lambda)\beta_2 \frac{1}{\beta_2}\right) \quad (222)$$

$$=-f(1) \quad (223)$$

$$=0, \quad (224)$$

where the inequalities in (219) and (222) follow from Jensen's inequality [59, Section 6.3.5] and the fact that the function $f$ in (217) is strictly concave. Note that from (218), in (219), for all $i \in \{1, 2\}$, $\int \frac{g(x)}{\beta_i} dQ_i(x) = 1$; while in (222),

$$\lambda\beta_1 + (1-\lambda)\beta_2 = \int g(\nu)d(\lambda Q_1 + (1-\lambda)Q_2)(\nu) \quad (225)$$

$$= \int d(\lambda P_1 + (1-\lambda)P_2)(\nu) \quad (226)$$

$$=\lambda \int dP_1(\nu) + (1-\lambda) \int dP_2(\nu) \quad (227)$$

$$=1. \quad (228)$$

Given the strict convexity of the function $f$ in (214), equality in (219) and (222) hold if and only if $P_1 = P_2$ and $Q_1 = Q_2$. This completes the proof.

## APPENDIX B

### PROOF OF LEMMA 1

The proof is divided into two parts. The first part is as follows. Under the assumption that the set $K_{Q,z}$ in (23) is empty, there is nothing to prove. Alternatively, under the assumption that the set $K_{Q,z}$ is not empty, there always exists a real $b \in K_{Q,z}$, such that $K_{Q,z}(-\frac{1}{b}) > +\infty$. Note that for all $\theta \in \mathcal{M}$,

$$\frac{d}{dt} \exp \left( -\frac{1}{t} \mathbf{L}_z(\boldsymbol{\theta}) \right) = \frac{1}{t^2} \mathbf{L}_z(\boldsymbol{\theta}) \exp \left( -\frac{1}{t} \mathbf{L}_z(\boldsymbol{\theta}) \right) \ge 0, (229)$$

with $\mathbf{L}_z$ in (3). Thus, from (22), it follows that $K_{Q,z}(-\frac{1}{b})$ is nondecreasing with $b$. This implies that $(0, b] \subseteq K_{Q,z}$.

$$\text{Let } b^* \in (0, +\infty] \text{ be} \\ b^* = \sup K_{Q,z}. \quad (230)$$

$$\text{Hence, if } b^* = +\infty, \text{ it follows from (23) that} \\ K_{Q,z} = (0, +\infty). \quad (231)$$

$$\text{Alternatively, if } b^* < +\infty, \text{ it holds that} \\ (0, b^*) \subseteq K_{Q,z} \subseteq (0, b^*]. \quad (232)$$

In either case, it follows that $K_{Q,z}$ is a convex set. This completes the first part of the proof.

The second part of the proof is under the assumption that $Q$ is a probability measure. Under this assumption, for all $\theta \in \mathcal{M}$ and for all $t > 0$, it follows that

$$\exp\left(-\frac{1}{t}\mathbf{L}_z(\boldsymbol{\theta})\right) \le 1, \quad (233)$$

$$\text{with } \mathbf{L}_z \text{ in (3). Thus,} \\ K_{Q,z}\left(-\frac{1}{t}\right) = \log\left(\int \exp\left(-\frac{1}{t}\mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta})\right) \quad (234) \\ \le \log\left(\int dQ(\boldsymbol{\theta})\right) \quad (235) \\ =0, \quad (236)$$

which implies that $(0, +\infty) \subseteq K_{Q,z}$. Thus, if $Q$ is a probability measure, from (23), it holds that $K_{Q,z} = (0, +\infty)$, which completes the proof.

## APPENDIX C

### PROOF OF THEOREM 3

The optimization problem in (19) can be re-written in terms of the Radon-Nikodym derivative of the optimization measure $P$ with respect to the measure $Q$, denoted by $\frac{dP}{dQ}: \mathcal{M} \to [0, \infty)$, which yields:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \int \mathbf{L}_z(\boldsymbol{\theta}) \frac{dP}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \\ + \lambda \int \frac{dP}{dQ}(\boldsymbol{\theta}) \log \left( \frac{dP}{dQ}(\boldsymbol{\theta}) \right) dQ(\boldsymbol{\theta}) \quad (237a)$$

$$\text{s. t.} \quad \int \frac{dP}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) = 1. \quad (237b)$$

The remainder of the proof focuses on the problem in which the optimization is over the function $\frac{dP}{dQ}$ instead of the measure $P$. This is due to the fact that for all $P \in \Delta_Q(\mathcal{M})$, the Radon-Nikodym derivate $\frac{dP}{dQ}$ is unique up to sets of zero measure with respect to the measure $Q$. Let $\mathcal{M}$ be the set of measurable functions $\mathcal{M} \to \mathbb{R}$ with respect to the measurable spaces $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$ that are absolutely integrable with respect to $Q$. That is, for all $\hat{g} \in \mathcal{M}$, it holds that

$$\int |\hat{g}(\boldsymbol{\theta})| dQ(\boldsymbol{\theta}) < \infty. \quad (238)$$

Hence, the optimization problem of interest is:

$$\min_{g \in \mathcal{M}} \int \mathbf{L}_z(\boldsymbol{\theta}) g(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) + \lambda \int g(\boldsymbol{\theta}) \log(g(\boldsymbol{\theta})) dQ(\boldsymbol{\theta}) \quad (239a)$$

$$\text{s. t.} \quad \int g(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) = 1. \quad (239b)$$

Let the Lagrangian of the optimization problem in (239) be the functional $L: \mathcal{M} \times \mathbb{R} \to \mathbb{R}$ such that

$$L(g, \beta) = \int \mathbf{L}_z(\boldsymbol{\nu}) g(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) + \lambda \int g(\boldsymbol{\nu}) \log(g(\boldsymbol{\nu})) dQ(\boldsymbol{\nu}) \\ + \beta \left( \int g(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) - 1 \right), \quad (240)$$

where $\beta$ is a real that acts as a Lagrangian multiplier due to the constraint (239b). Let $\hat{g}: \mathcal{M} \to \mathbb{R}$ be a function in $\mathcal{M}$. The Gateaux differential of the functional $L$ in (240) at $(g, \beta) \in \mathcal{M} \times \mathbb{R}$ in the direction of $\hat{g}$, if it exists, is

$$\partial L(g, \beta; \hat{g}) \triangleq \left. \frac{d}{d\gamma} L(g + \gamma \hat{g}, \beta) \right|_{\gamma=0}. \quad (241)$$

The proof continues under the assumption that the functions $g$ and $\hat{g}$ are such that the Gateaux differential in (241) exists.
$$=-\lambda\beta_1 f\left(\frac{1}{\beta_1}\right) - (1-\lambda)\beta_2 f\left(\frac{1}{\beta_2}\right) \quad (221)$$

$$\leq -f\left(\lambda\beta_1 \frac{1}{\beta_1} + (1-\lambda)\beta_2 \frac{1}{\beta_2}\right) \quad (222)$$

$$=-f(1) \quad (223)$$

$$=0, \quad (224)$$

where the inequalities in (219) and (222) follow from Jensen's inequality [59, Section 6.3.5] and the fact that the function $f$ in (217) is strictly concave. Note that from (218), in (219), for all $i \in \{1, 2\}$, $\int \frac{g(x)}{\beta_i} dQ_i(x) = 1$; while in (222),

$$\lambda\beta_1 + (1-\lambda)\beta_2 = \int g(\nu)d(\lambda Q_1 + (1-\lambda)Q_2)(\nu) \quad (225)$$

$$= \int d(\lambda P_1 + (1-\lambda)P_2)(\nu) \quad (226)$$

$$=\lambda \int dP_1(\nu) + (1-\lambda) \int dP_2(\nu) \quad (227)$$

$$=1. \quad (228)$$

Given the strict convexity of the function $f$ in (214), equality in (219) and (222) hold if and only if $P_1 = P_2$ and $Q_1 = Q_2$. This completes the proof.

# APPENDIX B

## PROOF OF LEMMA 1

The proof is divided into two parts. The first part is as follows. Under the assumption that the set $K_{Q,z}$ in (23) is empty, there is nothing to prove. Alternatively, under the assumption that the set $K_{Q,z}$ is not empty, there always exists a real $b \in K_{Q,z}$, such that $K_{Q,z}(-\frac{1}{b}) > +\infty$. Note that for all $\theta \in \mathcal{M}$,

$$\frac{d}{dt} \exp \left( -\frac{1}{t} \mathbf{L}_z(\boldsymbol{\theta}) \right) = \frac{1}{t^2} \mathbf{L}_z(\boldsymbol{\theta}) \exp \left( -\frac{1}{t} \mathbf{L}_z(\boldsymbol{\theta}) \right) \ge 0, (229)$$

with $\mathbf{L}_z$ in (3). Thus, from (22), it follows that $K_{Q,z}(-\frac{1}{b})$ is nondecreasing with $b$. This implies that $(0, b] \subseteq K_{Q,z}$.

$$\text{Let } b^* \in (0, +\infty] \text{ be} \\ b^* = \sup K_{Q,z}. \quad (230)$$

$$\text{Hence, if } b^* = +\infty, \text{ it follows from (23) that} \\ K_{Q,z} = (0, +\infty). \quad (231)$$

$$\text{Alternatively, if } b^* < +\infty, \text{ it holds that} \\ (0, b^*) \subseteq K_{Q,z} \subseteq (0, b^*]. \quad (232)$$

In either case, it follows that $K_{Q,z}$ is a convex set. This completes the first part of the proof.

The second part of the proof is under the assumption that $Q$ is a probability measure. Under this assumption, for all $\theta \in \mathcal{M}$ and for all $t > 0$, it follows that

$$\exp\left(-\frac{1}{t}\mathbf{L}_z(\boldsymbol{\theta})\right) \le 1, \quad (233)$$

$$\text{with } \mathbf{L}_z \text{ in (3). Thus,} \\ K_{Q,z}\left(-\frac{1}{t}\right) = \log\left(\int \exp\left(-\frac{1}{t}\mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta})\right) \quad (234) \\ \le \log\left(\int dQ(\boldsymbol{\theta})\right) \quad (235) \\ =0, \quad (236)$$

which implies that $(0, +\infty) \subseteq K_{Q,z}$. Thus, if $Q$ is a probability measure, from (23), it holds that $K_{Q,z} = (0, +\infty)$, which completes the proof.

# APPENDIX C

## PROOF OF THEOREM 3

The optimization problem in (19) can be re-written in terms of the Radon-Nikodym derivative of the optimization measure $P$ with respect to the measure $Q$, denoted by $\frac{dP}{dQ}: \mathcal{M} \to [0, \infty)$, which yields:

$$\min_{P \in \Delta_Q(\mathcal{M}, \mathcal{B}(\mathcal{M}))} \int \mathbf{L}_z(\boldsymbol{\theta}) \frac{dP}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) \\ + \lambda \int \frac{dP}{dQ}(\boldsymbol{\theta}) \log \left( \frac{dP}{dQ}(\boldsymbol{\theta}) \right) dQ(\boldsymbol{\theta}) \quad (237a)$$

$$\text{s. t.} \quad \int \frac{dP}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) = 1. \quad (237b)$$

The remainder of the proof focuses on the problem in which the optimization is over the function $\frac{dP}{dQ}$ instead of the measure $P$. This is due to the fact that for all $P \in \Delta_Q(\mathcal{M})$, the Radon-Nikodym derivate $\frac{dP}{dQ}$ is unique up to sets of zero measure with respect to the measure $Q$. Let $\mathcal{M}$ be the set of measurable functions $\mathcal{M} \to \mathbb{R}$ with respect to the measurable spaces $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$ that are absolutely integrable with respect to $Q$. That is, for all $\hat{g} \in \mathcal{M}$, it holds that

$$\int |\hat{g}(\boldsymbol{\theta})| dQ(\boldsymbol{\theta}) < \infty. \quad (238)$$

Hence, the optimization problem of interest is:

$$\min_{g \in \mathcal{M}} \int \mathbf{L}_z(\boldsymbol{\theta}) g(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) + \lambda \int g(\boldsymbol{\theta}) \log(g(\boldsymbol{\theta})) dQ(\boldsymbol{\theta}) \quad (239a)$$

$$\text{s. t.} \quad \int g(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) = 1. \quad (239b)$$

Let the Lagrangian of the optimization problem in (239) be the functional $L: \mathcal{M} \times \mathbb{R} \to \mathbb{R}$ such that

$$L(g, \beta) = \int \mathbf{L}_z(\boldsymbol{\nu}) g(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) + \lambda \int g(\boldsymbol{\nu}) \log(g(\boldsymbol{\nu})) dQ(\boldsymbol{\nu}) \\ + \beta \left( \int g(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) - 1 \right), \quad (240)$$

where $\beta$ is a real that acts as a Lagrangian multiplier due to the constraint (239b). Let $\hat{g}: \mathcal{M} \to \mathbb{R}$ be a function in $\mathcal{M}$. The Gateaux differential of the functional $L$ in (240) at $(g, \beta) \in \mathcal{M} \times \mathbb{R}$ in the direction of $\hat{g}$, if it exists, is

$$\partial L(g, \beta; \hat{g}) \triangleq \left. \frac{d}{d\gamma} L(g + \gamma \hat{g}, \beta) \right|_{\gamma=0}. \quad (241)$$

The proof continues under the assumption that the functions $g$ and $\hat{g}$ are such that the Gateaux differential in (241) exists.
Under such an assumption, let the function $r: \mathbb{R} \to \mathbb{R}$ satisfy for all $\alpha \in (-\epsilon, \epsilon)$, with $\epsilon$ arbitrarily small, that

$$
\begin{align}
r(\alpha) &= \int \mathbf{L}_z(\boldsymbol{\nu}) (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}) \nonumber \\
&\quad + \beta \left( \int (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}) - 1 \right) \nonumber \\
&\quad + \lambda \int (\hat{g}(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}) \tag{242} \\
&= \int g(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \beta) \, dQ(\boldsymbol{\nu}) - \beta \nonumber \\
&\quad + \alpha \left( \int \hat{g}(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \beta) \, dQ(\boldsymbol{\nu}) \right) \nonumber \\
&\quad + \lambda \int (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}), \tag{243}
\end{align}
$$

where the last equality is simply an algebraic re-arrangement of terms. From the assumption that the functions $g$ and $\hat{g}$ are such that the Gateaux differential in (241) exists, it follows that the function $r$ in (243) is differentiable at zero. Note that the first two terms in (243) are independent of $\alpha$; the third term is linear with $\alpha$; and the fourth term can be written using the function $\hat{r}: \mathbb{R} \to \mathbb{R}$ such that for all $\alpha \in (-\epsilon, \epsilon)$, with $\epsilon$ arbitrarily small, satisfies

$$
\begin{align}
\hat{r}(\alpha) &= \lambda \int (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) dQ(\boldsymbol{\nu}) \tag{244} \\
&= \lambda \int f(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) dQ(\boldsymbol{\nu}), \tag{245}
\end{align}
$$

where $f: (0, +\infty) \to \mathbb{R}$ is such that $f(t) = t \log(t)$. Under the same assumption, it follows that the function $\hat{r}$ in (244) is differentiable at zero. That is, the limit

$$
\lim_{\delta \to 0} \frac{1}{\delta} (\hat{r}(\gamma + \delta) - \hat{r}(\gamma)) \quad (246)
$$

exists for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon$ arbitrarily small. Note that the function $f$ in (245) is continuous and differentiable (with finite derivate) in $(0, +\infty)$. Thus, the function $f$ is also Lipschitz continuous. This implies that for all $\theta \in \text{supp } Q$, and for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$ arbitrarily small, it holds that

$$
|f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))| \le c|\hat{g}(\boldsymbol{\theta})||\delta|, (247)
$$

with $\delta > 0$, for some constant $c$ positive and finite. This
implies that

$$
\left| \frac{f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))}{\delta} \right| \le c|\hat{g}(\boldsymbol{\theta})|. (248)
$$

Using these arguments, the limit in (246) satisfies for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$ arbitrarily small, that

$$
\begin{align*}
& \lim_{\delta \to 0} \frac{1}{\delta} (\hat{r}(\gamma + \delta) - \hat{r}(\gamma)) \\
&= \lambda \lim_{\delta \to 0} \int \frac{f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))}{\delta} \, dQ(\boldsymbol{\theta}) \tag{249} \\
&= \lambda \int \hat{f}(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))\hat{g}(\boldsymbol{\theta}) \, dQ(\boldsymbol{\theta}) \tag{250} \\
&< \infty,
\end{align*}
$$

where the function $\hat{f}: (0, +\infty) \to \mathbb{R}$ is the derivative of $f$. That is, $\hat{f}(t) = 1 + \log(t)$. The equality in (249) and the

inequality in (250) follow from noticing that the conditions for
the dominated convergence theorem hold [59, Theorem 1.6.9],
namely:

* For all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$, the inequality in (248) holds;

* The function $\hat{g}$ in (248) satisfies the inequality in (238); and

* For all $\boldsymbol{\theta} \in \text{supp } Q$ and for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$ arbitrarily small, it holds that

$$
\begin{align}
& \lim_{\delta \to 0} \frac{f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))}{\delta} \nonumber \\
&= \frac{d}{d\gamma} f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta})) \tag{251}
\end{align}
$$

$$
= \hat{f}(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))\hat{g}(\boldsymbol{\theta}). \quad (252)
$$

Hence, the derivative of the real function $r$ in (243) is

$$
\begin{align}
\frac{d}{d\alpha} r(\alpha) &= \int \mathbf{L}_z(\boldsymbol{\nu}) \hat{g}(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) + \beta \int \hat{g}(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) \nonumber \\
&\quad + \lambda \int \hat{g}(\boldsymbol{\nu}) (1 + \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu}))) dQ(\boldsymbol{\nu}). \tag{253}
\end{align}
$$

From (241) and (253), it follows that

$$
\begin{align}
& \partial L(g, \beta; \hat{g}) \nonumber \\
& = \int \hat{g}(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \lambda(1 + \log(g(\boldsymbol{\nu}))) + \beta) dQ(\boldsymbol{\nu}). \tag{254}
\end{align}
$$

The relevance of the Gateaux differential in (254) stems from
[73, Theorem 1, page 178], which unveils the fact that a
necessary condition for the functional $L$ in (240) to have
a minimum at $\left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}, \beta \right) \in \mathcal{M} \times \mathbb{R}$ is that for all
functions $\hat{g} \in \mathcal{M}$,

$$
\partial L \left( \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}, \beta; \hat{g} \right) = 0. \quad (255)
$$

From (255), it follows that $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ must satisfy for all functions $\hat{g}$ in $\mathcal{M}$ that

$$
0 = \int \hat{g}(\boldsymbol{\nu}) \left( \mathbf{L}_z(\boldsymbol{\nu}) + \lambda \left( 1 + \log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) \right) \right) + \beta \right) dQ(\boldsymbol{\nu}),
$$

which implies that for all $\boldsymbol{\nu} \in \text{supp } Q$,

$$
\mathbf{L}_z(\boldsymbol{\nu}) + \lambda \left( 1 + \log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) \right) \right) + \beta = 0, \quad (256)
$$

and thus,

$$
\frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) = \exp\left(-\frac{\beta + \lambda}{\lambda}\right) \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\lambda}\right), \quad (257)
$$
Under such an assumption, let the function $r: \mathbb{R} \to \mathbb{R}$ satisfy for all $\alpha \in (-\epsilon, \epsilon)$, with $\epsilon$ arbitrarily small, that

$$
\begin{align}
r(\alpha) &= \int \mathbf{L}_z(\boldsymbol{\nu}) (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}) \nonumber \\
&\quad + \beta \left( \int (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}) - 1 \right) \nonumber \\
&\quad + \lambda \int (\hat{g}(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}) \tag{242} \\
&= \int g(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \beta) \, dQ(\boldsymbol{\nu}) - \beta \nonumber \\
&\quad + \alpha \left( \int \hat{g}(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \beta) \, dQ(\boldsymbol{\nu}) \right) \nonumber \\
&\quad + \lambda \int (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \, dQ(\boldsymbol{\nu}), \tag{243}
\end{align}
$$

where the last equality is simply an algebraic re-arrangement of terms. From the assumption that the functions $g$ and $\hat{g}$ are such that the Gateaux differential in (241) exists, it follows that the function $r$ in (243) is differentiable at zero. Note that the first two terms in (243) are independent of $\alpha$; the third term is linear with $\alpha$; and the fourth term can be written using the function $\hat{r}: \mathbb{R} \to \mathbb{R}$ such that for all $\alpha \in (-\epsilon, \epsilon)$, with $\epsilon$ arbitrarily small, satisfies

$$
\begin{align}
\hat{r}(\alpha) &= \lambda \int (g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) dQ(\boldsymbol{\nu}) \tag{244} \\
&= \lambda \int f(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu})) dQ(\boldsymbol{\nu}), \tag{245}
\end{align}
$$

where $f: (0, +\infty) \to \mathbb{R}$ is such that $f(t) = t \log(t)$. Under the same assumption, it follows that the function $\hat{r}$ in (244) is differentiable at zero. That is, the limit

$$
\lim_{\delta \to 0} \frac{1}{\delta} (\hat{r}(\gamma + \delta) - \hat{r}(\gamma)) \quad (246)
$$

exists for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon$ arbitrarily small. Note that the function $f$ in (245) is continuous and differentiable (with finite derivate) in $(0, +\infty)$. Thus, the function $f$ is also Lipschitz continuous. This implies that for all $\theta \in \text{supp } Q$, and for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$ arbitrarily small, it holds that

$$
|f(g(\theta) + (\gamma + \delta)\hat{g}(\theta)) - f(g(\theta) + \gamma\hat{g}(\theta))| \le c|\hat{g}(\theta)||\delta|, (247)
$$

with $\delta > 0$, for some constant $c$ positive and finite. This
implies that

$$
\left| \frac{f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))}{\delta} \right| \le c|\hat{g}(\boldsymbol{\theta})|. (248)
$$

Using these arguments, the limit in (246) satisfies for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$ arbitrarily small, that

$$
\begin{align*}
& \lim_{\delta \to 0} \frac{1}{\delta} (\hat{r}(\gamma + \delta) - \hat{r}(\gamma)) \\
&= \lambda \lim_{\delta \to 0} \int \frac{f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))}{\delta} \, dQ(\boldsymbol{\theta}) \\
&= \lambda \int \hat{f}(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))\hat{g}(\boldsymbol{\theta}) \, dQ(\boldsymbol{\theta}) \tag{249} \\
&< \infty, \tag{250}
\end{align*}
$$

where the function $\hat{f}: (0, +\infty) \to \mathbb{R}$ is the derivative of $f$. That is, $\hat{f}(t) = 1 + \log(t)$. The equality in (249) and the

inequality in (250) follow from noticing that the conditions for
the dominated convergence theorem hold [59, Theorem 1.6.9],
namely:

* For all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$, the inequality in (248) holds;

* The function $\hat{g}$ in (248) satisfies the inequality in (238); and

* For all $\theta \in \text{supp } Q$ and for all $\gamma \in (-\epsilon, \epsilon)$, with $\epsilon > 0$ arbitrarily small, it holds that

$$
\begin{align}
& \lim_{\delta \to 0} \frac{f(g(\boldsymbol{\theta}) + (\gamma + \delta)\hat{g}(\boldsymbol{\theta})) - f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta}))}{\delta} \nonumber \\
&= \frac{d}{d\gamma} f(g(\boldsymbol{\theta}) + \gamma\hat{g}(\boldsymbol{\theta})) \tag{251}
\end{align}
$$

$$
= \hat{f}(g(\boldsymbol{\theta}) + \gamma \hat{g}(\boldsymbol{\theta})) \hat{g}(\boldsymbol{\theta}). \quad (252)
$$

Hence, the derivative of the real function $r$ in (243) is

$$
\begin{align}
\frac{d}{d\alpha} r(\alpha) &= \int \mathbf{L}_z(\boldsymbol{\nu}) \hat{g}(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) + \beta \int \hat{g}(\boldsymbol{\nu}) dQ(\boldsymbol{\nu}) \nonumber \\
&\quad + \lambda \int \hat{g}(\boldsymbol{\nu}) (1 + \log(g(\boldsymbol{\nu}) + \alpha \hat{g}(\boldsymbol{\nu}))) dQ(\boldsymbol{\nu}). \tag{253}
\end{align}
$$

From (241) and (253), it follows that

$$
\begin{align}
& \partial L(g, \beta; \hat{g}) \nonumber \\
& = \int \hat{g}(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \lambda(1 + \log(g(\boldsymbol{\nu}))) + \beta) dQ(\boldsymbol{\nu}). \tag{254}
\end{align}
$$

The relevance of the Gateaux differential in (254) stems from
[73, Theorem 1, page 178], which unveils the fact that a
necessary condition for the functional $L$ in (240) to have
a minimum at $\left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}, \beta \right) \in \mathcal{M} \times \mathbb{R}$ is that for all
functions $\hat{g} \in \mathcal{M}$,

$$
\partial L \left( \frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}, \beta; \hat{g} \right) = 0. \quad (255)
$$

From (255), it follows that $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ must satisfy for all functions $\hat{g}$ in $\mathcal{M}$ that

$$
0 = \int \hat{g}(\boldsymbol{\nu}) \left( \mathbf{L}_z(\boldsymbol{\nu}) + \lambda \left( 1 + \log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) \right) \right) + \beta \right) dQ(\boldsymbol{\nu}),
$$

which implies that for all $\boldsymbol{\nu} \in \text{supp } Q$,

$$
\mathbf{L}_z(\boldsymbol{\nu}) + \lambda \left( 1 + \log \left( \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) \right) \right) + \beta = 0, \quad (256)
$$

and thus,

$$
\frac{d P_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) = \exp\left(-\frac{\beta + \lambda}{\lambda}\right) \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\lambda}\right), \quad (257)
$$
with $\beta$ chosen to satisfy (237b). That is,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) = \frac{\exp\left(-\frac{L_z(\boldsymbol{\nu})}{\lambda}\right)}{\int \exp\left(-\frac{L_z(\boldsymbol{\theta})}{\lambda}\right) dQ(\boldsymbol{\theta})} \quad (258) $$

$$ = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\boldsymbol{\nu})\right). \quad (259) $$

The proof continues by verifying that the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ that satisfies (258) is the unique solution to the ERM-RER problem in (19). Such verification is done by showing that the objective function in (19) is strictly convex with the optimization variable. Let $P_1$ and $P_2$ be two different probability measures in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and let $\alpha$ be in $(0, 1)$. Hence,

$$ \begin{aligned} & R_z(\alpha P_1 + (1-\alpha)P_2) + \lambda D(\alpha P_1 + (1-\alpha)P_2 \| Q) \\ & = \alpha R_z(P_1) + (1-\alpha)R_z(P_2) + \lambda D(\alpha P_1 + (1-\alpha)P_2 \| Q) \\ & > \alpha(R_z(P_1) + \lambda D(P_1 \| Q)) + (1-\alpha)(R_z(P_2) + \lambda D(P_2 \| Q)) \end{aligned} $$

where the functional $R_z$ is defined in (18). The equality above follows from the properties of the Lebesgue integral, while the inequality follows from Theorem 2. This proves that the solution is unique due to the strict concavity of the objective function, which completes the proof.

## APPENDIX D
### PROOF OF LEMMA 2

From Theorem 3, it follows that for all $\theta \in \text{supp } Q$,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) \quad (260) $$

$$ \leq \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right)\right) \quad (261) $$

$$ < +\infty, \quad (262) $$

where the inequality in (261) follows from the fact that the function $L_z$ is nonnegative; and the equality in (262) follows from the fact that $\lambda \in \mathcal{K}_{Q,z}$. This completes the proof of finiteness.

The proof of positivity follows from observing that $\lambda \in \mathcal{K}_{Q,z}$ and thus, $K_{Q,z}(-\frac{1}{\lambda}) < +\infty$. Moreover, for all $\theta \in \text{supp } Q$, it holds that $L_z(\theta) \leq +\infty$, with equality if and only if $L_z(\theta) = +\infty$. These two observations put together yield

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) \quad (263) $$

$$ \geq 0, \quad (264) $$

with equality if and only if $L_z(\theta) = +\infty$. The proof continues by showing that $Q(\{\theta \in \mathcal{M} : L_z(\theta) = +\infty\}) = 0$. From the assumption that $\lambda \in \mathcal{K}_{Q,z}$, it follows that

$$ +\infty > \log\left(\int \exp\left(-\frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta})\right) \quad (265) $$

$$ \geq \log\left(\int \left(1 - \frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta})\right) \quad (266) $$

$$ = \log\left(Q(\mathcal{M}) - \frac{1}{\lambda} \int L_z(\boldsymbol{\theta}) dQ(\boldsymbol{\theta})\right), \quad (267) $$

where the inequality in (266) holds from the fact that for all $x \geq 0$, $\exp(x) > 1+x$. Hence, the inequality in (267) implies that

$$ Q(\mathcal{M}) < +\infty \quad \text{and} \quad (268) $$

$$ \int L_z(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) < +\infty. \quad (269) $$

Finally, from [59, Lemma 1.6.6] and the inequality in (269) it follows that $Q(\{\theta \in \mathcal{M} : L_z(\theta) = +\infty\}) = 0$. Hence,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) > 0, \quad (270) $$

almost surely with respect to the measure $Q$. And this completes the proof.

## APPENDIX E
### PROOF OF LEMMA 3

The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies for all $C \in \mathcal{B}(\mathcal{M})$,

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(C) = \int_C \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}), \quad (271) $$

and thus, if $Q(C) = 0$, then

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(C) = 0, \quad (272) $$

which implies the absolute continuity of $P_{\Theta|Z=z}^{(Q,\lambda)}$ with respect to $Q$.

Alternatively, given a set $C \in \mathcal{B}(\mathcal{M})$, assume now that $P_{\Theta|Z=z}^{(Q,\lambda)}(C) = 0$. Hence, it follows that

$$ 0 = P_{\Theta|Z=z}^{(Q,\lambda)}(C) \quad (273) $$

$$ = \int_C \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}). \quad (274) $$

From Lemma 2, it follows that $Q(\{\theta \in \mathcal{M} : \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) = 0\}) = 0$. Hence,

$$ \int_C \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) = 0, \quad (275) $$

if and only if $Q(C) = 0$. This verifies the absolute continuity of $Q$ with respect to $P_{\Theta|Z=z}^{(Q,\lambda)}$, and completes the proof.

## APPENDIX F
### PROOF OF LEMMA 4

Consider the function $g: \mathcal{M} \to [0, +\infty)$,

$$ g(\boldsymbol{\theta}) = \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) \left( \frac{dP_{\Theta|Z=z}^{(Q,\beta)}}{dQ}(\boldsymbol{\theta}) \right)^{-1}, \quad (276) $$

and note that for all $\boldsymbol{\theta} \in \text{supp } Q \setminus \{\boldsymbol{\nu} \in \mathcal{M} : L_z(\boldsymbol{\nu}) = +\infty\}$, $g(\boldsymbol{\theta}) > 0$. Alternatively, for all $\boldsymbol{\theta} \in \{\boldsymbol{\nu} \in \mathcal{M} : L_z(\boldsymbol{\nu}) = +\infty\}$, $g(\boldsymbol{\theta}) = 0$, which follows from the assumption $0 \cdot \frac{1}{0} = 0$.
with $\beta$ chosen to satisfy (237b). That is,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) = \frac{\exp\left(-\frac{L_z(\boldsymbol{\nu})}{\lambda}\right)}{\int \exp\left(-\frac{L_z(\boldsymbol{\theta})}{\lambda}\right) dQ(\boldsymbol{\theta})} \quad (258) $$

$$ = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\boldsymbol{\nu})\right) \quad (259) $$

The proof continues by verifying that the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ that satisfies (258) is the unique solution to the ERM-RER problem in (19). Such verification is done by showing that the objective function in (19) is strictly convex with the optimization variable. Let $P_1$ and $P_2$ be two different probability measures in $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ and let $\alpha$ be in $(0, 1)$. Hence,

$$ \begin{aligned} & R_z(\alpha P_1 + (1-\alpha)P_2) + \lambda D(\alpha P_1 + (1-\alpha)P_2 \| Q) \\ & = \alpha R_z(P_1) + (1-\alpha)R_z(P_2) + \lambda D(\alpha P_1 + (1-\alpha)P_2 \| Q) \\ & > \alpha(R_z(P_1) + \lambda D(P_1 \| Q)) + (1-\alpha)(R_z(P_2) + \lambda D(P_2 \| Q)) \end{aligned} $$

where the functional $R_z$ is defined in (18). The equality above follows from the properties of the Lebesgue integral, while the inequality follows from Theorem 2. This proves that the solution is unique due to the strict concavity of the objective function, which completes the proof.

## APPENDIX D
### PROOF OF LEMMA 2

From Theorem 3, it follows that for all $\theta \in \text{supp } Q$,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) \quad (260) $$

$$ \leq \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right)\right) \quad (261) $$

$$ < +\infty, \quad (262) $$

where the inequality in (261) follows from the fact that the function $L_z$ is nonnegative; and the equality in (262) follows from the fact that $\lambda \in \mathcal{K}_{Q,z}$. This completes the proof of finiteness.

The proof of positivity follows from observing that $\lambda \in \mathcal{K}_{Q,z}$ and thus, $K_{Q,z}(-\frac{1}{\lambda}) < +\infty$. Moreover, for all $\theta \in \text{supp } Q$, it holds that $L_z(\theta) \leq +\infty$, with equality if and only if $L_z(\theta) = +\infty$. These two observations put together yield

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) = \exp\left(-K_{Q,z}\left(-\frac{1}{\lambda}\right) - \frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) \quad (263) $$

$$ \geq 0, \quad (264) $$

with equality if and only if $L_z(\theta) = +\infty$. The proof continues by showing that $Q(\{\theta \in \mathcal{M} : L_z(\theta) = +\infty\}) = 0$. From the assumption that $\lambda \in \mathcal{K}_{Q,z}$, it follows that

$$ +\infty > \log\left(\int \exp\left(-\frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta})\right) \quad (265) $$

$$ \geq \log\left(\int \left(1 - \frac{1}{\lambda}L_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta})\right) \quad (266) $$

$$ = \log\left(Q(\mathcal{M}) - \frac{1}{\lambda} \int L_z(\boldsymbol{\theta}) dQ(\boldsymbol{\theta})\right), \quad (267) $$

where the inequality in (266) holds from the fact that for all $x \geq 0$, $\exp(x) > 1+x$. Hence, the inequality in (267) implies that

$$ Q(\mathcal{M}) < +\infty \quad \text{and} \quad (268) $$

$$ \int L_z(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) < +\infty. \quad (269) $$

Finally, from [59, Lemma 1.6.6] and the inequality in (269) it follows that $Q(\{\theta \in \mathcal{M} : L_z(\theta) = +\infty\}) = 0$. Hence,

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) > 0, \quad (270) $$

almost surely with respect to the measure $Q$. And this completes the proof.

## APPENDIX E
### PROOF OF LEMMA 3

The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies for all $\mathcal{C} \in \mathcal{B}(\mathcal{M})$,

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{C}) = \int_{\mathcal{C}} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}), \quad (271) $$

and thus, if $Q(\mathcal{C}) = 0$, then

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{C}) = 0, \quad (272) $$

which implies the absolute continuity of $P_{\Theta|Z=z}^{(Q,\lambda)}$ with respect to $Q$.

Alternatively, given a set $\mathcal{C} \in \mathcal{B}(\mathcal{M})$, assume now that $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{C}) = 0$. Hence, it follows that

$$ 0 = P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{C}) \quad (273) $$

$$ = \int_{\mathcal{C}} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}). \quad (274) $$

From Lemma 2, it follows that $Q(\{\theta \in \mathcal{M} : \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) = 0\}) = 0$. Hence,

$$ \int_{\mathcal{C}} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) dQ(\boldsymbol{\theta}) = 0, \quad (275) $$

if and only if $Q(\mathcal{C}) = 0$. This verifies the absolute continuity of $Q$ with respect to $P_{\Theta|Z=z}^{(Q,\lambda)}$, and completes the proof.

## APPENDIX F
### PROOF OF LEMMA 4

Consider the function $g: \mathcal{M} \to [0, +\infty)$,

$$ g(\boldsymbol{\theta}) = \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\theta}) \left( \frac{dP_{\Theta|Z=z}^{(Q,\beta)}}{dQ}(\boldsymbol{\theta}) \right)^{-1}, \quad (276) $$

and note that for all $\boldsymbol{\theta} \in \text{supp } Q \setminus \{\boldsymbol{\nu} \in \mathcal{M} : L_z(\boldsymbol{\nu}) = +\infty\}$, $g(\boldsymbol{\theta}) > 0$. Alternatively, for all $\boldsymbol{\theta} \in \{\boldsymbol{\nu} \in \mathcal{M} : L_z(\boldsymbol{\nu}) = +\infty\}$, $g(\boldsymbol{\theta}) = 0$, which follows from the assumption $0 \cdot \frac{1}{0} = 0$.
Consider a measure $P$ on $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, such that for all sets $A \in \mathcal{B}(\mathcal{M})$,

$$P(A) = \int_A g(\theta) dP_{\Theta|Z=z}^{(Q,\beta)}(\theta), \quad (277)$$

and note that if $P_{\Theta|Z=z}^{(Q,\beta)}(A) = 0$, then $P(A) = 0$. This implies that $P$ is absolutely continuous with respect to $P_{\Theta|Z=z}^{(Q,\beta)}(A)$. Moreover, from (277), it follows that

$$P(A) = \int_A \frac{dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta)}{dQ(\theta)} \left( \frac{dP_{\Theta|Z=z}^{(Q,\beta)}(\theta)}{dQ(\theta)} \right)^{-1} dP_{\Theta|Z=z}^{(Q,\beta)}(\theta) \quad (278)$$

$$= \int_A \frac{dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta)}{dQ(\theta)} \left( \frac{dP_{\Theta|Z=z}^{(Q,\beta)}(\theta)}{dQ(\theta)} \right)^{-1} \frac{dP_{\Theta|Z=z}^{(Q,\beta)}(\theta)}{dQ(\theta)} dQ(\theta) \quad (279)$$

$$= \int_A \frac{dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta)}{dQ(\theta)} dQ(\theta) \quad (280)$$

$$= \int_A dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta) \quad (281)$$

$$= P_{\Theta|Z=z}^{(Q,\alpha)}(A), \quad (282)$$

which implies that the probability measures $P$ in (277) and $P_{\Theta|Z=z}^{(Q,\alpha)}$ are identical. Thus, $P_{\Theta|Z=z}^{(Q,\alpha)}$ is absolutely continuous with respect to $P_{\Theta|Z=z}^{(Q,\beta)}$. The proof that $P_{\Theta|Z=z}^{(Q,\beta)}$ is absolutely continuous with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$ follows the same argument. This completes the proof.

## APPENDIX G
### PROOF OF LEMMA 6

From Theorem 3, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta)}{dQ(\theta)} = \frac{\exp\left(-\frac{L_z(\theta)}{\lambda}\right)}{\int \exp\left(-\frac{L_z(\nu)}{\lambda}\right) dQ(\nu)} \quad (283)$$

$$= \left( \exp\left(\frac{L_z(\theta)}{\lambda}\right) \int \exp\left(-\frac{L_z(\nu)}{\lambda}\right) dQ(\nu) \right)^{-1} \quad (284)$$

$$= \left( \int \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (285)$$

Given $\theta \in \text{supp } Q$, consider the partition of $\text{supp } Q$ formed by the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$, which satisfy the following:

$$A_0(\theta) \triangleq \{\nu \in \text{supp } Q : L_z(\theta) - L_z(\nu) = 0\}, \quad (286a)$$

$$A_1(\theta) \triangleq \{\nu \in \text{supp } Q : L_z(\theta) - L_z(\nu) < 0\}, \quad \text{and} \quad (286b)$$

$$A_2(\theta) \triangleq \{\nu \in \text{supp } Q : L_z(\theta) - L_z(\nu) > 0\}. \quad (286c)$$

Using the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$ in (285), the following holds for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ(\theta)} = \left( \int_{A_0(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) + \int_{A_2(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (287)$$

$$= \left( Q(A_0(\theta)) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) + \int_{A_2(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (288)$$

Note that the sets

$$\{\nu \in \text{supp } Q : L_z(\nu) = \delta_{Q,z}^*\}, \quad (289)$$

$$\{\nu \in \text{supp } Q : L_z(\nu) > \delta_{Q,z}^*\}, \quad \text{and} \quad (290)$$

$$\{\nu \in \text{supp } Q : L_z(\nu) < \delta_{Q,z}^*\}, \quad (291)$$

with $\delta_{Q,z}^*$ in (37), form a partition of the set $\text{supp } Q$. Following this observation, the rest of the proof is divided into three parts. The first part evaluates $\lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ(\theta)}$, with $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\}$. The second part considers the case in which $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) > \delta_{Q,z}^*\}$. The third part considers the remaining case.

The first part is as follows. Consider that $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\}$ and note that $\{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\} = \mathcal{L}_{Q,z}^*$. Hence, the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$ in (286) satisfy the following:

$$A_0(\theta) = \mathcal{L}_{Q,z}^*, \quad (292a)$$

$$A_1(\theta) = \{\mu \in \text{supp } Q : L_z(\mu) > \delta_{Q,z}^*\}, \quad \text{and} \quad (292b)$$

$$A_2(\theta) = \{\mu \in \text{supp } Q : L_z(\mu) < \delta_{Q,z}^*\}. \quad (292c)$$

From the definition of $\delta_{Q,z}^*$ in (37), it follows that $Q(A_2(\theta)) = 0$. Plugging the equalities in (292) in (288) yields for all $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\}$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ(\theta)} = \left( Q(\mathcal{L}_{Q,z}^*) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (293)$$
Consider a measure $P$ on $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, such that for all sets $A \in \mathcal{B}(\mathcal{M})$,

$$P(A) = \int_A g(\theta) dP_{\Theta|Z=z}^{(Q,\beta)}(\theta), \quad (277)$$

and note that if $P_{\Theta|Z=z}^{(Q,\beta)}(A) = 0$, then $P(A) = 0$. This implies that $P$ is absolutely continuous with respect to $P_{\Theta|Z=z}^{(Q,\beta)}(A)$. Moreover, from (277), it follows that

$$P(A) = \int_A \frac{dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta)}{dQ} \left( \frac{dP_{\Theta|Z=z}^{(Q,\beta)}(\theta)}{dQ} \right)^{-1} dP_{\Theta|Z=z}^{(Q,\beta)}(\theta) \quad (278)$$

$$= \int_A \frac{dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta)}{dQ} \left( \frac{dP_{\Theta|Z=z}^{(Q,\beta)}(\theta)}{dQ} \right)^{-1} \frac{dP_{\Theta|Z=z}^{(Q,\beta)}(\theta)}{dQ} dQ(\theta) \quad (279)$$

$$= \int_A \frac{dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta)}{dQ} dQ(\theta) \quad (280)$$

$$= \int_A dP_{\Theta|Z=z}^{(Q,\alpha)}(\theta) \quad (281)$$

$$= P_{\Theta|Z=z}^{(Q,\alpha)}(A), \quad (282)$$

which implies that the probability measures $P$ in (277) and $P_{\Theta|Z=z}^{(Q,\alpha)}$ are identical. Thus, $P_{\Theta|Z=z}^{(Q,\alpha)}$ is absolutely continuous with respect to $P_{\Theta|Z=z}^{(Q,\beta)}$. The proof that $P_{\Theta|Z=z}^{(Q,\beta)}$ is absolutely continuous with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$ follows the same argument. This completes the proof.

# APPENDIX G
## PROOF OF LEMMA 6

From Theorem 3, the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25) satisfies for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta)}{dQ} = \frac{\exp\left(-\frac{L_z(\theta)}{\lambda}\right)}{\int \exp\left(-\frac{L_z(\nu)}{\lambda}\right) dQ(\nu)} \quad (283)$$

$$= \left( \exp\left(\frac{L_z(\theta)}{\lambda}\right) \int \exp\left(-\frac{L_z(\nu)}{\lambda}\right) dQ(\nu) \right)^{-1} \quad (284)$$

$$= \left( \int \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (285)$$

Given $\theta \in \text{supp } Q$, consider the partition of supp $Q$ formed by the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$, which satisfy the following:

$$A_0(\theta) \triangleq \{\nu \in \text{supp } Q : L_z(\theta) - L_z(\nu) = 0\}, \quad (286a)$$

$$A_1(\theta) \triangleq \{\nu \in \text{supp } Q : L_z(\theta) - L_z(\nu) < 0\}, \quad \text{and} \quad (286b)$$

$$A_2(\theta) \triangleq \{\nu \in \text{supp } Q : L_z(\theta) - L_z(\nu) > 0\}. \quad (286c)$$

Using the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$ in (285), the following holds for all $\theta \in \text{supp } Q$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ} = \left( \int_{A_0(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) + \int_{A_2(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (287)$$

$$= \left( Q(A_0(\theta)) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) + \int_{A_2(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (288)$$

Note that the sets

$$\{\nu \in \text{supp } Q : L_z(\nu) = \delta_{Q,z}^*\}, \quad (289)$$

$$\{\nu \in \text{supp } Q : L_z(\nu) > \delta_{Q,z}^*\}, \quad \text{and} \quad (290)$$

$$\{\nu \in \text{supp } Q : L_z(\nu) < \delta_{Q,z}^*\}, \quad (291)$$

with $\delta_{Q,z}^*$ in (37), form a partition of the set supp $Q$. Following this observation, the rest of the proof is divided into three parts. The first part evaluates $\lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)$, with $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\}$. The second part considers the case in which $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) > \delta_{Q,z}^*\}$. The third part considers the remaining case.

The first part is as follows. Consider that $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\}$ and note that $\{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\} = \mathcal{L}_{Q,z}^*$. Hence, the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$ in (286) satisfy the following:

$$A_0(\theta) = \mathcal{L}_{Q,z}^*, \quad (292a)$$

$$A_1(\theta) = \{\mu \in \text{supp } Q : L_z(\mu) > \delta_{Q,z}^*\}, \quad \text{and} \quad (292b)$$

$$A_2(\theta) = \{\mu \in \text{supp } Q : L_z(\mu) < \delta_{Q,z}^*\}. \quad (292c)$$

From the definition of $\delta_{Q,z}^*$ in (37), it follows that $Q(A_2(\theta)) = 0$. Plugging the equalities in (292) in (288) yields for all $\theta \in \{\nu \in \mathcal{M} : L_z(\nu) = \delta_{Q,z}^*\}$,

$$\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ} = \left( Q(\mathcal{L}_{Q,z}^*) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(L_z(\theta) - L_z(\nu))\right) dQ(\nu) \right)^{-1} \quad (293)$$
The equality in (293) implies that for all $\theta \in \{\nu \in \mathcal{M} : \mathcal{L}_z(\nu) = \delta_{Q,z}^*\}$,

$$ 
\begin{align} 
& \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \nonumber \\
&= \left( \lim_{\lambda \to 0^+} \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) + \right. \nonumber \\
& \qquad \left. + Q(\mathcal{L}_{Q,z}^*) \right)^{-1} \tag{294} \\
&= \left\{ \begin{array}{ll} +\infty & \text{if } Q(\mathcal{L}_{Q,z}^*) = 0 \\ \frac{1}{Q(\mathcal{L}_{Q,z}^*)} & \text{otherwise.} \end{array} \right. \tag{295}
\end{align} 
$$

where the equality in (295) follows from verifying that the dominated convergence theorem [59, Theorem 2.6.9] holds. That is,

(a) For all $\nu \in A_1(\theta)$, it holds that $\exp(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))) < 1$; and

(b) For all $\nu \in A_1(\theta)$, it holds that

$$ \lim_{\lambda \to 0^+} \exp \left( \frac{1}{\lambda} (\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu)) \right) = 0. \quad (296) $$

This completes the first part of the proof.

The second part is as follows. For all $\delta > \delta_{Q,z}^*$ and for all $\theta \in \{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) = \delta\}$, the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$ in (286) satisfy the following:

$$ A_0(\theta) = \{\mu \in \text{supp } Q : \mathcal{L}_z(\mu) = \delta\}, \quad (297a) $$

$$ A_1(\theta) = \{\mu \in \text{supp } Q : \mathcal{L}_z(\mu) > \delta\}, \quad \text{and} \quad (297b) $$

$$ A_2(\theta) = \{\mu \in \text{supp } Q : \mathcal{L}_z(\mu) < \delta\}. \quad (297c) $$

Consider the sets

$$ A_{2,1}(\theta) \triangleq \{\mu \in A_2(\theta) : \mathcal{L}_z(\mu) < \delta_{Q,z}^*\}, \quad \text{and} \quad (298) $$

$$ A_{2,2}(\theta) \triangleq \{\mu \in A_2(\theta) : \delta_{Q,z}^* \le \mathcal{L}_z(\mu) < \delta\}, \quad (299) $$

and note that $A_{2,1}(\theta)$ and $A_{2,2}(\theta)$ form a partition of $A_2(\theta)$. Moreover, from the definition of $\delta_{Q,z}^*$ in (37), it holds that

$$ Q(A_{2,1}(\theta)) = 0. \quad (300) $$

Hence, plugging the equalities in (297) and (300) in (288) yields, for all $\delta > \delta_{Q,z}^*$ and for all $\theta \in \{\nu \in \mathcal{M} : \mathcal{L}_z(\nu) = \delta\}$,

$$ 
\begin{align} 
& \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \nonumber \\
&= \left( Q(A_0(\theta)) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right. \nonumber \\
& \qquad \left. + \int_{A_{2,2}(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right)^{-1}. \tag{301}
\end{align} 
$$

The equality in (301) implies that for all $\delta > \delta_{Q,z}^*$ and for all $\theta \in \{\nu \in \mathcal{M} : \mathcal{L}_z(\nu) = \delta\}$,

$$ 
\begin{align} 
& \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \nonumber \\
&= \left( \lim_{\lambda \to 0^+} \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right. \nonumber \\
& \qquad \left. + \lim_{\lambda \to 0^+} \int_{A_{2,2}(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right) \nonumber \\
& \qquad + Q(A_0(\theta)) \right)^{-1} \tag{302}
\end{align} 
$$

$$ 
\begin{align} 
&= \left( \lim_{\lambda \to 0^+} \int_{A_{2,2}(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right. \nonumber \\
& \qquad \left. + Q(A_0(\theta)) \right)^{-1} \tag{303} \\
&= (Q(A_0(\theta)) + \infty)^{-1} \tag{304}
\end{align} 
$$

$$ = (Q(A_0(\theta)) + \infty)^{-1} \quad (304) $$

$$ = 0, \quad (305) $$

where the equality in (303) follows by verifying that the dominated convergence theorem [59, Theorem 2.6.9] holds. That is,

(a) For all $\nu \in A_1(\theta)$, it holds that $\exp(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))) < 1$; and

(b) For all $\nu \in A_1(\theta)$, it holds that

$$ \lim_{\lambda \to 0^+} \exp \left( \frac{1}{\lambda} (\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu)) \right) = 0. \quad (306) $$

This completes the second part.

The third part of the proof follows by noticing that the set $\{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) < \delta_{Q,z}^*\}$ is a negligible set with respect to $Q$ and thus, for all $\theta \in \{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) < \delta_{Q,z}^*\}$, the value $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)$ is immaterial. Hence, it is arbitrarily assumed that for all $\theta \in \{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) < \delta_{Q,z}^*\}$, it holds that

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = 0. \quad (307) $$

This completes the third part and completes the proof.

## APPENDIX H
### PROOF OF LEMMA 7

Consider the following partition of the set $\mathcal{M}$ formed by the sets

$$ A_0 \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) = \delta_{Q,z}^*\}, \quad (308a) $$

$$ A_1 \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) < \delta_{Q,z}^*\}, \quad \text{and} \quad (308b) $$

$$ A_2 \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) > \delta_{Q,z}^*\}, \quad (308c) $$
The equality in (293) implies that for all $\theta \in \{\nu \in \mathcal{M} : \mathcal{L}_z(\nu) = \delta_{Q,z}^*\}$,

$$ 
\begin{align} 
& \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \nonumber \\
& = \left( \lim_{\lambda \to 0^+} \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) + \right. \tag{294} \\
& \qquad \left. + Q(\mathcal{L}_{Q,z}^*) \right)^{-1} \nonumber \\
& = \left\{ \begin{array}{ll} +\infty & \text{if } Q(\mathcal{L}_{Q,z}^*) = 0 \\ \frac{1}{Q(\mathcal{L}_{Q,z}^*)} & \text{otherwise.} \end{array} \right. \tag{295}
\end{align} 
$$

where the equality in (295) follows from verifying that the dominated convergence theorem [59, Theorem 2.6.9] holds. That is,

(a) For all $\nu \in A_1(\theta)$, it holds that $\exp(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))) < 1$; and

(b) For all $\nu \in A_1(\theta)$, it holds that

$$ \lim_{\lambda \to 0^+} \exp \left( \frac{1}{\lambda} (\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu)) \right) = 0. \quad (296) $$

This completes the first part of the proof.

The second part is as follows. For all $\delta > \delta_{Q,z}^*$ and for all $\theta \in \{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) = \delta\}$, the sets $A_0(\theta)$, $A_1(\theta)$, and $A_2(\theta)$ in (286) satisfy the following:

$$ A_0(\theta) = \{\mu \in \text{supp } Q : \mathcal{L}_z(\mu) = \delta\}, \quad (297a) $$

$$ A_1(\theta) = \{\mu \in \text{supp } Q : \mathcal{L}_z(\mu) > \delta\}, \quad \text{and} \quad (297b) $$

$$ A_2(\theta) = \{\mu \in \text{supp } Q : \mathcal{L}_z(\mu) < \delta\}. \quad (297c) $$

Consider the sets

$$ A_{2,1}(\theta) \triangleq \{\mu \in A_2(\theta) : \mathcal{L}_z(\mu) < \delta_{Q,z}^*\}, \quad \text{and} \quad (298) $$

$$ A_{2,2}(\theta) \triangleq \{\mu \in A_2(\theta) : \delta_{Q,z}^* \le \mathcal{L}_z(\mu) < \delta\}, \quad (299) $$

and note that $A_{2,1}(\theta)$ and $A_{2,2}(\theta)$ form a partition of $A_2(\theta)$. Moreover, from the definition of $\delta_{Q,z}^*$ in (37), it holds that

$$ Q(A_{2,1}(\theta)) = 0. \quad (300) $$

Hence, plugging the equalities in (297) and (300) in (288) yields, for all $\delta > \delta_{Q,z}^*$ and for all $\theta \in \{\nu \in \mathcal{M} : \mathcal{L}_z(\nu) = \delta\}$,

$$ 
\begin{align} 
& \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \nonumber \\
& = \left( Q(A_0(\theta)) + \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right. \nonumber \\
& \qquad \left. + \int_{A_{2,2}(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right)^{-1}. \tag{301}
\end{align} 
$$

The equality in (301) implies that for all $\delta > \delta_{Q,z}^*$ and for all $\theta \in \{\nu \in \mathcal{M} : \mathcal{L}_z(\nu) = \delta\}$,

$$ 
\begin{align} 
& \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) \nonumber \\
& = \left( \lim_{\lambda \to 0^+} \int_{A_1(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right. \nonumber \\
& \qquad \left. + \lim_{\lambda \to 0^+} \int_{A_{2,2}(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right) \nonumber \\
& \qquad + Q(A_0(\theta)) \right)^{-1} \tag{302}
\end{align} 
$$

$$ = \left( \lim_{\lambda \to 0^+} \int_{A_{2,2}(\theta)} \exp\left(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))\right) dQ(\nu) \right. \tag{303} \\
\left. + Q(A_0(\theta)) \right)^{-1} $$

$$ = (Q(A_0(\theta)) + \infty)^{-1} \quad (304) $$

$$ = 0, \quad (305) $$

where the equality in (303) follows by verifying that the dominated convergence theorem [59, Theorem 2.6.9] holds. That is,

(a) For all $\nu \in A_1(\theta)$, it holds that $\exp(\frac{1}{\lambda}(\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu))) < 1$; and

(b) For all $\nu \in A_1(\theta)$, it holds that

$$ \lim_{\lambda \to 0^+} \exp \left( \frac{1}{\lambda} (\mathcal{L}_z(\theta) - \mathcal{L}_z(\nu)) \right) = 0. \quad (306) $$

This completes the second part.

The third part of the proof follows by noticing that the set $\{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) < \delta_{Q,z}^*\}$ is a negligible set with respect to $Q$ and thus, for all $\theta \in \{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) < \delta_{Q,z}^*\}$, the value $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta)$ is immaterial. Hence, it is arbitrarily assumed that for all $\theta \in \{\nu \in \text{supp } Q : \mathcal{L}_z(\nu) < \delta_{Q,z}^*\}$, it holds that

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = 0. \quad (307) $$

This completes the third part and completes the proof.

## APPENDIX H
### PROOF OF LEMMA 7

Consider the following partition of the set $\mathcal{M}$ formed by the sets

$$ A_0 \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) = \delta_{Q,z}^*\}, \quad (308a) $$

$$ A_1 \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) < \delta_{Q,z}^*\}, \quad \text{and} \quad (308b) $$

$$ A_2 \triangleq \{\theta \in \mathcal{M} : \mathcal{L}_z(\theta) > \delta_{Q,z}^*\}, \quad (308c) $$
with $\delta_{Q,z}^*$ in (37) and the function $L_z$ in (3). Note that $A_0 = \mathcal{L}_{Q,z}^*$, with $\mathcal{L}_{Q,z}^*$ in (38) and

$$ 
\begin{aligned}
1 &= P_{\Theta|Z=z}^{(Q,\lambda)} (A_0) + P_{\Theta|Z=z}^{(Q,\lambda)} (A_1) + P_{\Theta|Z=z}^{(Q,\lambda)} (A_2) && (309) \\
&= P_{\Theta|Z=z}^{(Q,\lambda)} (A_0) + P_{\Theta|Z=z}^{(Q,\lambda)} (A_2) && (310)
\end{aligned}
$$

$$ = P_{\Theta|Z=z}^{(Q,\lambda)} (A_0) + \int_{A_2} dP_{\Theta|Z=z}^{(Q,\lambda)} (\theta), \quad (311) $$

where, the equality in (310) follows from noticing that $P_{\Theta|Z=z}^{(Q,\lambda)}(A_1) = 0$, which follows from the definition of $\delta_{Q,z}^*$ in (37) and the fact that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is absolutely continuous with respect to the measure $Q$.

The above implies that

$$ 1 = \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A_0) + \lim_{\lambda \to 0^+} \int_{A_2} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta)}{dQ}(\theta) dQ(\theta) \quad (312) $$

$$ = \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A_0) + \int_{A_2} \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta)}{dQ}(\theta) dQ(\theta) \quad (313) $$

$$ = \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A_0), \quad (314) $$

where, the equality in (313) follows from the dominated convergence theorem [59, Theorem 1.6.9], given that the Randon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ is positive and finite (Lemma 2); and the inequality in (314) holds from the fact that for all $\theta \in A_2$, it holds that $\lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = 0$ (Lemma 6). Hence, it finally holds that

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) = 1, \quad (315) $$

which completes the proof.

# APPENDIX I
## PROOF OF LEMMA 8

The proof is presented in two parts. The first part shows that if for all $\delta \in (\rho^*, +\infty)$, the inequality in (49) holds, then, $Q$ is coherent. The second part shows that if $Q$ is not coherent, then there exists a $\delta \in (\rho^*, +\infty)$ such that

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) = 0. \quad (316) $$

The first part is as follows. Note that for all $\delta \in (\rho^*, +\infty)$ and for all $\theta \in \mathcal{L}_z(\delta) \cap \text{supp} Q$, it holds from Lemma 2 that

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) > 0. \quad (317) $$

Hence, if for all $\delta \in (\rho^*, +\infty)$, the inequality in (49) holds, then

$$ 0 < P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) \quad (318) $$

$$ = \int_{\mathcal{L}_z(\delta)} dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (319) $$

$$ = \int_{\mathcal{L}_z(\delta)} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) dQ(\theta), \quad (320) $$

which, together with (317), implies that for all $\delta \in (\rho^*, +\infty)$, $Q(\mathcal{L}_z(\delta)) > 0$. Hence, $Q$ is coherent.

The second part is as follows. Assume that $Q$ is not coherent. Then, there exists a $\delta \in (\rho^*, +\infty)$ such that $Q(\mathcal{L}_z(\delta)) = 0$. Hence, from the fact that $P_{\Theta|Z=z}^{(Q,\lambda)}$ is absolutely continuous with respect to $Q$, it follows that $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) = 0$. This completes the proof.

# APPENDIX J
## PROOF OF THEOREM 4

The optimization problem in (82) can be re-written in terms of the Radon-Nikodym derivative of the optimization measure $P$ with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$, denoted by $\frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}: M \to [0, +\infty)$, which yields:

$$ \min_{P \in \Delta_Q(M, \mathcal{B}(M))} \int_{L_z(\nu)} \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu), \quad (321a) $$

subject to:

$$ \int \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\nu) \log \left( \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\nu) \right) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \le c, \quad \text{and} \quad (321b) $$

$$ \int \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) = 1. \quad (321c) $$

The remainder of the proof focuses on the problem in which the optimization is over the function $\frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}$ instead of the measure $P$. This is due to the fact that for all $P \in \Delta_Q(M)$, the Radon-Nikodym derivative $\frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}$ is unique up to sets of zero measure with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$. Let $M$ be the set of measurable functions $M \to \mathbb{R}$ with respect to the measurable spaces $(M, \mathcal{B}(M))$ and $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$ that are absolutely integrable with respect to $P_{\Theta|Z=z}^{(Q,\lambda)}$. That is, for all $\hat{\theta} \in M$, it holds that

$$ \int |\hat{\theta}(\theta)| dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) < \infty. \quad (322) $$

Hence, the optimization problem of interest is:

$$ \min_{g \in \mathcal{M}} \int_{L_z(\nu)} g(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \quad (323a) $$

$$ \text{s.t.:} \quad \int g(\nu) \log(g(\nu)) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \le c, \quad \text{and} \quad (323b) $$

$$ \int g(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) = 1. \quad (323c) $$

The Lagrangian of the optimization problem in (323) is a functional $L: \mathcal{M} \times [0, +\infty)^2 \to \mathbb{R}$ of the form

$$ L(g, \alpha, \beta) = \int_{L_z(\nu)} g(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) + \alpha \left( \int g(\nu) \log(g(\nu)) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) - c \right) + \beta \left( \int g(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) - 1 \right), \quad (324) $$

where the reals $\alpha$ and $\beta$ are both nonnegative and act as Lagrangian multipliers due to the constraints (323b) and (323c), respectively.
with $\delta_{Q,z}^*$ in (37) and the function $L_z$ in (3). Note that $A_0 = \mathcal{L}_{Q,z}^*$, with $\mathcal{L}_{Q,z}^*$ in (38) and

$$ 
\begin{aligned}
1 &= P_{\Theta|Z=z}^{(Q,\lambda)} (A_0) + P_{\Theta|Z=z}^{(Q,\lambda)} (A_1) + P_{\Theta|Z=z}^{(Q,\lambda)} (A_2) && (309) \\
&= P_{\Theta|Z=z}^{(Q,\lambda)} (A_0) + P_{\Theta|Z=z}^{(Q,\lambda)} (A_2) && (310)
\end{aligned}
$$

$$ = P_{\Theta|Z=z}^{(Q,\lambda)} (A_0) + \int_{A_2} dP_{\Theta|Z=z}^{(Q,\lambda)} (\theta), \quad (311) $$

where, the equality in (310) follows from noticing that $P_{\Theta|Z=z}^{(Q,\lambda)}(A_1) = 0$, which follows from the definition of $\delta_{Q,z}^*$ in (37) and the fact that the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is absolutely continuous with respect to the measure $Q$.

The above implies that

$$ 1 = \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A_0) + \lim_{\lambda \to 0^+} \int_{A_2} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta)}{dQ}(\theta) dQ(\theta) \quad (312) $$

$$ = \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A_0) + \int_{A_2} \lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta)}{dQ}(\theta) dQ(\theta) \quad (313) $$

$$ = \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(A_0), \quad (314) $$

where, the equality in (313) follows from the dominated convergence theorem [59, Theorem 1.6.9], given that the Randon-Nikodym derivative $\frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}$ is positive and finite (Lemma 2); and the inequality in (314) holds from the fact that for all $\theta \in A_2$, it holds that $\lim_{\lambda \to 0^+} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) = 0$ (Lemma 6). Hence, it finally holds that

$$ \lim_{\lambda \to 0^+} P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_{Q,z}^*) = 1, \quad (315) $$

which completes the proof.

# APPENDIX I
## PROOF OF LEMMA 8

The proof is presented in two parts. The first part shows that if for all $\delta \in (\rho^*, +\infty)$, the inequality in (49) holds, then, $Q$ is coherent. The second part shows that if $Q$ is not coherent, then there exists a $\delta \in (\rho^*, +\infty)$ such that

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) = 0. \quad (316) $$

The first part is as follows. Note that for all $\delta \in (\rho^*, +\infty)$ and for all $\theta \in \mathcal{L}_z(\delta) \cap \text{supp} Q$, it holds from Lemma 2 that

$$ \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) > 0. \quad (317) $$

Hence, if for all $\delta \in (\rho^*, +\infty)$, the inequality in (49) holds, then

$$ 0 < P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) \quad (318) $$

$$ = \int_{\mathcal{L}_z(\delta)} dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) \quad (319) $$

$$ = \int_{\mathcal{L}_z(\delta)} \frac{dP_{\Theta|Z=z}^{(Q,\lambda)}}{dQ}(\theta) dQ(\theta), \quad (320) $$

which, together with (317), implies that for all $\delta \in (\rho^*, +\infty)$, $Q(\mathcal{L}_z(\delta)) > 0$. Hence, $Q$ is coherent.

The second part is as follows. Assume that $Q$ is not coherent. Then, there exists a $\delta \in (\rho^*, +\infty)$ such that $Q(\mathcal{L}_z(\delta)) = 0$. Hence, from the fact that $P_{\Theta|Z=z}^{(Q,\lambda)}$ is absolutely continuous with respect to $Q$, it follows that $P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{L}_z(\delta)) = 0$. This completes the proof.

# APPENDIX J
## PROOF OF THEOREM 4

The optimization problem in (82) can be re-written in terms of the Radon-Nikodym derivative of the optimization measure $P$ with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$, denoted by $\frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}: M \to [0, +\infty)$, which yields:

$$ \min_{P \in \Delta_Q(M, \mathcal{B}(M))} \int_{L_z(\nu)} \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu), \quad (321a) $$

subject to:

$$ \int \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\nu) \log \left( \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\nu) \right) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \le c, \quad \text{and} \quad (321b) $$

$$ \int \frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) = 1. \quad (321c) $$

The remainder of the proof focuses on the problem in which the optimization is over the function $\frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}$ instead of the measure $P$. This is due to the fact that for all $P \in \Delta_Q(M)$, the Radon-Nikodym derivative $\frac{dP}{dP_{\Theta|Z=z}^{(Q,\lambda)}}$ is unique up to sets of zero measure with respect to the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$. Let $M$ be the set of measurable functions $M \to \mathbb{R}$ with respect to the measurable spaces $(M, \mathcal{B}(M))$ and $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$ that are absolutely integrable with respect to $P_{\Theta|Z=z}^{(Q,\lambda)}$. That is, for all $\hat{\theta} \in M$, it holds that

$$ \int |\hat{\theta}(\theta)| dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) < \infty. \quad (322) $$

Hence, the optimization problem of interest is:

$$ \min_{g \in \mathcal{M}} \int_{L_z(\nu)} g(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \quad (323a) $$

$$ \text{s.t.:} \quad \int g(\nu) \log(g(\nu)) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \le c, \quad \text{and} \quad (323b) $$

$$ \int g(\theta) dP_{\Theta|Z=z}^{(Q,\lambda)}(\theta) = 1. \quad (323c) $$

The Lagrangian of the optimization problem in (323) is a functional $L: \mathcal{M} \times [0, +\infty)^2 \to \mathbb{R}$ of the form

$$ \begin{aligned} L(g, \alpha, \beta) = & \int_{L_z(\nu)} g(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) \\ & + \alpha \left( \int g(\nu) \log(g(\nu)) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) - c \right) \\ & + \beta \left( \int g(\nu) dP_{\Theta|Z=z}^{(Q,\lambda)}(\nu) - 1 \right), \end{aligned} \quad (324) $$

where the reals $\alpha$ and $\beta$ are both nonnegative and act as Lagrangian multipliers due to the constraints (323b) and (323c), respectively.
Let $h: \mathcal{M} \to \mathbb{R}$ be a function in $\mathcal{M}$. The Gateaux differential of the functional $L$ in (324) at $(g, \alpha, \beta) \in \mathcal{M} \times [0, +\infty)^2$ in the direction of $h$, if it exists, is

$$ \partial L(g, \alpha, \beta; h) \triangleq \left. \frac{d}{d\gamma} r(\gamma) \right|_{\gamma=0}, \quad (325) $$

where the real function $r: \mathbb{R} \to \mathbb{R}$ is such that for all $\gamma \in (-\epsilon, \epsilon)$, with some $\epsilon > 0$, satisfies

$$ \begin{aligned} r(\gamma) &= \int \mathbf{L}_z(\boldsymbol{\nu}) (g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) \\ &\quad + \alpha \left( \int (g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) - c \right) \\ &\quad + \beta \left( \int (g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) - 1 \right). \end{aligned} \quad (326) $$

The proof continues under the assumption that the functions $g$ and $h$ are such that the Gateaux differential in (325) exists. That is, the function $r$ in (326) is differentiable in $(-\epsilon, \epsilon)$, with some $\epsilon > 0$. Using the same arguments as in the proof of Theorem 3, it follows that the derivative of the real function $r$ in (326) is

$$ \begin{aligned} \frac{d}{d\gamma} r(\gamma) &= \int \mathbf{L}_z h(\boldsymbol{\nu}) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) + \alpha \int h(\boldsymbol{\nu}) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) \\ &\quad + \alpha \int h(\boldsymbol{\nu}) \log(g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) \\ &\quad + \beta \int h(\boldsymbol{\nu}) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}). \end{aligned} \quad (327) $$

From (325) and (327), it follows that

$$ \begin{aligned} \partial L(g, \alpha, \beta; h) &= \int h(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \alpha (1 + \log g(\boldsymbol{\nu}))) + \beta) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}). \end{aligned} \quad (328) $$

From [73, Theorem 1, page 217], it holds that a necessary condition for the functional $L$ in (324) to have a minimum at $(g, \alpha, \beta) \in \mathcal{M} \times [0, +\infty)^2$ is that for all functions $h \in \mathcal{M}$,

$$ \partial L(g, \alpha, \beta; h) = 0, \quad (329) $$

which implies that for all $\boldsymbol{\nu} \in \mathcal{M}$,

$$ \mathbf{L}_z(\boldsymbol{\nu}) + \alpha (1 + \log g(\boldsymbol{\nu})) + \beta = 0. \quad (330) $$

Thus,

$$ g(\boldsymbol{\nu}) = \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right) \exp\left(-\frac{\beta + \alpha}{\alpha}\right), \quad (331) $$

where $\alpha$ and $\beta$ are chosen to satisfy their corresponding constraints with equality. Denote by $P^*$ the solution of the optimization problem in (82). Hence, from (331), it follows that

$$ \frac{dP^*}{dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu})} = \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\alpha}\right) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\theta})}, \quad (332) $$

where $\alpha$ is chosen to satisfy

$$ D(P^* \| P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}) = c. \quad (333) $$

From Lemma 3, it follows that the probability measure $P^*$ and the $\sigma$-finite measure $Q$ satisfy,

$$ \frac{dP^*}{dQ}(\boldsymbol{\nu}) = \frac{dP^*}{dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}}(\boldsymbol{\nu}) \frac{dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) \quad (334) $$

$$ = \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\alpha}\right) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\theta})} \right) \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\lambda}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\lambda}\right) dQ(\boldsymbol{\theta})} \right) \quad (335) $$

$$ = \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right)}{\int \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\alpha}\right) \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\lambda}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\alpha})}{\lambda}\right) dQ(\boldsymbol{\alpha})} dQ(\boldsymbol{\theta})} \right) \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\lambda}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\lambda}\right) dQ(\boldsymbol{\theta})} \right) \quad (336) $$

$$ = \frac{\exp\left(-\left(\frac{1}{\alpha} + \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\nu})\right)}{\int \exp\left(-\left(\frac{1}{\alpha} + \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\nu})\right) dQ(\boldsymbol{\theta})}, \quad (337) $$

which implies that $P^*$ is a Gibbs probability measure on $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, with energy function $\mathbf{L}_z$, reference measure $Q$, and regularization parameter $\frac{1}{\alpha+\lambda}$, where $\alpha$ is chosen to satisfy (333). Let the positive real $\omega$ be $\omega \triangleq \frac{\alpha\lambda}{\alpha+\lambda}$ and note that $\omega \in (0, \lambda]$ and satisfies $D(P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\omega)}(\boldsymbol{\nu}) \| P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}) = c$. The proof ends by verifying that the objective function in (324) is strictly convex, and thus, the measure $P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\omega)}$ is the unique minimizer. This completes the proof.

## APPENDIX K
### PROOF OF LEMMA 15

Note that for all $(\lambda_1, \lambda_2) \in \{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}^2$, such that $\lambda_1 > \lambda_2$, it follows that for all $\boldsymbol{\theta} \in \text{supp } Q$, the inequality $\exp(\lambda_2 \mathbf{L}_z(\boldsymbol{\theta})) \le \exp(\lambda_1 \mathbf{L}_z(\boldsymbol{\theta}))$ holds. This implies that $K_{Q,z}(\lambda_2) \le K_{Q,z}(\lambda_1) < +\infty$, which proves that the function is nondecreasing.

The proof of continuity of the function $K_{Q,z}$ follows from observing that for all $\alpha \in \{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$, it holds
Let $h: \mathcal{M} \to \mathbb{R}$ be a function in $\mathcal{M}$. The Gateaux differential of the functional $L$ in (324) at $(g, \alpha, \beta) \in \mathcal{M} \times [0, +\infty)^2$ in the direction of $h$, if it exists, is

$$ \partial L(g, \alpha, \beta; h) \triangleq \left. \frac{d}{d\gamma} r(\gamma) \right|_{\gamma=0}, \quad (325) $$

where the real function $r: \mathbb{R} \to \mathbb{R}$ is such that for all $\gamma \in (-\epsilon, \epsilon)$, with some $\epsilon > 0$, satisfies

$$ \begin{aligned} r(\gamma) &= \int \mathbf{L}_z(\boldsymbol{\nu}) (g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) \\ &\quad + \alpha \left( \int (g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) \log(g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) - c \right) \\ &\quad + \beta \left( \int (g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) - 1 \right). \end{aligned} \quad (326) $$

The proof continues under the assumption that the functions $g$ and $h$ are such that the Gateaux differential in (325) exists. That is, the function $r$ in (326) is differentiable in $(-\epsilon, \epsilon)$, with some $\epsilon > 0$. Using the same arguments as in the proof of Theorem 3, it follows that the derivative of the real function $r$ in (326) is

$$ \begin{aligned} \frac{d}{d\gamma} r(\gamma) &= \int \mathbf{L}_z h(\boldsymbol{\nu}) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) + \alpha \int h(\boldsymbol{\nu}) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) \\ &\quad + \alpha \int h(\boldsymbol{\nu}) \log(g(\boldsymbol{\nu}) + \gamma h(\boldsymbol{\nu})) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}) \\ &\quad + \beta \int h(\boldsymbol{\nu}) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}). \end{aligned} \quad (327) $$

From (325) and (327), it follows that

$$ \begin{aligned} \partial L(g, \alpha, \beta; h) &= \int h(\boldsymbol{\nu}) (\mathbf{L}_z(\boldsymbol{\nu}) + \alpha (1 + \log g(\boldsymbol{\nu}))) + \beta) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu}). \end{aligned} \quad (328) $$

From [73, Theorem 1, page 217], it holds that a necessary condition for the functional $L$ in (324) to have a minimum at $(g, \alpha, \beta) \in \mathcal{M} \times [0, +\infty)^2$ is that for all functions $h \in \mathcal{M}$,

$$ \partial L(g, \alpha, \beta; h) = 0, \quad (329) $$

which implies that for all $\boldsymbol{\nu} \in \mathcal{M}$,

$$ \mathbf{L}_z(\boldsymbol{\nu}) + \alpha (1 + \log g(\boldsymbol{\nu})) + \beta = 0. \quad (330) $$

Thus,

$$ g(\boldsymbol{\nu}) = \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right) \exp\left(-\frac{\beta + \alpha}{\alpha}\right), \quad (331) $$

where $\alpha$ and $\beta$ are chosen to satisfy their corresponding constraints with equality. Denote by $P^*$ the solution of the optimization problem in (82). Hence, from (331), it follows that

$$ \frac{dP^*}{dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\nu})} = \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\alpha}\right) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\theta})}, \quad (332) $$

where $\alpha$ is chosen to satisfy

$$ D(P^* \| P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}) = c. \quad (333) $$

From Lemma 3, it follows that the probability measure $P^*$ and the $\sigma$-finite measure $Q$ satisfy,

$$ \frac{dP^*}{dQ}(\boldsymbol{\nu}) = \frac{dP^*}{dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}}(\boldsymbol{\nu}) \frac{dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}}{dQ}(\boldsymbol{\nu}) \quad (334) $$

$$ = \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\alpha}\right) dP_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}(\boldsymbol{\theta})} \right) \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\lambda}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\lambda}\right) dQ(\boldsymbol{\theta})} \right) \quad (335) $$

$$ = \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\alpha}\right)}{\int \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\alpha}\right) \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\lambda}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\alpha})}{\lambda}\right) dQ(\boldsymbol{\alpha})} dQ(\boldsymbol{\theta})} \right) \left( \frac{\exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\nu})}{\lambda}\right)}{\int \exp\left(-\frac{\mathbf{L}_z(\boldsymbol{\theta})}{\lambda}\right) dQ(\boldsymbol{\theta})} \right) \quad (336) $$

$$ = \frac{\exp\left(-\left(\frac{1}{\alpha} + \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\nu})\right)}{\int \exp\left(-\left(\frac{1}{\alpha} + \frac{1}{\lambda}\right) \mathbf{L}_z(\boldsymbol{\nu})\right) dQ(\boldsymbol{\theta})}, \quad (337) $$

which implies that $P^*$ is a Gibbs probability measure on $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$, with energy function $\mathbf{L}_z$, reference measure $Q$, and regularization parameter $\frac{1}{\alpha+\lambda}$, where $\alpha$ is chosen to satisfy (333). Let the positive real $\omega$ be $\omega \triangleq \frac{\alpha\lambda}{\alpha+\lambda}$ and note that $\omega \in (0, \lambda]$ and satisfies $D(P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\omega)}(\boldsymbol{\nu}) \| P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\lambda)}) = c$. The proof ends by verifying that the objective function in (324) is strictly convex, and thus, the measure $P_{\boldsymbol{\Theta}|\mathbf{Z}=z}^{(Q,\omega)}$ is the unique minimizer. This completes the proof.

## APPENDIX K
### PROOF OF LEMMA 15

Note that for all $(\lambda_1, \lambda_2) \in \{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}^2$, such that $\lambda_1 > \lambda_2$, it follows that for all $\boldsymbol{\theta} \in \text{supp} Q$, the inequality $\exp(\lambda_2 \mathbf{L}_z(\boldsymbol{\theta})) \le \exp(\lambda_1 \mathbf{L}_z(\boldsymbol{\theta}))$ holds. This implies that $K_{Q,z}(\lambda_2) \le K_{Q,z}(\lambda_1) < +\infty$, which proves that the function is nondecreasing.

The proof of continuity of the function $K_{Q,z}$ follows from observing that for all $\alpha \in \{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$, it holds
that

$$
\lim_{t \to \alpha} K_{Q,z}(t) = \lim_{t \to \alpha} \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (338)
$$

$$
= \log \left( \lim_{t \to \alpha} \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (339)
$$

$$
= \log \left( \int \lim_{t \to \alpha} \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (340)
$$

$$
= \log \left( \int \exp (\alpha \mathbf{L}_z (\boldsymbol{\theta})) \mathrm{d}Q (\boldsymbol{\theta}) \right) \quad (341)
$$

$$
= K_{Q,z}(\alpha), \tag{342}
$$

where (339) and (341) follow from the fact that both the
logarithmic and exponential functions are continuous; and the
equality in (340) follows from the monotone convergence
theorem [59, Theorem 1.6.2]. This shows that the function
$K_{Q,z}$ is continuous in $\{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$.

The proof of differentiability follows by considering
the transport of the σ-finite measure Q in (22) from
the measure space (M, B(M)) to the measure
space ([0, +∞), B([0, +∞))) through the function Lz in (3).
Denote the resulting measure in ([0, +∞), B([0, +∞)))
by P. More specifically, for all A ∈ B([0, +∞)),
it holds that P(A) = Q({θ ∈ M : Lz(θ) ∈ A}).
Hence, the function KQ,z satisfies for all t ∈
{ν ∈ R : KQ,z(ν) < +∞},

$$
K_{Q,z}(t) = \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (343)
$$

$$
= \log \left( \int \exp(tw) \, dP(w) \right), \quad (344)
$$

where the equality (344) follows from [59, Theorem 1.6.12].
Denote by φ the Laplace transform of the measure P. That is,
for all t ∈ {x ∈ R : K<sub>Q,z</sub>(x) < +∞},

$$
\phi(t) = \int \exp(tv) \, dP(v). \quad (345)
$$

Hence, $\phi(t) = \exp(K_{Q,z}(t))$. From [74, Theorem 1a (page 439)], it follows that the function $\phi$ has derivatives of all orders in $\{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$, and thus, so does the function $K_{Q,z}$ in the interior of $\{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$. This completes the proof.

## APPENDIX L
PROOF OF LEMMA 16

Let $(\gamma_1, \gamma_2) \in \mathbb{R}^2$, with $\gamma_1 \neq \gamma_2$ and $\alpha \in [0, 1]$ be fixed.
Assume that $K_{Q,z}(\gamma_1) < +\infty$ and $K_{Q,z}(\gamma_2) < +\infty$. Then,
for all $\alpha \in (0, 1)$, the following holds

$$
\begin{align}
& \alpha K_{Q,z}(\gamma_1) + (1-\alpha) K_{Q,z}(\gamma_2) \nonumber \\
& \quad = \alpha \log \left( \int \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \nonumber \\
& \quad \quad + (1-\alpha) \log \left( \int \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \tag{346}
\end{align}
$$

$$
\begin{align}
&= \log \left( \left( \int \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) \, \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\alpha} \right) \nonumber \\
&\quad + \log \left( \left( \int \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta})) \, \mathrm{d}Q(\boldsymbol{\theta}) \right)^{(1-\alpha)} \right) \tag{347}
\end{align}
$$

$$
= \log \left( \left( \int \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\alpha} \right) \\
\left( \int \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right)^{(1-\alpha)} \tag{348}
$$

$$
= \log \left( \left( \int \exp (\gamma_1 \alpha \mathbf{L}_z(\boldsymbol{\theta}))^p \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\frac{1}{p}} \right. \\
\left. \qquad \left( \int \exp (\gamma_2 (1-\alpha) \mathbf{L}_z(\boldsymbol{\theta}))^q \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\frac{1}{q}} \right) \quad (349)
$$

$$
\ge \log \left( \int \exp(\gamma_1 \alpha \mathbf{L}_z(\boldsymbol{\theta})) \exp(\gamma_2 (1-\alpha) \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (350)
$$

$$
= \log \left( \int \exp \left( \left( \gamma_1 \alpha + \gamma_2 (1-\alpha) \right) \mathbf{L}_z(\boldsymbol{\theta}) \right) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (351)
$$

$$
= K_{Q,z}(\gamma_1 \alpha + \gamma_2 (1 - \alpha)), \quad (352)
$$

where the inequality in (349) follows with $\alpha \triangleq \frac{1}{p}$ and $1-\alpha \triangleq \frac{1}{q}$; the inequality in (350) follows from Hölder's inequality. Hence, equality in (350) holds if and only if there exist two constants $\beta_1$ and $\beta_2$, not simultaneously equal to zero, such that the set

$$
\begin{align}
\mathcal{A} \triangleq \{\boldsymbol{\theta} \in \mathcal{M} : \beta_1 \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) = \beta_2 \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta}))\} \nonumber \\
&= \left\{\boldsymbol{\theta} \in \mathcal{M} : \exp\left((\gamma_1 - \gamma_2) \mathbf{L}_z(\boldsymbol{\theta})\right) = \frac{\beta_2}{\beta_1}\right\} \tag{353}
\end{align}
$$

$$
= \left\{ \theta \in \mathcal{M} : \mathbf{L}_z(\theta) = \frac{\log \frac{\beta_2}{\beta_1}}{(\gamma_1 - \gamma_2)} \right\}, \quad (354)
$$

satisfies $Q(\mathcal{A}) = 1$. That is, strict inequality in (350) holds if and only if the function $\mathbf{L}_z$ is separable with respect to the $\sigma$-finite measure $Q$. When $\alpha = 0$ or $\alpha = 1$, the proof is trivial. This completes the proof.

## APPENDIX M
PROOF OF LEMMA 17

For all $s \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the equality in (95) implies the following,

$$
K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) = \frac{d}{dt} \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \Big|_{t=-\frac{1}{s}} \quad (355)
$$

$$
= \int \frac{\mathbf{L}_z(\boldsymbol{\theta}) \exp(t \mathbf{L}_z(\boldsymbol{\theta}))}{\int \exp(t \mathbf{L}_z(\boldsymbol{v})) \mathrm{d}Q(\boldsymbol{v})} \mathrm{d}Q(\boldsymbol{\theta}) \Big|_{t=-\frac{1}{s}} \quad (356)
$$

$$
= \int \frac{\mathbf{L}_z(\boldsymbol{\theta}) \exp\left(-\frac{1}{s} \mathbf{L}_z(\boldsymbol{\theta})\right)}{\int \exp\left(-\frac{1}{s} \mathbf{L}_z(\boldsymbol{v})\right) \mathrm{d}Q(\boldsymbol{v})} \mathrm{d}Q(\boldsymbol{\theta}) \quad (357)
$$

$$
= \exp\left(-K_{Q,z}\left(-\frac{1}{s}\right)\right) \int \mathbf{L}_z(\boldsymbol{\theta}) \exp\left(-\frac{1}{s}\mathbf{L}_z(\boldsymbol{\theta})\right) \mathrm{d}Q(\boldsymbol{\theta}) \quad (358)
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) \exp\left(-K_{Q,z}\left(-\frac{1}{s}\right) - \frac{1}{s} \mathbf{L}_z(\boldsymbol{\theta})\right) \mathrm{d}Q(\boldsymbol{\theta}) \quad (359)
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}), \quad (360)
$$

where the equality in (356) holds from the dominated
convergence theorem [59, Theorem 1.6.9]; the equality
that

$$ \lim_{t \to \alpha} K_{Q,z}(t) = \lim_{t \to \alpha} \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (338) $$

$$ = \log \left( \lim_{t \to \alpha} \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (339) $$

$$ = \log \left( \int \lim_{t \to \alpha} \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (340) $$

$$ = \log \left( \int \exp(\alpha \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (341) $$

$$ = K_{Q,z}(\alpha), \quad (342) $$

where (339) and (341) follow from the fact that both the logarithmic and exponential functions are continuous; and the equality in (340) follows from the monotone convergence theorem [59, Theorem 1.6.2]. This shows that the function $K_{Q,z}$ is continuous in $\{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$.

The proof of differentiability follows by considering the transport of the $\sigma$-finite measure $Q$ in (22) from the measure space $(\mathcal{M}, \mathcal{B}(\mathcal{M}))$ to the measure space $([0, +\infty), \mathcal{B}([0, +\infty)))$ through the function $\mathbf{L}_z$ in (3). Denote the resulting measure in $([0, +\infty), \mathcal{B}([0, +\infty)))$ by $P$. More specifically, for all $A \in \mathcal{B}([0, +\infty))$, it holds that $P(A) = Q(\{\boldsymbol{\theta} \in \mathcal{M} : \mathbf{L}_z(\boldsymbol{\theta}) \in A\})$. Hence, the function $K_{Q,z}$ satisfies for all $t \in \{\nu \in \mathbb{R} : K_{Q,z}(\nu) < +\infty\}$,

$$ K_{Q,z}(t) = \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (343) $$

$$ = \log \left( \int \exp(t w) \mathrm{d}P(w) \right), \quad (344) $$

where the equality (344) follows from [59, Theorem 1.6.12]. Denote by $\phi$ the Laplace transform of the measure $P$. That is, for all $t \in \{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$,

$$ \phi(t) = \int \exp(t v) \mathrm{d}P(v). \quad (345) $$

Hence, $\phi(t) = \exp(K_{Q,z}(t))$. From [74, Theorem 1a (page 439)], it follows that the function $\phi$ has derivatives of all orders in $\{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$, and thus, so does the function $K_{Q,z}$ in the interior of $\{x \in \mathbb{R} : K_{Q,z}(x) < +\infty\}$. This completes the proof.

## APPENDIX L
PROOF OF LEMMA 16

Let $(\gamma_1, \gamma_2) \in \mathbb{R}^2$, with $\gamma_1 \neq \gamma_2$ and $\alpha \in [0, 1]$ be fixed. Assume that $K_{Q,z}(\gamma_1) < +\infty$ and $K_{Q,z}(\gamma_2) < +\infty$. Then, for all $\alpha \in (0, 1)$, the following holds

$$ \begin{aligned} & \alpha K_{Q,z}(\gamma_1) + (1-\alpha) K_{Q,z}(\gamma_2) \\ &= \alpha \log \left( \int \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \\ & \quad + (1-\alpha) \log \left( \int \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \end{aligned} \quad (346) $$

$$ \begin{aligned} & = \log \left( \left( \int \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\alpha} \right) \\ & \quad + \log \left( \left( \int \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right)^{(1-\alpha)} \right) \end{aligned} \quad (347) $$

$$ = \log \left( \left( \int \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\alpha} \left( \int \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right)^{(1-\alpha)} \right) \quad (348) $$

$$ = \log \left( \left( \int \exp(\gamma_1 \alpha \mathbf{L}_z(\boldsymbol{\theta}))^p \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\frac{1}{p}} \left( \int \exp(\gamma_2(1-\alpha) \mathbf{L}_z(\boldsymbol{\theta}))^q \mathrm{d}Q(\boldsymbol{\theta}) \right)^{\frac{1}{q}} \right) \quad (349) $$

$$ \ge \log \left( \int \exp(\gamma_1 \alpha \mathbf{L}_z(\boldsymbol{\theta})) \exp(\gamma_2(1-\alpha) \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (350) $$

$$ = \log \left( \int \exp \left( \left( \gamma_1 \alpha + \gamma_2 (1-\alpha) \right) \mathbf{L}_z(\boldsymbol{\theta}) \right) \mathrm{d}Q(\boldsymbol{\theta}) \right) \quad (351) $$

$$ = K_{Q,z}(\gamma_1 \alpha + \gamma_2 (1-\alpha)), \quad (352) $$

where the inequality in (349) follows with $\alpha \triangleq \frac{1}{p}$ and $1-\alpha \triangleq \frac{1}{q}$; the inequality in (350) follows from Hölder's inequality. Hence, equality in (350) holds if and only if there exist two constants $\beta_1$ and $\beta_2$, not simultaneously equal to zero, such that the set

$$ \begin{aligned} & \mathcal{A} \triangleq \{\boldsymbol{\theta} \in \mathcal{M} : \beta_1 \exp(\gamma_1 \mathbf{L}_z(\boldsymbol{\theta})) = \beta_2 \exp(\gamma_2 \mathbf{L}_z(\boldsymbol{\theta}))\} \\ &= \left\{\boldsymbol{\theta} \in \mathcal{M} : \exp\left(\left(\gamma_1 - \gamma_2\right) \mathbf{L}_z(\boldsymbol{\theta})\right) = \frac{\beta_2}{\beta_1}\right\} \end{aligned} \quad (353) $$

$$ = \left\{ \boldsymbol{\theta} \in \mathcal{M} : \mathbf{L}_z(\boldsymbol{\theta}) = \frac{\log \frac{\beta_2}{\beta_1}}{(\gamma_1 - \gamma_2)} \right\}, \quad (354) $$

satisfies $Q(\mathcal{A}) = 1$. That is, strict inequality in (350) holds if and only if the function $\mathbf{L}_z$ is separable with respect to the $\sigma$-finite measure $Q$. When $\alpha = 0$ or $\alpha = 1$, the proof is trivial. This completes the proof.

## APPENDIX M
PROOF OF LEMMA 17

For all $s \in K_{Q,z}$, with $K_{Q,z}$ in (23), the equality in (95) implies the following,

$$ K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) = \frac{d}{dt} \log \left( \int \exp(t \mathbf{L}_z(\boldsymbol{\theta})) \mathrm{d}Q(\boldsymbol{\theta}) \right) \Big|_{t=-\frac{1}{s}} \quad (355) $$

$$ = \int \frac{\mathbf{L}_z(\boldsymbol{\theta}) \exp(t \mathbf{L}_z(\boldsymbol{\theta}))}{\int \exp(t \mathbf{L}_z(\boldsymbol{\nu})) \mathrm{d}Q(\boldsymbol{\nu})} \mathrm{d}Q(\boldsymbol{\theta}) \Big|_{t=-\frac{1}{s}} \quad (356) $$

$$ = \int \frac{\mathbf{L}_z(\boldsymbol{\theta}) \exp\left(-\frac{1}{s} \mathbf{L}_z(\boldsymbol{\theta})\right)}{\int \exp\left(-\frac{1}{s} \mathbf{L}_z(\boldsymbol{\nu})\right) \mathrm{d}Q(\boldsymbol{\nu})} \mathrm{d}Q(\boldsymbol{\theta}) \quad (357) $$

$$ = \exp\left(-K_{Q,z}\left(-\frac{1}{s}\right)\right) \int \mathbf{L}_z(\boldsymbol{\theta}) \exp\left(-\frac{1}{s} \mathbf{L}_z(\boldsymbol{\theta})\right) \mathrm{d}Q(\boldsymbol{\theta}) \quad (358) $$

$$ = \int \mathbf{L}_z(\boldsymbol{\theta}) \exp\left(-K_{Q,z}\left(-\frac{1}{s}\right) - \frac{1}{s} \mathbf{L}_z(\boldsymbol{\theta})\right) \mathrm{d}Q(\boldsymbol{\theta}) \quad (359) $$

$$ = \int \mathbf{L}_z(\boldsymbol{\theta}) \mathrm{d}P_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}), \quad (360) $$

where the equality in (356) holds from the dominated convergence theorem [59, Theorem 1.6.9]; the equality
in (358) follows from (22); and the equality in (360) follows
from (25).

For all $s \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the equalities in (95) and (359) imply that

$$
\begin{align}
& K_{Q,z}^{(2)} \left(-\frac{1}{s}\right) \nonumber \\
&= \left. \frac{d}{dt} \int \mathbf{L}_z(\boldsymbol{\theta}) \exp(-K_{Q,z}(t) + t \mathbf{L}_z(\boldsymbol{\theta})) dQ(\boldsymbol{\theta}) \right|_{t=-\frac{1}{s}} \tag{361}
\end{align}
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) (-K_{Q,z}^{(1)}(t) + \mathbf{L}_z(\boldsymbol{\theta})) \\
\exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) dQ(\boldsymbol{\theta}) \Big|_{t=-\frac{1}{s}} \quad (362)
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) \left( -K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) + \mathbf{L}_z(\boldsymbol{\theta}) \right) \\
\exp\left(-K_{Q,z}\left(-\frac{1}{s}\right) - \frac{1}{s}\mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta}) \quad (363)
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) \left( -K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) + \mathbf{L}_z(\boldsymbol{\theta}) \right) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \quad (364)
$$

$$
= -K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) \int \mathbf{L}_z(\boldsymbol{\theta}) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
+ \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \quad (365)
$$

$$
= - \left( K_{Q,z}^{(1)} \left( -\frac{1}{s} \right) \right)^2 + \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \quad (366)
$$

$$
= \int \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)} \left( -\frac{1}{s} \right) \right)^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}), \quad (367)
$$

where the equality in (362) follows from the dominated
convergence theorem [59, Theorem 1.6.9]; the equality
in (364) is due to a change of measure through the Radon-
Nikodym derivative in (25); and the equality in (366) follows
from (360).

For all $s \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the equalities in (95) and (366) imply that

$$
\begin{align}
& K_{Q,z}^{(3)} \left(-\frac{1}{s}\right) \nonumber \\
&= \left. \frac{d}{dt} \left( \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,-\frac{1}{s})}(\boldsymbol{\theta}) - (K_{Q,z}^{(1)}(t))^2 \right) \right|_{t=-\frac{1}{s}} \tag{368}
\end{align}
$$

$$
= \frac{d}{dt} \left( \int \left( (\mathbf{L}_z(\boldsymbol{\theta}))^2 \exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) \right. \\
\left. - (K_{Q,z}^{(1)}(t))^2 \right) \Bigg|_{t=-\frac{1}{s}} \qquad (369)
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( \left. \frac{d}{dt} \exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) \right|_{t=-\frac{1}{s}} \right) dQ(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}(t) K_{Q,z}^{(2)}(t) \Bigg|_{t=-\frac{1}{s}} \quad (370)
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( (\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}(t)) \exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) \right) \Bigg|_{t=-\frac{1}{s}} dQ(\boldsymbol{\theta}) \quad (371) \\
-2K_{Q,z}^{(1)}(t) K_{Q,z}^{(2)}(t) \Bigg|_{t=-\frac{1}{s}}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \right) \\
\exp\left(-K_{Q,z}\left(-\frac{1}{s}\right) - \frac{1}{s}\mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{372}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \right) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \qquad (373)
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
- K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{374}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
- K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \left( K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) + \left(K_{Q,z}^{(1)}\left(-\frac{1}{s}\right)\right)^2 \right) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{375}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) - K_{Q,z}^{(1)} \left(-\frac{1}{s}\right)^3 \quad (376)
$$

$$
-3K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) K_{Q,z}^{(2)} \left(-\frac{1}{s}\right) \qquad (377)
$$

$$
= \int \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)} \left( -\frac{1}{s} \right) \right)^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}), \quad (378)
$$

where the equality in (369) follows from (25); and the equality
in (370) follows from the dominated convergence theorem [59,
Theorem 1.6.9]; the equality in (373) follows from (25); and
the equality in (375) follows from (366).

This completes the proof.

APPENDIX N
PROOF OF THEOREM 5

The proof is based on the analysis of the derivative
of $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ with respect to $\lambda$ in $\text{int}\mathcal{K}_{Q,z}$. This is due to
Corollary 2. For instance, note that

$$
\frac{d}{d\lambda} \mathrm{R}_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) = \frac{d}{d\lambda} K_{Q,z}^{(1)} \left( -\frac{1}{\lambda} \right) \quad (379)
$$

$$
= \frac{1}{\lambda^2} K_{Q,z}^{(2)} \left( -\frac{1}{\lambda} \right) \quad (380)
$$

$$
\ge 0, \tag{381}
$$

where the equality in (380) follows from Lemma 17. The
inequality in (381) implies that the expected empirical
in (358) follows from (22); and the equality in (360) follows
from (25).

For all $s \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the equalities in (95) and (359) imply that

$$
\begin{align}
& K_{Q,z}^{(2)} \left(-\frac{1}{s}\right) \nonumber \\
&= \left. \frac{d}{dt} \int \mathbf{L}_z(\boldsymbol{\theta}) \exp(-K_{Q,z}(t) + t \mathbf{L}_z(\boldsymbol{\theta})) dQ(\boldsymbol{\theta}) \right|_{t=-\frac{1}{s}} \tag{361}
\end{align}
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) (-K_{Q,z}^{(1)}(t) + \mathbf{L}_z(\boldsymbol{\theta})) \\
\exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) dQ(\boldsymbol{\theta}) \Big|_{t=-\frac{1}{s}} \quad (362)
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) \left( -K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) + \mathbf{L}_z(\boldsymbol{\theta}) \right) \\
\exp\left(-K_{Q,z}\left(-\frac{1}{s}\right) - \frac{1}{s}\mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta}) \quad (363)
$$

$$
= \int \mathbf{L}_z(\boldsymbol{\theta}) \left( -K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) + \mathbf{L}_z(\boldsymbol{\theta}) \right) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \quad (364)
$$

$$
= -K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) \int \mathbf{L}_z(\boldsymbol{\theta}) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
+ \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \quad (365)
$$

$$
= - \left( K_{Q,z}^{(1)} \left( -\frac{1}{s} \right) \right)^2 + \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \quad (366)
$$

$$
= \int \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)} \left( -\frac{1}{s} \right) \right)^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}), \quad (367)
$$

where the equality in (362) follows from the dominated
convergence theorem [59, Theorem 1.6.9]; the equality
in (364) is due to a change of measure through the Radon-
Nikodym derivative in (25); and the equality in (366) follows
from (360).

For all $s \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the equalities in (95) and (366) imply that

$$
\begin{align}
& K_{Q,z}^{(3)} \left(-\frac{1}{s}\right) \nonumber \\
&= \left. \frac{d}{dt} \left( \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,-\frac{1}{s})}(\boldsymbol{\theta}) - (K_{Q,z}^{(1)}(t))^2 \right) \right|_{t=-\frac{1}{s}} \tag{368}
\end{align}
$$

$$
= \frac{d}{dt} \left( \int \left( (\mathbf{L}_z(\boldsymbol{\theta}))^2 \exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) \right) dQ(\boldsymbol{\theta}) - (K_{Q,z}^{(1)}(t))^2 \right) \Bigg|_{t=-\frac{1}{s}} \quad (369)
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( \left. \frac{d}{dt} \exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) \right|_{t=-\frac{1}{s}} \right) dQ(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}(t) K_{Q,z}^{(2)}(t) \Bigg|_{t=-\frac{1}{s}} \quad (370)
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( (\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}(t)) \exp(-K_{Q,z}(t) + t\mathbf{L}_z(\boldsymbol{\theta})) \right) \Bigg|_{t=-\frac{1}{s}} dQ(\boldsymbol{\theta}) \tag{371} \\
-2K_{Q,z}^{(1)}(t) K_{Q,z}^{(2)}(t) \Bigg|_{t=-\frac{1}{s}}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \right) \\
\exp\left(-K_{Q,z}\left(-\frac{1}{s}\right) - \frac{1}{s}\mathbf{L}_z(\boldsymbol{\theta})\right) dQ(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{372}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \right) dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{373}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
- K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \int (\mathbf{L}_z(\boldsymbol{\theta}))^2 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{374}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) \\
- K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) \left( K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) + (K_{Q,z}^{(1)}\left(-\frac{1}{s}\right))^2 \right) \\
-2K_{Q,z}^{(1)}\left(-\frac{1}{s}\right) K_{Q,z}^{(2)}\left(-\frac{1}{s}\right) \tag{375}
$$

$$
= \int (\mathbf{L}_z(\boldsymbol{\theta}))^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}) - K_{Q,z}^{(1)} \left(-\frac{1}{s}\right)^3 \quad (376)
$$

$$
-3K_{Q,z}^{(1)} \left(-\frac{1}{s}\right) K_{Q,z}^{(2)} \left(-\frac{1}{s}\right) \qquad (377)
$$

$$
= \int \left( \mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)} \left( -\frac{1}{s} \right) \right)^3 dP_{\Theta|Z=z}^{(Q,s)}(\boldsymbol{\theta}), \quad (378)
$$

where the equality in (369) follows from (25); and the equality
in (370) follows from the dominated convergence theorem [59,
Theorem 1.6.9]; the equality in (373) follows from (25); and
the equality in (375) follows from (366).

This completes the proof.

APPENDIX N

PROOF OF THEOREM 5

The proof is based on the analysis of the derivative
of $K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ with respect to $\lambda$ in $\text{int}\mathcal{K}_{Q,z}$. This is due to
Corollary 2. For instance, note that

$$
\frac{d}{d\lambda} \mathrm{R}_z \left( P_{\Theta|Z=z}^{(Q,\lambda)} \right) = \frac{d}{d\lambda} K_{Q,z}^{(1)} \left( -\frac{1}{\lambda} \right) \quad (379)
$$

$$
= \frac{1}{\lambda^2} K_{Q,z}^{(2)} \left( -\frac{1}{\lambda} \right) \quad (380)
$$

$$
\ge 0, \tag{381}
$$

where the equality in (380) follows from Lemma 17. The
inequality in (381) implies that the expected empirical
risk $\mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ in (104) is nondecreasing with respect to $\lambda$. The rest of the proof consists in showing that for all $\alpha \in \mathcal{K}_{Q,z}$, the function $K_{Q,z}^{(2)}$ in (95) satisfies $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$ if and only if the function $\mathbb{L}_z$ in (3) is separable. For doing so, a handful of preliminary results are described in the following subsection. The proof of Theorem 5 resumes in Subsection N-B

A. Preliminaries

Given a positive real $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), consider a partition of $\mathcal{M}$ formed by the sets $\mathcal{R}_0(\lambda)$, $\mathcal{R}_1(\lambda)$ and $\mathcal{R}_2(\lambda)$, such that

$$ \mathcal{R}_0(\lambda) \triangleq \{\nu \in \mathcal{M} : \mathbb{L}_z(\nu) = \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)})\}, \quad (382a) $$

$$ \mathcal{R}_1(\lambda) \triangleq \{\nu \in \mathcal{M} : \mathbb{L}_z(\nu) < \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)})\}, \quad \text{and} \quad (382b) $$

$$ \mathcal{R}_2(\lambda) \triangleq \{\nu \in \mathcal{M} : \mathbb{L}_z(\nu) > \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)})\}, \quad (382c) $$

where the functional $\mathbb{R}_z$ is in (18) and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25). The sets in (382) exhibit several properties that are central for proving the main results of this section.

Lemma 26: The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), satisfies

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_1(\lambda)) > 0, \quad (383) $$

if and only if

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_2(\lambda)) > 0, \quad (384) $$

where the sets $\mathcal{R}_1(\cdot)$ and $\mathcal{R}_2(\cdot)$ are in (382b) and (382c), respectively.

*Proof:* The proof is divided into two parts. In the first part, given a real $\alpha \in \mathcal{K}_{Q,z}$, it is proven that if the set $\mathcal{R}_1(\alpha)$ is nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$, then the set $\mathcal{R}_2(\alpha)$ is nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. The second part proves the converse.

The first part is proved by contradiction. Assume that set $\mathcal{R}_2(\alpha)$ is negligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. Hence, from

Lemma 17, it holds that

$$ K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) = \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \\ + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \quad (385) $$

$$ = \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \quad (386) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \quad (387) $$

$$ < K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ + K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_1(\alpha)) \quad (388) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) \left( P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) + P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_1(\alpha)) \right) \quad (389) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right), \quad (390) $$

which is a contradiction.

The second part of the proof follows the same arguments as in the first part. Assume that the set $\mathcal{R}_1(\alpha)$ is negligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. Hence, from Lemma 17, it holds that

$$ K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) = \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \\ + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \quad (391) $$

$$ = \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \quad (392) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\nu) dP_{\Theta|Z=z}^{(Q,\alpha)}(\nu) \quad (393) $$

$$ > K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ + K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_2(\alpha)) \quad (394) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) \left( P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) + P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_2(\alpha)) \right) \quad (395) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right), \quad (396) $$

which is also a contradiction. This completes the proof.

A more general result can be immediately obtained by combining Lemma 4 and Lemma 26.
risk $\mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)}) = K_{Q,z}^{(1)}(-\frac{1}{\lambda})$ in (104) is nondecreasing with respect to $\lambda$. The rest of the proof consists in showing that for all $\alpha \in \mathcal{K}_{Q,z}$, the function $K_{Q,z}^{(2)}$ in (95) satisfies $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$ if and only if the function $\mathbb{L}_z$ in (3) is separable. For doing so, a handful of preliminary results are described in the following subsection. The proof of Theorem 5 resumes in Subsection N-B

## A. Preliminaries

Given a positive real $\lambda \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), consider a partition of $\mathcal{M}$ formed by the sets $\mathcal{R}_0(\lambda)$, $\mathcal{R}_1(\lambda)$ and $\mathcal{R}_2(\lambda)$, such that

$$ \mathcal{R}_0(\lambda) \triangleq \{ \boldsymbol{\nu} \in \mathcal{M} : \mathbb{L}_z(\boldsymbol{\nu}) = \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \}, \quad (382a) $$

$$ \mathcal{R}_1(\lambda) \triangleq \{ \boldsymbol{\nu} \in \mathcal{M} : \mathbb{L}_z(\boldsymbol{\nu}) < \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \}, \quad \text{and} \quad (382b) $$

$$ \mathcal{R}_2(\lambda) \triangleq \{ \boldsymbol{\nu} \in \mathcal{M} : \mathbb{L}_z(\boldsymbol{\nu}) > \mathbb{R}_z(P_{\Theta|Z=z}^{(Q,\lambda)}) \}, \quad (382c) $$

where the functional $\mathbb{R}_z$ is in (18) and the probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ is in (25). The sets in (382) exhibit several properties that are central for proving the main results of this section.

**Lemma 26:** The probability measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), satisfies

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_1(\lambda)) > 0, \quad (383) $$

if and only if

$$ P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_2(\lambda)) > 0, \quad (384) $$

where the sets $\mathcal{R}_1(\cdot)$ and $\mathcal{R}_2(\cdot)$ are in (382b) and (382c), respectively.

**Proof:** The proof is divided into two parts. In the first part, given a real $\alpha \in \mathcal{K}_{Q,z}$, it is proven that if the set $\mathcal{R}_1(\alpha)$ is nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$, then the set $\mathcal{R}_2(\alpha)$ is nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. The second part proves the converse.

The first part is proved by contradiction. Assume that set $\mathcal{R}_2(\alpha)$ is negligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. Hence, from

Lemma 17, it holds that

$$ \begin{aligned} & K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) \\ &= \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \\ &\quad + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \end{aligned} \quad (385) $$

$$ = \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \quad (386) $$

$$ \begin{aligned} & K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ &\quad + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \end{aligned} \quad (387) $$

$$ \begin{aligned} & <K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ &\quad + K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_1(\alpha)) \end{aligned} \quad (388) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) \left( P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) + P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_1(\alpha)) \right) \quad (389) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right), \quad (390) $$

which is a contradiction.

The second part of the proof follows the same arguments as in the first part. Assume that the set $\mathcal{R}_1(\alpha)$ is negligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. Hence, from Lemma 17, it holds that

$$ \begin{aligned} & K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) \\ &= \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) + \int_{\mathcal{R}_1(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \\ &\quad + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \end{aligned} \quad (391) $$

$$ = \int_{\mathcal{R}_0(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \quad (392) $$

$$ \begin{aligned} & = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ &\quad + \int_{\mathcal{R}_2(\alpha)} \mathbb{L}_z(\boldsymbol{\nu}) dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\nu}) \end{aligned} \quad (393) $$

$$ \begin{aligned} & > K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) \\ &\quad + K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_2(\alpha)) \end{aligned} \quad (394) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right) \left( P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_0(\alpha)) + P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_2(\alpha)) \right) \quad (395) $$

$$ = K_{Q,z}^{(1)} \left(-\frac{1}{\alpha}\right), \quad (396) $$

which is also a contradiction. This completes the proof.

A more general result can be immediately obtained by combining Lemma 4 and Lemma 26.
**Lemma 27:** For all $\alpha \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), satisfies

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_1(\alpha)) > 0, \quad (397)$$

if and only if

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_2(\alpha)) > 0, \quad (398)$$

where the sets $\mathcal{R}_1(\alpha)$ and $\mathcal{R}_2(\alpha)$ are in (382b) and (382c), respectively.

## B. The proof

The rest of the proof of Theorem 5 is divided into two parts. In the first part, it is shown that if for all $\alpha \in \mathcal{K}_{Q,z}$, $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$, then the function $L_z$ in (3) is separable. The second part of the proof, consists in showing that if the function $L_z$ is separable, then, for all $\alpha \in \mathcal{K}_{Q,z}$, $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$.

The first part is as follows. From Lemma 17, it holds that for all $\alpha \in \mathcal{K}_{Q,z}$,

$$ 
\begin{aligned}
& K_{Q,z}^{(2)}\left(-\frac{1}{\alpha}\right) \\
&= \int \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \quad (399) \\
&= \int_{\mathcal{R}_0(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \\
&\quad + \int_{\mathcal{R}_1(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \\
&\quad + \int_{\mathcal{R}_2(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}), \quad (400)
\end{aligned}
$$

where the sets $\mathcal{R}_0(\alpha)$, $\mathcal{R}_1(\alpha)$, and $\mathcal{R}_2(\alpha)$ are respectively defined in (382). Hence,

$$ 
\begin{aligned}
& K_{Q,z}^{(2)}\left(-\frac{1}{\alpha}\right) \\
&= \int_{\mathcal{R}_1(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \quad (401) \\
&\quad + \int_{\mathcal{R}_2(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}). \quad (402)
\end{aligned}
$$

Under the assumption that for all $\alpha \in \mathcal{K}_{Q,z}$ the function $K_{Q,z}^{(2)}$ in (95) satisfies $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$, it follows that at least one of the following claims is true:

(a) $P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_1(\alpha)) > 0$; and

(b) $P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_2(\alpha)) > 0$.

Nonetheless, from Lemma 26, it follows that both claims (a) and (b) hold simultaneously. Hence, the sets $\mathcal{R}_1(\alpha)$ and $\mathcal{R}_2(\alpha)$ are both nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$ and moreover, it holds that for all $(\nu_1, \nu_2) \in \mathcal{R}_1(\alpha) \times \mathcal{R}_2(\alpha)$,

$$+\infty > \mathbf{L}_z(\nu_1) > K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right) > \mathbf{L}_z(\nu_2), \quad (403)$$

where $\mathbf{L}_z(\nu_1) < +\infty$ follows from the fact that $P_{\Theta|Z=z}^{(Q,\lambda)}(\{\boldsymbol{\theta} \in \mathcal{M} : \mathbf{L}_z(\boldsymbol{\theta}) = +\infty\}) = 0$ (Lemma 2). This

proves that under the assumption that for all $\alpha \in \mathcal{K}_{Q,z}$, $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$, the function $\mathbf{L}_z$ in (3) is separable with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. From Lemma 14, it holds that the function $\mathbf{L}_z$ is separable with respect to $\mathcal{Q}$. This completes the first part of the proof.

The second part of the proof is simpler. Assume that the empirical risk function $\mathbf{L}_z$ in (3) is separable with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. That is, for all $\gamma \in \mathcal{K}_{Q,z}$, there exist a positive real $c_\gamma > 0$; and two subsets $\mathcal{A}(\gamma)$ and $\mathcal{B}(\gamma)$ of $\mathcal{M}$ that are nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\gamma)}$ in (25) and verify that for all $(\nu_1, \nu_2) \in \mathcal{A}(\gamma) \times \mathcal{B}(\gamma)$,

$$+\infty > \mathbf{L}_z(\nu_1) > c_\gamma > \mathbf{L}_z(\nu_2). \quad (404)$$

From Lemma 17, it holds that

$$K_{Q,z}^{(2)}\left(-\frac{1}{\gamma}\right) = \int \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (405)$$

$$= \int_{\mathcal{A}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (406)$$

$$+ \int_{\mathcal{B}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (407)$$

$$+ \int_{\mathcal{M} \setminus (\mathcal{A}(\gamma) \cup \mathcal{B}(\gamma))} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (408) \\ > 0, \quad (409)$$

where the inequality (409) follows from the following facts. First, if $c_\gamma < K_{Q,z}^{(1)}(-\frac{1}{\gamma})$, with $c_\gamma$ in (404), then for all $\boldsymbol{\nu} \in \mathcal{B}(\gamma)$, it holds that $K_{Q,z}^{(1)}(-\frac{1}{\gamma}) > c_\gamma > \mathbf{L}_z(\boldsymbol{\nu})$, and thus,

$$\left(\mathbf{L}_z(\boldsymbol{\nu}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2, \quad (410)$$

which implies,

$$ \int_{\mathcal{B}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \\ > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{B}(\gamma)) \quad (411) $$

$$> 0. \quad (412)$$

Second, if $c_\gamma \ge K_{Q,z}^{(1)}(-\frac{1}{\gamma})$ then for all $\boldsymbol{\nu} \in \mathcal{A}(\gamma)$, it holds that $\mathbf{L}_z(\boldsymbol{\nu}) > c_\gamma \ge K_{Q,z}^{(1)}(-\frac{1}{\gamma})$, and thus,

$$\left(\mathbf{L}_z(\boldsymbol{\nu}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2, \quad (413)$$

which implies,

$$ \int_{\mathcal{A}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \\ > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{A}(\gamma)) \quad (414) $$

$$> 0. \quad (415)$$
**Lemma 27:** For all $\alpha \in \mathcal{K}_{Q,z}$, with $\mathcal{K}_{Q,z}$ in (23), the measure $P_{\Theta|Z=z}^{(Q,\lambda)}$ in (25), satisfies

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_1(\alpha)) > 0, \quad (397)$$

if and only if

$$P_{\Theta|Z=z}^{(Q,\lambda)}(\mathcal{R}_2(\alpha)) > 0, \quad (398)$$

where the sets $\mathcal{R}_1(\alpha)$ and $\mathcal{R}_2(\alpha)$ are in (382b) and (382c), respectively.

## B. The proof

The rest of the proof of Theorem 5 is divided into two parts. In the first part, it is shown that if for all $\alpha \in \mathcal{K}_{Q,z}$, $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$, then the function $L_z$ in (3) is separable. The second part of the proof, consists in showing that if the function $L_z$ is separable, then, for all $\alpha \in \mathcal{K}_{Q,z}$, $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$.

The first part is as follows. From Lemma 17, it holds that for all $\alpha \in \mathcal{K}_{Q,z}$,

$$ 
\begin{aligned}
& K_{Q,z}^{(2)}\left(-\frac{1}{\alpha}\right) \\
&= \int \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \quad (399) \\
&= \int_{\mathcal{R}_0(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \\
&\quad + \int_{\mathcal{R}_1(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \\
&\quad + \int_{\mathcal{R}_2(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}), \quad (400)
\end{aligned}
$$

where the sets $\mathcal{R}_0(\alpha)$, $\mathcal{R}_1(\alpha)$, and $\mathcal{R}_2(\alpha)$ are respectively defined in (382). Hence,

$$ 
\begin{aligned}
& K_{Q,z}^{(2)}\left(-\frac{1}{\alpha}\right) \\
&= \int_{\mathcal{R}_1(\alpha)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}) \quad (401) \\
&\quad + \int_{\mathcal{R}_2(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\alpha)}(\boldsymbol{\theta}). \quad (402)
\end{aligned}
$$

Under the assumption that for all $\alpha \in \mathcal{K}_{Q,z}$ the function $K_{Q,z}^{(2)}$ in (95) satisfies $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$, it follows that at least one of the following claims is true:

(a) $P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_1(\alpha)) > 0$; and

(b) $P_{\Theta|Z=z}^{(Q,\alpha)}(\mathcal{R}_2(\alpha)) > 0$.

Nonetheless, from Lemma 26, it follows that both claims (a) and (b) hold simultaneously. Hence, the sets $\mathcal{R}_1(\alpha)$ and $\mathcal{R}_2(\alpha)$ are both nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$ and moreover, it holds that for all $(\nu_1, \nu_2) \in \mathcal{R}_1(\alpha) \times \mathcal{R}_2(\alpha)$,

$$+\infty > \mathbf{L}_z(\nu_1) > K_{Q,z}^{(1)}\left(-\frac{1}{\alpha}\right) > \mathbf{L}_z(\nu_2), \quad (403)$$

where $\mathbf{L}_z(\nu_1) < +\infty$ follows from the fact that $P_{\Theta|Z=z}^{(Q,\lambda)}(\{\boldsymbol{\theta} \in \mathcal{M} : \mathbf{L}_z(\boldsymbol{\theta}) = +\infty\}) = 0$ (Lemma 2). This

proves that under the assumption that for all $\alpha \in \mathcal{K}_{Q,z}$, $K_{Q,z}^{(2)}(-\frac{1}{\alpha}) > 0$, the function $\mathbf{L}_z$ in (3) is separable with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. From Lemma 14, it holds that the function $\mathbf{L}_z$ is separable with respect to $Q$. This completes the first part of the proof.

The second part of the proof is simpler. Assume that the empirical risk function $\mathbf{L}_z$ in (3) is separable with respect to $P_{\Theta|Z=z}^{(Q,\alpha)}$. That is, for all $\gamma \in \mathcal{K}_{Q,z}$, there exist a positive real $c_\gamma > 0$; and two subsets $\mathcal{A}(\gamma)$ and $\mathcal{B}(\gamma)$ of $\mathcal{M}$ that are nonnegligible with respect to $P_{\Theta|Z=z}^{(Q,\gamma)}$ in (25) and verify that for all $(\nu_1, \nu_2) \in \mathcal{A}(\gamma) \times \mathcal{B}(\gamma)$,

$$+\infty > \mathbf{L}_z(\nu_1) > c_\gamma > \mathbf{L}_z(\nu_2). \quad (404)$$

From Lemma 17, it holds that

$$ K_{Q,z}^{(2)}\left(-\frac{1}{\gamma}\right) = \int \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (405) $$

$$= \int_{\mathcal{A}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (406)$$

$$+ \int_{\mathcal{B}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (407)$$

$$+ \int_{\mathcal{M} \setminus (\mathcal{A}(\gamma) \cup \mathcal{B}(\gamma))} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \quad (408) \\ > 0, \quad (409)$$

where the inequality (409) follows from the following facts. First, if $c_\gamma < K_{Q,z}^{(1)}(-\frac{1}{\gamma})$, with $c_\gamma$ in (404), then for all $\boldsymbol{\nu} \in \mathcal{B}(\gamma)$, it holds that $K_{Q,z}^{(1)}(-\frac{1}{\gamma}) > c_\gamma > \mathbf{L}_z(\boldsymbol{\nu})$, and thus,

$$\left(\mathbf{L}_z(\boldsymbol{\nu}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2, \quad (410)$$

which implies,

$$ \int_{\mathcal{B}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \\ > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{B}(\gamma)) \quad (411) $$

$$> 0. \quad (412)$$

Second, if $c_\gamma \ge K_{Q,z}^{(1)}(-\frac{1}{\gamma})$ then for all $\boldsymbol{\nu} \in \mathcal{A}(\gamma)$, it holds that $\mathbf{L}_z(\boldsymbol{\nu}) > c_\gamma \ge K_{Q,z}^{(1)}(-\frac{1}{\gamma})$, and thus,

$$\left(\mathbf{L}_z(\boldsymbol{\nu}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2, \quad (413)$$

which implies,

$$ \int_{\mathcal{A}(\gamma)} \left(\mathbf{L}_z(\boldsymbol{\theta}) - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 dP_{\Theta|Z=z}^{(Q,\gamma)}(\boldsymbol{\theta}) \\ > \left(c_\gamma - K_{Q,z}^{(1)}\left(-\frac{1}{\gamma}\right)\right)^2 P_{\Theta|Z=z}^{(Q,\gamma)}(\mathcal{A}(\gamma)) \quad (414) $$

$$> 0. \quad (415)$$