SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we present the formal version of Theorem 3.1. We first state the general conditions under which this result is true, and give an outline of its proof. Recall that an instance of the optimization problem is given by a tuple  $\mathcal{I} \coloneqq (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$ , where  $\Omega$  is a closed interval in  $\mathbb{R}$  (i.e.,  $\Omega$  is one of  $[a, b], [a, \infty), (-\infty, b], (-\infty, \infty)$  for some suitable real values  $a$  or  $b$ ),  $f_{\mathcal{D}}$  is the density of the true utility that is Lebesgue measurable,  $\ell$  is the loss function,  $\tau$  is the resource-information parameter and  $\alpha \geq 1$  is the risk-averseness parameter. The main result, Theorem C.1, states that under mild conditions, the instance  $\mathcal{I}$  has a unique optimal solution. We present applications of Theorem C.1 when the input density is Gaussian, Pareto, Exponential, and Laplace in Sections H, I, J, and K respectively.
In Section C.1, we state the primal optimization formulation and the assumptions needed by Theorem C.1. Clearly, we need that the instance  $\mathcal{I}$  has a non-empty set of solutions, i.e., there exists a density of entropy at least  $\tau$  (assumption (A0)). In Section C.2, we state the dual of the convex program PrimalOpt and show that weak duality holds. This proof requires integrability of the loss function with respect to the measure induced by the density  $f_{\mathcal{D}}$  (assumption (A3)). This assumption also ensures that the optimal value does not become infinite.
In Section C.3, we show that strong duality holds. We use Slater's condition to prove this. This proof also shows that there exist optimal Lagrange dual variables. However, this does not show that there is an optimal solution  $f^{\star}$  to the primal convex program. In fact, one can construct examples involving natural loss functions and densities where strong duality holds, but the optimal solution does not exist. In order to prove the existence of an optimal solution  $f^{\star}$  to the instance  $\mathcal{I}$ , we show that the optimal Lagrange dual variable  $\gamma^{\star}$  (corresponding to the entropy constraint in PrimalOpt) is strictly positive. This proof requires several technical steps.
We first study properties of the function  $I_{f_{\mathcal{D}},\alpha,\ell}(x)$  in Section C.4, which is the expected loss if the estimated value is  $x$ . It is easy to verify that the objective function of PrimalOpt,  $\operatorname{Err}_{\ell,\alpha}(f_{\mathcal{D}},f)$ , is the integral of  $I_{f_{\mathcal{D}},\alpha,\ell}(x)f(x)$  over  $\Omega$ . Therefore, the optimal solution  $f^{\star}(x)$  would place a higher probability mass on regions where  $I_{f_{\mathcal{D}},\alpha,\ell}(x)$  is small. Under natural monotonicity conditions on the loss function (assumption (A1)), we show that  $I_{f_{\mathcal{D}},\alpha,\ell}(x)$  can be expressed as a sum of an increasing and a decreasing function. This decomposition shows that for natural loss functions, e.g., those which or concave or convex in each of the coordinates, the function  $I_{f_{\mathcal{D}},\alpha,\ell}(x)$  is unimodal; we state this as an assumption (A5) to take care of general loss function settings.
In Section C.5, we show that  $\gamma^{\star} > 0$ . This proof hinges on the following result: For every instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  with finite optimal value, there is a positive constant  $\eta$  such that the optimal value is at least  $I_{f_{\mathcal{D}}, \ell, \alpha}(x^{\star}) + \eta$ , where  $x^{\star}$  is the minimizer of  $I_{f_{\mathcal{D}}, \ell, \alpha}(x)$ . This requires unimodality of the function  $I_{f_{\mathcal{D}}, \ell, \alpha}(x)$  and bounded mean (and median) of  $f_{\mathcal{D}}$  (assumption A4). Somewhat surprisingly, this result also requires that the loss function  $\ell(v, x)$  grows at least logarithmically as a function of  $|v - x|$  (assumption (A2)). Without this logarithmic growth, an optimal solution need not exist: it may happen that there are near-optimal solutions that place vanishingly small probability mass outside the global minimum  $x^{\star}$  of  $I_{f_{\mathcal{D}}, \ell, \alpha}(x)$ . Thus, even though we have an entropy constraint, there will be a sequence of solutions, whose error converges to the optimal value, that converges to a delta function.
In Section C.6, we use the positivity of  $\gamma^{\star}$  to explicitly write down an expression for the optimal primal solution. We use strict convexity of the feasible region for PrimalOpt to show that the optimal solution is unique. Finally, in Section C.7, we show that if  $f_{\mathcal{D}}$  has bounded entropy, then there is a suitable choice for the loss function  $\ell$ , and parameters  $\alpha$  and  $\tau$ , such that we recover  $f_{\mathcal{D}}$  as the unique optimal solution in the resulting instance.
C.1 The primal optimization problem and assumptions
We re-write the optimization problem OptProg here. An instance  $\mathcal{I}$  of this problem is given by a tuple  $\mathcal{I} \coloneqq (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$ , where  $\Omega$  is a closed interval in  $\mathbb{R}$ ,  $f_{\mathcal{D}}$  is the density of the true utility that is Lebesgue measurable,  $\ell$  is the loss function,  $\tau$  is the resource-information parameter and  $\alpha \geq 1$  is the risk-averseness parameter. Recall that  $\mu$  is the Lebesgue measure on  $\mathbb{R}$ . Note that  $\Omega$  can be of the form  $(-\infty, \infty)$ , or  $[a, \infty)$  for a real  $a$ ,  $(- \infty, b]$  for a real  $b$ , or  $[a, b]$  for real  $a, b, a < b$ .
The primal problem for  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  is as follows. Here,  $\operatorname{Dom}$  denotes the set  $\{f: \Omega \to \mathbb{R}_{\geq 0}\}$ .
$$
\min  _ {f \in \operatorname {D o m}} \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) := \int_ {\Omega} \left[ \int_ {\Omega} \ell_ {\alpha} (x, v) f (x) d \mu (x) \right] f _ {\mathcal {D}} (v) d \mu (v) \quad (\text {P r i m a l O p t})
$$
such that  $\mathsf{Ent}(f)\coloneqq -\int_{\Omega}f(x)\ln f(x)d\mu (x)\geq \tau$  (6)
$$
\int_ {\Omega} f (x) d \mu (x) = 1. \tag {7}
$$
We state the assumptions and justify each of these:
A0 (Strict feasibility of the instance) The interval  $\Omega$  has length strictly larger than  $e^{\tau}$ . This is satisfied trivially if  $\Omega$  is infinite.
Remark: In order to ensure that the set of feasible solutions to an instance of PrimalOpt is non-empty, we require that  $\Omega$  has length at least  $e^{\tau}$ ; otherwise even the uniform distribution on  $\Omega$  shall have entropy less than  $\tau$ . The strict inequality is needed to ensure strong duality (Slater's condition).
A1 (Monotonicity of the loss function) We assume that the loss function  $\ell : \Omega \times \Omega \to \mathbb{R}$  is continuous and  $\ell(x, x) = 0$  for all  $x \in \Omega$ . We consider two types of loss functions, TYPEP and TYPEN. The TYPEP loss functions have the following monotonicity property: For any fixed  $v \in \Omega$ ,  $\ell(v, x)$  strictly increases as  $|v - x|$  increases. It follows that  $\ell(v, x) \geq 0$  with equality if and only if  $v = x$ . The TYPEN loss functions have the following property: For a fixed  $v \in \Omega$ ,  $\ell(v, x)$  is a strictly increasing function of  $x$ . It follows that  $\ell(v, x) \geq 0$  if  $x \geq v$  and  $\ell(v, x) < 0$  if  $x < v$ . For example,  $\ell(x, v) = (x - v)^2$ ,  $|x - v|$  are of TYPEP, whereas  $\ell(x, v) = \ln x - \ln v$ ,  $(x - v)$  are of TYPEN.
Remark: These are natural monotonicity properties. A TYPEP loss function ensures that the optimal density to an instance  $\mathcal{I}$  of PrimalOpt assigns higher values to points where the input density  $f_{\mathcal{D}}$  is more concentrated. A TYPEN loss function ensures that the optimal density does not place high probability mass at points that are much larger than the mean of  $f_{\mathcal{D}}$ .
A2 (Growth rate of the loss function) We would like to assume that the loss function  $\ell(x, v)$  grows reasonably rapidly as  $|x - v|$  increases. It turns out that the "right" growth rate would be at least logarithmic. For instance, we could require that  $|\ell(x, v)|$  is  $\Omega(|\ln(x - v)|)$ . However, unless we assume some form of the triangle inequality, such a lower bound would not imply a lower bound on  $|\ell(x_2, v) - \ell(x_1, v)|$  for any  $x_2, x_1, v \in \Omega$ . Thus, we make the following assumption: There is a constant  $C$ , such that for all  $x_2, x_1, v \in \Omega$ , with  $|x_1 - x_2| \geq C$ ,  $v \notin (x_1, x_2)$ ,
$$
\left| \ell \left(x _ {2}, v\right) - \ell \left(x _ {1}, v\right) \right| \geq \ln \left| x _ {2} - x _ {1} \right| - \theta_ {x _ {1}},
$$
where  $\theta_{x_1}$  is a value which depends on  $x_1$  only. For example, when  $\ell(x, v) = (x - v)$ , the above is satisfied with  $\theta_x = 0$ ,  $C = 1$ ; and when  $\ell(x, v) = \ln x - \ln v$ , the above property holds with  $\theta_x = \ln x$ ,  $C = 0$ .
Remark: This is a subtle condition, and is needed to ensure that an optimal solution exists. Consider for example, an instance  $\mathcal{I}$  with  $\Omega = [0,\infty)$ ,  $f_{\mathcal{D}}$  being the exponential density,  $\ell(x,v) = \ln \ln (x + 2) - \ln \ln (v + 2)$  (the “+2” factor is to ensure that  $\ln (x + 2)$  does not become negative). The parameters  $\alpha, \tau$  can be arbitrary. Let  $\varepsilon > 0$  be an arbitrarily small constant. Now, consider a solution that places  $1 - \varepsilon$  probability mass at  $x = 0$  and spreads the remaining  $\varepsilon$  probability mass uniformly over an interval of length  $e^{\tau/\varepsilon}$  to achieve entropy  $\tau$ . Since the loss function grows slowly, the expected loss of this solution decreases with decreasing  $\varepsilon$ . Thus, even though strong duality holds in this instance, an optimal solution does not exist. In fact, we have a sequence of solutions, with error converging to the optimal value of  $\mathcal{I}$ , converging to the delta-function at 0.
A3 (Integrability of the loss function) We assume that the function  $\ell(x, v) f_{\mathcal{D}}(v)$  is in  $L^1(\mu)$  for every  $x \in \Omega$ . In other words, for each  $x \in \Omega$ ,
$$
\int_ {\Omega} | \ell (x, v) | f _ {\mathcal {D}} (v) d \mu (v) <   \infty .
$$
Remark: This is needed in order to carry out basic operations like the swapping of integrals (Fubini's Theorem) on the objective function. Further, this ensures that the objective function does not become infinite under reasonable conditions.
A4 (Bounded mean and half-radius) We assume that the true utility density  $f_{\mathcal{D}}$  has a bounded mean  $m$ . Moreover, we assume that there is a finite value  $R$  such that at least half of the probability mass of  $f_{\mathcal{D}}$  lies in the range  $[m - R, m + R]$ , i.e.,
$$
\int_ {\Omega \cap [ m - R, m + R ]} f _ {\mathcal {D}} (v) d \mu (v) \geq 1 / 2.
$$
Remark: This condition ensures that  $f_{\mathcal{D}}$  decays at a reasonable rate (though it is much milder than requiring bounded variance).
A5 (Unique global minimum for estimated loss function) Let  $I_{f_{\mathcal{D}},\ell ,\alpha}(x)$  denote the expected loss when the estimated value is  $x$ , i.e.,
$$
I _ {f _ {\mathcal {D}}, \ell , \alpha} (x) := \int_ {\Omega} \ell_ {\alpha} (x, v) f _ {\mathcal {D}} (v) d \mu (v).
$$
We assume that this function has a unique minimum on  $\Omega$ . Moreover, if  $x^{\star}$  denotes  $\operatorname{argmin}_{x \in \Omega} I_{f_{\mathcal{D}}, \ell, \alpha}(x)$ , we also assume that for all other local minima  $x$  of this function,  $I_{f_{\mathcal{D}}, \ell, \alpha}(x)$  is larger than  $I_{f_{\mathcal{D}}, \ell, \alpha}(x^{\star})$  by a fixed constant. We state this condition formally as follows: Given any  $\delta > 0$ , there is an  $\varepsilon_{\delta} > 0$  such that for all  $x$  satisfying  $|x - x^{\star}| > \delta$ , we have  $I_{f_{\mathcal{D}}, \ell, \alpha}(x) \geq I_{f_{\mathcal{D}}, \ell, \alpha}(x^{\star}) + \varepsilon_{\delta}$ .
Remark: This condition is needed to ensure the uniqueness of the optimal solution. The optimal density for an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  tries to place higher probability mass in regions where  $I_{f_{\mathcal{D}}, \ell, \alpha}(x)$  is low. Therefore, a unique global minimum (and well-separatedness from other local minima) is needed to ensure that the optimal solution is unique. When the loss function is TYPEN, this condition is always satisfied (unless the optimal value is  $-\infty$ ). For a TYPEP loss function, this condition is satisfied if the loss function  $\ell(x, v)$  is concave or convex in  $x$  (for each fixed value of  $v$ ), e.g., when  $\ell(x, v) = (x - v)^2, |x - v|$ ,  $|\ln x - \ln v|$ .
The following is the formal version of Theorem 3.1.
Theorem C.1 (Characterization of optimal density). Consider an instance  $\mathcal{I}$  of the optimization problem PrimalOpt defined on a closed interval  $\Omega \subseteq \mathbb{R}$ . Let  $\ell, \alpha, \tau, f_{\mathcal{D}}$  denote the loss function, risk-averseness parameter, resource-information parameter, and the density of the true utility respectively in  $\mathcal{I}$ . If assumptions (A0)-(A5) are satisfied, then there is a unique solution  $f^{\star}$  to the instance  $\mathcal{I}$ . This solution satisfies the following condition:
$$
f ^ {\star} (x) \propto \exp \left(- \frac {I _ {f _ {\mathcal {D}} , \ell , \alpha} (x)}{\gamma^ {\star}}\right), \tag {8}
$$
where  $\gamma^{\star}$  is the Lagrange variable corresponding to the entropy constraint (6) and is strictly positive. Moreover,  $\mathsf{Ent}(f^{\star}) = \tau$ .
C.2 Dual formulation and weak duality
Let  $\gamma \geq 0$  and  $\phi \in \mathbb{R}$  denote the Lagrange variables for the constraints (6) and (7) respectively. Then, the Lagrangian is
$$
L (f, \gamma , \phi) := \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) + \gamma (\tau - \operatorname {E n t} (f)) + \phi \left(\int_ {\Omega} f (x) d \mu (x) - 1\right).
$$
Given  $\gamma \geq 0, \phi$ , define
$$
g (\gamma , \phi) := \inf  _ {f \in \operatorname {D o m}} L (f, \gamma , \phi). \tag {9}
$$
The dual problem is
$$
\max  _ {\gamma \in \mathbb {R} _ {\geq 0}, \phi \in \mathbb {R}} g (\gamma , \phi). \quad \text {(D u a l O p t)}
$$
We first show weak duality.
Theorem C.2 (Weak duality). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of PrimalOpt satisfying assumption (A3). Let  $g(\gamma, \phi)$  be as defined in (9). Then,
$$
\max_{\gamma \in \mathbb{R}_{\geq 0},\phi \in \mathbb{R}}g(\gamma ,\phi)\leq \min_{f:f\text{feasible for PrimalOpt}}\operatorname{Err}_{\ell ,\alpha}(f_{\mathcal{D}},f).
$$
Proof. We assume that the primal problem has a feasible solution, otherwise the desired inequality follows trivially. Let  $f$  be a feasible solution to PrimalOpt and let  $\gamma, \phi$  be real values with  $\gamma \geq 0$ . Then,
$$
\begin{array}{l} g (\gamma , \phi) \leq L (f, \gamma , \phi) \\ = \operatorname {E r r} _ {\ell , \alpha} (f _ {\mathcal {D}}, f) + \gamma (\tau - \operatorname {E n t} (f)) + \phi \left(\int_ {\Omega} f (x) d \mu (x)\right) \\ \leq \operatorname {E r r} _ {\ell , \alpha} (f _ {\mathcal {D}}, f), \\ \end{array}
$$
where the first inequality follows from the definition of  $g(\gamma, \phi)$ , and the last inequality follows from the fact that  $f$  is a feasible solution to the instance  $\mathcal{I}$  of PrimalOpt.
C.3 Strong duality
Theorem C.3 (Strong duality). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of PrimalOpt satisfying assumptions (A0), (A1) and (A3). Let  $f^{\star}$  and  $(\gamma^{\star}, \phi^{\star})$  be the optimal solutions to PrimalOpt and DualOpt respectively. Then  $g(\gamma^{\star}, \phi^{\star}) = \mathrm{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f^{\star})$ .
Proof. We first observe that there is a feasible solution to the instance  $\mathcal{I}$ . This is so because, by assumption (A0),  $\Omega$  must contain an interval  $I$  of length at least  $e^{\tau}$ . Now, we define  $f$  as the uniform distribution on  $I$ . Then  $\mathrm{Ent}(f) = \tau$ , and hence,  $f$  is a feasible solution to the instance  $\mathcal{I}$ . We now argue that  $\mathrm{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f)$  is finite.
Claim C.4. Consider the function  $f$  and the interval  $I$  defined above. Then,  $\operatorname{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f) < \infty$ .
Proof. Let  $a, b$  denote the left and the right end-points of the interval  $I$  respectively. For any  $x \in I$  and  $v \in \Omega$ , we claim that  $\ell_{\alpha}(x, v) \leq \max(\ell_{\alpha}(a, v), \ell_{\alpha}(b, v))$ . First, consider the case when the loss function is TYPEP. If  $v$  is at least  $x$ , then  $\ell(x, v) \leq \ell(a, v)$ ; otherwise  $\ell(x, v) \leq \ell(b, v)$ . If the loss function is TYPEN, then we know that it is an increasing function of  $x$ , and therefore,  $\ell(x, v) \leq \ell(b, v)$ . Thus, we see that for any  $x \in I$ ,  $v \in \Omega$ ,  $\ell_{\alpha}(x, v) \leq \max(\ell_{\alpha}(a, v), \ell_{\alpha}(b, v)) \leq |\ell_{\alpha}(a, v)| + |\ell_{\alpha}(b, v)|$ . Therefore (recall that  $f$  is the uniform distribution on  $I$ ),
$$
\operatorname {E r r} _ {\ell , \alpha} (f _ {\mathcal {D}}, f) \leq \frac {1}{| I |} \int_ {I} | \ell (a, v) | f _ {\mathcal {D}} (v) d \mu (v) + \frac {1}{| I |} \int_ {I} | \ell (b, v) | f _ {\mathcal {D}} (v) d \mu (v) <   \infty ,
$$
where the last inequality follows from assumption (A3).
Thus, we see that the optimal value of PrimalOpt is either finite or  $-\infty$ . If it is  $-\infty$ , then weak duality implies that DualOpt has optimal value  $-\infty$  as well. Thus, we have shown strong duality in this case.
For the rest of the proof, assume that the optimal value,  $p^{\star}$ , of PrimalOpt is finite. We show strong duality using Slater's condition. Towards this, we define two sets  $\mathcal{A}$  and  $\mathcal{B}$ , both of which are contained in  $\mathbb{R}^3$ . Define
$$
\mathcal {A} := \left\{(u, v, t): u \geq \tau - \operatorname {E n t} (f), v = \int_ {\Omega} f (x) d \mu (x) - 1, t \geq \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right), \text {f o r s o m e} f \in \operatorname {D o m} \right\},
$$
$$
\mathcal {B} := \left\{(0, 0, q): q <   p ^ {\star} \right\}.
$$
It is easy to see that  $\mathcal{B}$  is convex. We show that  $\mathcal{A}$  is also convex.
Claim C.5. The set  $\mathcal{A}$  as defined above is a convex subset of  $\mathbb{R}^3$ .
Proof. The proof follows from the fact that  $\operatorname{Ent}(f)$  is a concave function of  $f$  and  $\operatorname{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f)$  is linear in  $f$ . Formally, suppose  $(u_1, v_1, t_1), (u_2, v_2, t_2) \in \mathcal{A}$  and  $\lambda \in [0, 1]$ . We need to show that  $(u, v, t) := \lambda(u_1, v_1, t_1) + (1 - \lambda)(u_2, v_2, t_2) \in \mathcal{A}$ . By the definition of the set  $\mathcal{A}$ , there exist  $f_1, f_2 \in \operatorname{Dom}$  such that
$$
u _ {i} \geq \tau - \operatorname {E n t} (f _ {i}), v _ {i} = \int_ {\Omega} f _ {i} (x) d \mu (x) - 1, t _ {i} \geq \operatorname {E r r} _ {\ell , \alpha} (f _ {\mathcal {D}}, f _ {i}), i \in \{1, 2 \}.
$$
Consider  $f = \lambda f_{1} + (1 - \lambda)f_{2}$ . Then  $f$  is also a density and is in Dom. Further,
$$
\operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) = \lambda_ {1} \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f _ {1}\right) + \left(1 - \lambda_ {1}\right) \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f _ {2}\right) = \lambda t _ {1} + (1 - \lambda) t _ {2} = t.
$$
Similarly, since  $-x\ln x$  is concave,
$$
\begin{array}{l} \operatorname {E n t} (f) = - \int_ {\Omega} f (x) \ln f (x) d \mu (x) \\ \geq - \lambda \int_ {\Omega} f _ {1} (x) \ln f _ {1} (x) d \mu (x) - (1 - \lambda) \int_ {\Omega} f _ {2} (x) \ln f _ {2} (x) d \mu (x) \geq \tau . \\ \end{array}
$$
Thus,  $(u,v,t)\in \mathcal{A}$
We argue that  $\mathcal{A}$  and  $\mathcal{B}$  are disjoint. Indeed, otherwise, there is an  $f\in \mathrm{Dom}$  such that  $f$  is a density,  $\operatorname {Ent}(f)\geq \tau$  and  $\operatorname{Err}_{\ell ,\alpha}(f_{\mathcal{D}},f) < p^{\star}$ , a contradiction. By the hyperplane separation theorem [34, 143], there is a hyperplane  $(\tilde{\gamma},\tilde{\phi},\tilde{\nu})^{\top}(x_1,x_2,x_3) = c$  such that  $\mathcal{A}$  and  $\mathcal{B}$  lie on different sides of this hyperplane. In other words, for every  $(x_{1},x_{2},x_{3})\in \mathcal{A}$
$$
\tilde {\gamma} x _ {1} + \tilde {\phi} x _ {2} + \tilde {\nu} x _ {3} \geq c,
$$
and for every  $(x_{1},x_{2},x_{3})\in \mathcal{B}$
$$
\tilde {\gamma} x _ {1} + \tilde {\phi} x _ {2} + \tilde {\nu} x _ {3} \leq c.
$$
Therefore, for each  $f \in \mathrm{Dom}$ ,
$$
\tilde {\gamma} (\tau - \operatorname {E n t} (f)) + \tilde {\phi} \left(\int_ {\Omega} f (x) d \mu (x) - 1\right) + \tilde {\nu} \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) \geq c, \tag {10}
$$
$$
\tilde {\nu} p ^ {\star} \leq c. \tag {11}
$$
Claim C.6.  $\tilde{\gamma}$  and  $\tilde{\nu}$  are non-negative.
Proof. Suppose, for the sake of contradiction, that  $\tilde{\gamma} < 0$ . Consider  $f \in \mathrm{Dom}$  as given by Claim C.4. Choose values  $(x_{1}, x_{2}, x_{3})$  as follows:  $x_{1}$  is a large enough value greater than  $\tau - \operatorname{Ent}(f)$ ,  $x_{2} = 0$ ,  $x_{3} = \operatorname{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f)$ . Claim C.4 shows that  $x_{3}$  is finite. Since  $(x_{1}, x_{2}, x_{3}) \in \mathcal{A}$ ,  $\tilde{\gamma} x_{1} + \tilde{\nu} x_{3} \geq \alpha$ . Since  $\tilde{\gamma} < 0$ , we can choose  $x_{1}$  large enough to make  $\tilde{\gamma} x_{1} + \tilde{\nu} x_{3}$  go below  $\alpha$ , which is a contradiction. Similarly, we can show that  $\tilde{\nu} \geq 0$ .
Thus, two cases arise (i)  $\tilde{\nu} > 0$ , or (ii)  $\tilde{\nu} = 0$ . First, consider the case when  $\tilde{\nu} > 0$ . Define  $\gamma^{\star} = \tilde{\gamma} / \tilde{\nu}$ ,  $\phi^{\star} = \tilde{\phi} / \tilde{\nu}$ . Inequalities (10) and (11) show that for all  $f \in \mathrm{Dom}$ ,
$$
\gamma^ {\star} (\tau - \operatorname {E n t} (f)) + \phi^ {\star} \left(\int_ {\Omega} f (x) d \mu (x) - 1\right) + \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) \geq p ^ {\star},
$$
i.e.,  $g(\gamma^{\star}, \phi^{\star}) \geq p^{\star}$ . It follows from weak duality (Theorem C.2) that  $g(\gamma^{\star}, \phi^{\star}) = p^{\star}$  and the dual optimum value is equal to  $p^{\star}$ .
Now consider the case when  $\tilde{\nu} = 0$ . Again, inequalities (10) and (11) show that for all  $f \in \mathrm{Dom}$ ,
$$
\tilde {\gamma} (\tau - \operatorname {E n t} (f)) + \tilde {\phi} \left(\int_ {\Omega} f (x) d \mu (x) - 1\right) \geq 0.
$$
Next, we observe that there is an  $f_0 \in \mathsf{Dom}$  such that  $f_0$  is a density and  $\mathsf{Ent}(f_0) > \tau$ . Indeed, let  $f_0$  be the uniform distribution over an interval of length strictly larger than  $e^{\tau}$  (such an interval exists by assumption (A0)). Substitution  $f = f_0$  in the above inequality, and assuming  $\tilde{\gamma} > 0$ , the l.h.s. of the above inequality becomes strictly less than 0, which is a contradiction. Therefore, it must be the case that  $\tilde{\gamma} = 0$ . Hence, we see that for all  $f \in \mathsf{Dom}$ ,
$$
\tilde {\phi} \left(\int_ {\Omega} f (x) d \mu (x) - 1\right) \geq 0.
$$
Since all the three quantities  $\tilde{\gamma},\tilde{\phi},\tilde{\nu}$  cannot be 0, it must be the case that  $\phi^{\star}\neq 0$ . But for any density  $f_0\in \mathrm{Dom}$ , by suitably scaling it by a positive real, we can make the quantity  $\left(\int_{\Omega}f(x)d\mu (x) - 1\right)$  strictly larger than or strictly smaller than 0, which is again a contradiction. Hence, we have concluded that  $\tilde{\nu}$  cannot be 0. This completes the proof of strong duality.