SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We claim that, for the values of dual variables  $\gamma \coloneqq 1$  and  $\phi \coloneqq \operatorname{Ent}(f_{\mathcal{D}}) - 1$ ,  $g(\gamma, \phi) = L(f_{\mathcal{D}}, \gamma, \phi)$ . Towards this, we show that  $f_{\mathcal{D}}$  minimizes  $L(f, \gamma, \phi)$  over all  $f \in \mathrm{Dom}$ . We first show that  $f_{\mathcal{D}}$  satisfies the condition (12) with  $\gamma^{\star} = \gamma$ ,  $\phi^{\star} = \phi$ ,  $f^{\star} = f_{\mathcal{D}}$ ; it shall then follow from exactly
Lemma C.18.  $\mathcal{F}$  is strictly convex.
the same arguments as in Theorem C.16 that  $f_{\mathcal{D}} = \operatorname*{argmin}_{f\in \mathrm{Dom}}L(f,\gamma ,\phi)$ . Now, we check that  $f_{\mathcal{D}}$  satisfies condition (12) (recall that the loss function  $\ell (x,v)\coloneqq \ln f_{\mathcal{D}}(v) - \ln f_{\mathcal{D}}(x)$ ):
$$
\begin{array}{l} I (x) + \gamma (1 + \ln f _ {\mathcal {D}} (x)) + \phi = \int_ {\Omega} \ell (x, v) f _ {\mathcal {D}} (v) d \mu (v) + \gamma (1 + \ln f _ {\mathcal {D}} (x)) + \phi \\ = \int_ {\Omega} f _ {\mathcal {D}} (v) \ln f _ {\mathcal {D}} (v) d \mu (v) - \ln f _ {\mathcal {D}} (x) \int_ {\Omega} f _ {\mathcal {D}} (v) d \mu (v) + (1 + \ln f _ {\mathcal {D}} (x)) + \operatorname {E n t} \left(f _ {\mathcal {D}}\right) - 1 \\ = - \operatorname {E n t} \left(f _ {\mathcal {D}}\right) - \ln f _ {\mathcal {D}} (x) + (1 + \ln f _ {\mathcal {D}} (x)) + \operatorname {E n t} \left(f _ {\mathcal {D}}\right) - 1 = 0. \\ \end{array}
$$
This proves the desired claim.
$$
g (\gamma , \phi) = L (f _ {\mathcal {D}}, \gamma , \phi) = \operatorname {E r r} _ {\ell , \alpha} (f _ {\mathcal {D}}, f _ {\mathcal {D}}).
$$
Thus,  $g(\gamma, \phi)$  is equal to the objective function value of PrimalOpt at  $f = f_{\mathcal{D}}$ . Hence,  $f_{\mathcal{D}}$  is an optimal solution to PrimalOpt for  $\mathcal{I}$ . In order to prove uniqueness, note that the above argument also yields an optimal solution  $(\gamma, \phi)$  to DualOpt. Since  $\gamma > 0$ , complementary slackness conditions imply that any optimal solution must have entropy exactly equal to  $\tau$ . Now, arguing as in Theorem C.19, we see that there is a unique optimal solution.
D Derivation of output density for a single individual
In this section, we consider the setting when a single individual with true utility  $u$  is being evaluated. In this case, the input density  $f_{\mathcal{D}}(x)$  is specified by the Dirac-delta function centered at  $v$ , denoted  $\delta_v(x)$ . Using Theorem 3.1, we can characterize the output density as follows:
Lemma D.1. Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of (OptProg) where  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{D}} = \delta_v(x)$  for some real  $v$ ,  $\ell(x, u) = (x - u)^2$ , and  $\alpha$  and  $\tau$  are arbitrary real parameters. Then the optimal density is given by
$$
f ^ {\star} (x) = \left\{ \begin{array}{l l} K e ^ {- \frac {(x - v) ^ {2}}{\gamma^ {\star}}} & \text {i f} x \leq v \\ K e ^ {- \frac {\alpha (x - v) ^ {2}}{\gamma^ {\star}}} & \text {o t h e r w i s e} \end{array} \right.,
$$
where  $K$  is the normalization constant and  $\gamma^{\star}$  is the optimal dual variable for the entropy constraint. The mean of this density is equal to  $v - \sqrt{\frac{\gamma^{\star}}{\pi}}\cdot \frac{\sqrt{\alpha} - 1}{\sqrt{\alpha}}$ .
Proof. We first evaluate the integral  $I(x)$  as follows:
$$
I (x) := \int_ {\Omega} f _ {\mathcal {D}} (u) \ell_ {\alpha} (x, u) d v = \alpha \int_ {- \infty} ^ {x} \delta_ {v} (u) (x - u) ^ {2} + \int_ {x} ^ {\infty} \delta_ {v} (u) (x - u) ^ {2}.
$$
When  $x \leq v$ , the first integral on the r.h.s. is 0, and hence, the above integral is equal to  $(x - v)^2$ . Similarly, if  $x > v$ , the above integral is equal to  $\alpha (x - v)^2$ . The expression for  $f^{\star}(x)$  in the statement of the Lemma now follows from (12).
A routine calculation shows that the normalization constant  $K$  is equal to  $\frac{2\sqrt{\alpha}}{\sqrt{\pi\gamma^*(1 + \sqrt{\alpha})}}$ . Now the mean of this density turns out to be
$$
v - \sqrt {\frac {\gamma^ {\star}}{\pi}} \cdot \frac {\sqrt {\alpha} - 1}{\sqrt {\alpha}}.
$$
E Connection to the Gibbs equation
Theorem E.1 (Gibbs equation). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of the optimization problem PrimalOpt that satisfies the assumptions of Theorem C.1. Then, the following holds:
$$
- \gamma^ {\star} \ln Z ^ {\star} = \operatorname {E r r} _ {\ell , \alpha} \left(f ^ {\star}, f _ {\mathcal {D}}\right) + \gamma^ {\star} \operatorname {E n t} \left(f ^ {\star}\right). \tag {13}
$$
Here,  $f^{\star}(x) \propto e^{-\frac{I(x)}{\gamma^{\star}}}$  is the solution to PrimalOpt,  $Z^{\star} \coloneqq \int e^{-\frac{I(x)}{\gamma^{\star}}} d\mu(x)$ , and  $\gamma^{\star} > 0$  is the solution to DualOpt. Recall that  $I(x) \coloneqq \int_{\Omega} \ell_{\alpha}(x, v) f_{\mathcal{D}}(v) d\mu(v)$ .
Proof. Theorem C.1 implies that there exists  $f^{\star}(x)$  and  $\gamma^{\star} > 0$  such that
$$
f ^ {\star} (x) \propto e ^ {- \frac {I (x)}{\gamma^ {\star}}}.
$$
Thus, if we let  $Z^{\star} \coloneqq \int e^{-\frac{I(x)}{\gamma^{\star}}} d\mu(x)$ , then
$$
f ^ {\star} (x) = \frac {e ^ {- \frac {I (x)}{\gamma^ {\star}}}}{Z ^ {\star}}.
$$
Thus, from the optimality condition (12), we obtain that there is a  $\phi^{\star}$  such that
$$
I (x) + \gamma^ {\star} (1 + \ln f ^ {\star} (x)) + \phi^ {\star} = 0.
$$
Since  $\gamma^{\star} > 0$ , we can divide by it to obtain  $Z^{\star} = e^{1 + \frac{\phi^{\star}}{\gamma^{\star}}}$ . We integrate the above with respect to the density  $f^{\star}(x)$  to get
$$
\operatorname {E r r} _ {\ell , \alpha} \left(f ^ {\star}, f _ {\mathcal {D}}\right) + \gamma^ {\star} - \gamma^ {\star} \tau + \phi^ {\star} = 0.
$$
Thus, we obtain:
$$
\gamma^ {\star} \tau = \gamma^ {\star} \operatorname {E n t} \left(f ^ {\star}\right) = \operatorname {E r r} _ {\ell , \alpha} \left(f ^ {\star}, f _ {\mathcal {D}}\right) + \gamma^ {\star} \ln Z ^ {\star}. \tag {14}
$$
Rearranging this equation we obtain the theorem.
In analogy with the Gibbs equation in statistical physics [94],  $Z^{\star}$  can be viewed as the partition function corresponding to the energy function  $I(x), \gamma^{\star}$  corresponds to the temperature,  $-\gamma^{\star} \ln Z^{\star}$  corresponds to the free energy and  $\mathrm{Err}(f^{\star}, f_{\mathcal{D}})$  is the internal energy.
It follows from the theorem that we can write  $f^{\star}(x)$  as
$$
f ^ {\star} (x) = e ^ {- \tau} \exp \left(- \frac {I (x) - \operatorname {E r r} _ {\ell , \alpha} \left(f ^ {\star} , f _ {\mathcal {D}}\right)}{\gamma^ {\star}}\right). \tag {15}
$$
F Effect of changing the resource-information parameter  $\tau$
F.1 Effect of decreasing  $\tau$
Theorem F.1 (Effect of decreasing  $\tau$ ). Fix  $\alpha \geq 1$ , a density  $f_{\mathcal{D}}$ , and loss function  $\ell$ . Assume that the function  $I_{f_{\mathcal{D}},\ell,\alpha}(x)$  satisfies condition (A5), and let  $x^{\star} \coloneqq \operatorname*{argmin}_{x \in \Omega} I_{f_{\mathcal{D}},\ell,\alpha}(x)$ . Given a  $\tau$ , let  $f_{\tau}^{\star}$  denote the optimal solution to PrimalOpt. For every  $\delta > 0$ , there exists a value  $T_{\delta}$  such that when  $\tau \leq T_{\delta}$ , the solution  $f_{\tau}^{\star}$  has the following property: For every  $x \in \Omega$ ,  $|x - x^{\star}| \geq \delta$ , we have
$$
\frac {f _ {\tau} ^ {\star} (x)}{f _ {\tau} ^ {\star} (x ^ {\star})} \leq \delta .
$$
In other words, the density outside an interval of length  $\delta$  around  $x^{\star}$  has a much smaller value than at  $x^{\star}$ .
In order to prove the above result, we first show that the optimal dual value  $\gamma_{\tau}^{\star}$  goes to 0 as  $\tau$  goes to  $-\infty$ . Then we shall use Theorem E.1 to show that the optimal density is highly sensitive to small changes in  $I(x)$ .
Lemma F.2. Fix an interval  $\Omega \subseteq \mathbb{R}$ , parameter  $\alpha \geq 1$ , density  $f_{\mathcal{D}}$  of true utility and loss function  $\ell$ . Assume that the function  $I_{f_{\mathcal{D}},\ell,\alpha}(x)$  has a unique global minimum. Let  $\mathcal{I}_{\tau}$  denote the instance  $(\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$ . Let  $\gamma_{\tau}^{\star}$  be the optimal Lagrange variable corresponding to this instance (assuming it has a non-empty solution). Then
$$
\lim  _ {\tau \rightarrow - \infty} \gamma_ {\tau} ^ {\star} = 0.
$$
Proof. We first observe that the instance  $\mathcal{I}_{\tau}$  will always have a feasible solution for small enough  $\tau$ . Indeed, when  $e^{\tau}$  is less than the length of  $\Omega$ , there is always a feasible solution to  $\mathcal{I}_{\tau}$ . Let  $\tau_0$  be a value of  $\tau$  for which the instance  $\mathcal{I}_{\tau}$  has a feasible solution. For sake of brevity, let  $I(x)$  denote  $I_{f_{\mathcal{D}},\ell ,\alpha}(x)$ , and let  $x^{\star} := \operatorname*{argmin}_{x\in \Omega}I(x)$ . We first argue that  $\mathrm{Err}(f_{\mathcal{D}},f_{\tau}^{\star})$  remains in a bounded range.
Claim F.3. Consider a value of  $\tau \leq \tau_0$ . Let  $f_{\tau}^{\star}$  be the optimal solution to the instance  $\mathcal{I}_{\tau}$ . Then
$$
I \left(x ^ {\star}\right) \leq \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f _ {\tau} ^ {\star}\right) \leq \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f _ {\tau_ {0}} ^ {\star}\right).
$$
Proof. The first inequality follows from the fact that
$$
\operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f _ {\tau} ^ {\star}\right) = \int_ {\Omega} I (x) f _ {\tau} ^ {\star} (x) d \mu (x) \geq I \left(x ^ {\star}\right).
$$
The second inequality follows from the fact that the solution  $f_{\tau_0}^{\star}$  is also a feasible solution for the instance  $\mathcal{I}_{\tau}$ .
Suppose for the sake of contradiction,
$$
\lim  _ {\tau \rightarrow - \infty} \gamma_ {\tau} ^ {\star} > 0.
$$
In other words,  $\theta_0\coloneqq \lim \sup_{\tau \to -\infty}\gamma_\tau^\star >0$ . It follows that there is an infinite sequence  $A\coloneqq (\tau_0>\tau_1 > \dots)$  going to  $-\infty$  such that  $\gamma_{\tau_i}^\star \geq \theta_0 / 2$  for all  $\tau_{i}\in A$ .
Consider an interval of finite but non-zero length in  $\Omega$  - let  $U = [s,t]$  be such an interval. Since  $U$  is closed and  $I(x)$  is a continuous function, there are finite values  $a_0,b_0$  such that  $a_0\leq I(x)\leq b_0$  for all  $x\in U$ . Thus we get:
Claim F.4. There is a positive real  $\eta$ , such that
$$
\exp \left(\frac {\operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}} , f _ {\tau} ^ {\star}\right) - I (x)}{\gamma_ {\tau} ^ {\star}}\right) \geq \eta
$$
holds for all  $x\in U,\tau \in A$
Proof. It follows from Claim F.3 and the observation above that for all  $x \in U$  and all  $\tau \leq \tau_0$ ,  $|\mathrm{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f_{\tau}^{\star}) - I(x)|$  lies in the range  $[I(x^{\star}) - b_0, \mathrm{Err}_{\ell, \alpha}(f_{\mathcal{D}}, f_{\tau}^{\star}) + a_0]$ . Since  $\gamma_{\tau} \geq \theta_0 / 2$  for all  $\tau \in A$ , the result follows.
The above claim along with Theorem E.1 shows that  $f_{\tau}^{\star}(x) \geq \eta e^{-\tau}$  for all  $x \in U, \tau \in A$ . Since  $A$  contains an infinite sequence of values going to  $-\infty$ , we can choose a value  $\tau \in A$  such that  $\eta e^{-\tau} > 1 / |U|$ . But then  $f_{\tau}^{\star}(x) > \frac{1}{|U|}$  for all  $x \in U$ , which is not possible because  $f^{\star}$  is a density. This proves the lemma.
We now prove Theorem F.1. Consider the instance  $\mathcal{I}$  as stated in this statement of this theorem. Let  $\tau_0$  be a value of the information resource parameter for which there is a density with entropy  $\tau$  in  $\Omega$  (i.e., when PrimalOpt has a feasible solution). Recall that  $x^{\star} = \mathrm{argmin}_{x\in \Omega}I(x)$ . Consider a  $\delta > 0$ . Assumption (A5) shows that there is a value  $\varepsilon_{\delta} > 0$  such that  $I(x) - I(x^{\star}) > \varepsilon_{\delta}$  for all  $x$  satisfying
$|x - x^{\star}| \geq \delta$ . Now consider an  $x$  such that  $|x - x^{\star}| \geq \delta$ . Using Theorem E.1, we see that, for all  $\tau \leq \tau_0$
$$
\frac {f _ {\tau} ^ {\star} (x)}{f _ {\tau} ^ {\star} (x ^ {\star})} = \exp \left(\frac {I (x ^ {\star}) - I (x)}{\gamma_ {\tau} ^ {\star}}\right) \leq \exp \left(- \varepsilon_ {\delta} / \gamma_ {\tau} ^ {\star}\right),
$$
where the last inequality follows from the fact that  $\gamma_{\tau}^{\star}$  is positive. Now, Lemma F.2 shows that there a value  $T_{\delta}$  such that for all  $\tau \leq T_{\delta}$ ,  $0 < \gamma_{\tau}^{\star} \leq \frac{\delta \varepsilon_{\delta}}{\ln(1 / \delta)}$ . Therefore, for all  $x$ ,  $|x - x^{\star}| \geq \delta$
This proves Theorem F.1.
F.2 Effect of increasing  $\tau$
In this section, we consider the effect of an increase in  $\tau$  on the variance and the mean of the optimal density.
Theorem F.5 (Effect of increasing  $\tau$  on variance). Consider a continuous density  $f_{\mathcal{D}}$  on  $\mathbb{R}$ , loss function  $\ell$  and information-resource parameter  $\alpha$ . For a given risk-averse parameter  $\tau$ , let  $\mathcal{I}_{\tau}$  denote the instance  $(\mathbb{R}, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of PrimalOpt. Let  $f_{\tau}^{\star}$  be the optimal solution to the instance  $\mathcal{I}_{\tau}$ . Then the variance of  $f_{\tau}^{\star}$  is at least  $\frac{1}{2\pi} e^{2\tau - 1}$ .
The proof relies on the following result.
Theorem F.6 (Gaussian maximizes entropy; Theorem 3.2 in [48]). For a continuous probability density function  $f$  on  $\mathbb{R}$  with variance  $\sigma^2$ ,  $\mathsf{Ent}(f) \leq \frac{1}{2} + \frac{1}{2} \ln \left(2\pi \sigma^2\right)$  with equality if and only if  $f$  is a Gaussian density with variance  $\sigma^2$ .
Proof of Theorem F.5. Consider the instance  $\mathcal{I}_{\tau}$  and the optimal solution  $f_{\tau}^{\star}$  for this instance. Since  $f_{\tau}^{\star}$  is a feasible solution,  $\mathrm{Ent}(f_{\tau}^{\star}) \geq \tau$ . If  $f_{\tau}^{\star}$ , has unbounded variance, then the desired result follows trivially. Hence, assume that  $f^{\star}$  has bounded variance, say  $\sigma^2$ . Now, Theorem F.6 shows that
$$
\mathsf {E n t} (f _ {\tau} ^ {\star}) \leq \frac {1}{2} + \frac {1}{2} \ln (2 \pi \sigma^ {2}).
$$
Using the fact that  $\mathrm{Ent}(f_{\tau}^{\star}) \geq \tau$ , the above inequality implies that  $\sigma^2 \geq \frac{1}{2\pi} e^{2\tau - 1}$ . This proves the desired result.
We now show that the mean of the optimal density also increases with increasing  $\tau$  when the input density  $f_{\mathcal{D}}$  is supported on  $[0,\infty)$ .
Theorem F.7 (Effect of increasing  $\tau$  on mean). Consider a continuous density  $f_{\mathcal{D}}$  on  $\Omega \coloneqq [0,\infty)$ , loss function  $\ell$  and information-resource parameter  $\alpha$ . For a given risk-averse parameter  $\tau$ , let  $\mathcal{I}_{\tau}$  denote the instance  $(\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of PrimalOpt, and  $f_{\tau}^{\star}$  denotes the optimal solution to the instance  $\mathcal{I}_{\tau}$ . Then the mean of  $f_{\tau}^{\star}$  is at least  $e^{\tau - 1}$ .
Theorem F.8 (Exponential maximizes entropy; Theorem 3.3 in [48]). For a continuous probability density function  $f$  on  $[0, \infty)$  with mean  $\lambda$ ,  $\operatorname{Ent}(f) \leq 1 + \ln(\lambda)$ .
Proof of Theorem F.7. The proof proceeds along similar lines as that of Theorem F.5. We know that  $\mathrm{Ent}(f_{\tau}^{\star}) \geq \tau$  because it is a feasible solution to the instance  $\mathcal{I}_{\tau}$ . If  $f_{\tau}^{\star}$  has unbounded mean, we are done; therefore, assume its mean, denoted  $\lambda$ , is finite. Theorem F.8 shows that  $\mathrm{Ent}(f) \leq 1 + \ln \lambda$ . Since  $\mathrm{Ent}(f_{\tau}^{\star}) \geq \tau$ , we see that  $\lambda \geq e^{\tau - 1}$ . This proves the theorem.
G Effect of changing the risk-averseness parameter  $\alpha$
Theorem G.1 (Monotonicity of  $I(x)$  with respect to  $\alpha$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of the optimization problem PrimalOpt. Then, for any  $x \in \Omega$ ,  $I_{f_{\mathcal{D}}, \ell, \alpha}(x)$  is an increasing function of  $\alpha$ .
Proof. By definition
$$
I _ {\alpha} (x) := I _ {f _ {\mathcal {D}}, \ell , \alpha} (x) = \int_ {v \leq x} \ell_ {\alpha} (x, v) f _ {\mathcal {D}} (v) d \mu (v) + \int_ {v > x} \ell (x, v) f _ {\mathcal {D}} (v) d \mu (v).
$$
Recall from (4), that for any  $x, v \in \Omega$ ,  $\ell_{\alpha}(x, v) \coloneqq \ell(x, v)$  if  $x < v$  and  $\ell_{\alpha}(x, v) \coloneqq \alpha \ell(x, v)$  when  $x \geq v$ . Since our model requires  $\ell(x, v) \geq 0$  whenever  $x \geq v$ ,  $\ell_{\alpha}(x, v) \geq \ell_{\alpha'}(x, v)$  for any  $x \geq v$  and  $\alpha \geq \alpha'$ . Thus, we have that  $I_{\alpha}(x) \geq I_{\alpha'}(x)$  for all  $x \in \Omega$ .
Theorem G.2 (Monotonicity of Err with respect to  $\alpha$ ). Consider an instance  $\mathcal{I}_{\alpha} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  of the optimization problem PrimalOpt. Suppose  $\mathcal{I}_{\alpha}$  satisfies the assumption of Theorem C.1 and let  $f_{\alpha}^{\star}$  be the optimal solution to instance  $\mathcal{I}_{\alpha}$ . Then, the function  $\mathrm{Err}_{\ell, \alpha}(f_{\alpha}^{\star}, f_{\mathcal{D}})$  is an increasing function of  $\alpha$ .
Proof. As noted in Section C, if the instance  $\mathcal{I}$  satisfies the assumptions for  $\alpha = 1$ , then the instances obtained by changing  $\alpha$  continue to satisfy the assumptions needed in Theorem C.1. Thus, we may assume the optimal solution exists for each version of  $\mathcal{I}$  where we vary  $\alpha \geq 1$ , and let  $f_{\alpha}^{\star}$  denote the optimal density. We first show that for any fixed density  $f$ ,  $\mathrm{Err}_{\ell, \alpha}(f, f_{\mathcal{D}})$  is an increasing function of  $\alpha$ . This is so because
$$
\begin{array}{l} \operatorname {E r r} _ {\ell , \alpha} (f, f _ {\mathcal {D}}) = \int_ {v \in \Omega} \left(\int_ {x <   v} \ell (x, v) f (x) d \mu (x)\right) f _ {\mathcal {D}} (v) d \mu (v) \\ + \int_ {v \in \Omega} \left(\int_ {x \geq v} \ell_ {\alpha} (x, v) f (x) d \mu (x)\right) f _ {\mathcal {D}} (v) d \mu (v) \\ \end{array}
$$
and  $\ell_{\alpha}(x,v)$  is an increasing function of  $\alpha$  for any  $x,v\in \Omega$ . Consider two values of the parameter  $\alpha$ :  $1\leq \alpha_{1} < \alpha_{2}$ . Note that the instances corresponding to both  $\alpha_{1}$  and  $\alpha_{2}$  are feasible as  $\alpha$  only appears in the objective and, hence, does not affect feasibility. Suppose for the sake of contradiction that  $\mathrm{Err}_{\ell ,\alpha_2}(f_{\alpha_2}^\star ,f_{\mathcal{D}}) < \mathrm{Err}_{\ell ,\alpha_1}(f_{\alpha_1}^\star ,f_{\mathcal{D}})$ .  $f_{\alpha_2}^\star$  satisfies  $\mathrm{Ent}(f_{\alpha_2}^\star)\geq \tau$  as it is a feasible solution of the problem instance defined by  $\alpha_{2}$  and, hence, it is also a feasible solution for the problem instance defined by  $\alpha_{1}$ . This and the definition of  $f_{\alpha_1}^\star$  imply that  $\mathrm{Err}_{\ell ,\alpha_1}(f_{\alpha_1}^\star ,f_{\mathcal{D}})\leq \mathrm{Err}_{\ell ,\alpha_1}(f_{\alpha_2}^\star ,f_{\mathcal{D}})$ . Thus, we get  $\mathrm{Err}_{\ell ,\alpha_2}(f_{\alpha_2}^\star ,f_{\mathcal{D}}) < \mathrm{Err}_{\ell ,\alpha_1}(f_{\alpha_2}^\star ,f_{\mathcal{D}})$ , which contradicts the (above observed) monotonicity of  $\mathrm{Err}_{\ell ,\alpha}(f_{\alpha_2}^\star ,f_{\mathcal{D}})$  with respect to  $\alpha$ .
H Gaussian density
The Gaussian density is defined as follows over  $\Omega = \mathbb{R}$  and has parameters  $m\in \mathbb{R}$  and  $\sigma$  ..
$$
f _ {\mathcal {G}} (x) := \frac {1}{\sqrt {2 \pi \sigma^ {2}}} e ^ {- \frac {(x - m) ^ {2}}{2 \sigma^ {2}}}, x \in \mathbb {R}.
$$
$m$  is the mean and  $\sigma^2$  is the variance. The differential entropy of  $f_{\mathcal{G}}$  is  $\frac{1}{2} + \frac{1}{2}\ln (2\pi \sigma^2)$  [148]. We consider the loss function to be  $\ell(x, v) \coloneqq (x - v)^2$ . First, we compute the expression of  $I(x)$  which we use to verify the applicability of Theorem C.1 with the above parameters.
Lemma H.1 (Expression for  $I_{\alpha}(x)$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{G}}, \ell, \alpha, \tau)$  of PrimalOpt where  $\Omega = \mathbb{R}$ ,  $\ell(x, v) := (x - v)^2$ , and  $f_{\mathcal{G}}$  is the Gaussian density with mean  $m$  and variance  $\sigma^2$ . Then
$$
I (x) = (\alpha - 1) \sigma^ {2} \left((w ^ {2} + 1) \Phi (w) + w \phi (w)\right) + \sigma^ {2} \left(w ^ {2} + 1\right),
$$
where  $w \coloneqq \frac{x - m}{\sigma}$ ,  $\phi(x) \coloneqq \frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}}$ , and  $\Phi(x) \coloneqq \int_{-\infty}^{x} \phi(x) d\mu(x)$  denotes the cumulative distribution function of the Gaussian density.
(b)
Figure 3: Mean and variance of the output density  $f$ , i.e.,  $\mathbb{E}_{x \sim f}[x]$  and  $\mathrm{Var}_{x \sim f}[x]$ , as a function of  $\tau$  when  $f_{\mathcal{D}}$  is the standard normal density and  $\alpha$  is fixed to 2.
Proof. By definition,
$$
I (x) = \alpha \int_ {- \infty} ^ {x} (x - v) ^ {2} f _ {\mathcal {G}} (v) d \mu (v) + \int_ {x} ^ {\infty} (x - v) ^ {2} f _ {\mathcal {G}} (v) d \mu (v).
$$
We first make a change of variables. Let  $w \coloneqq \frac{x - m}{\sigma}$  and  $y \coloneqq \frac{v - m}{\sigma}$ . Thus, the density of  $y$  is Gaussian with mean 0 and variance 1. We denote this density by  $\phi$ . The above integral becomes
$$
\begin{array}{l} \alpha \sigma^ {2} \int_ {- \infty} ^ {w} (w - y) ^ {2} f _ {\mathcal {G}} (v) d \mu (v) + \sigma^ {2} \int_ {w} ^ {\infty} (w - y) ^ {2} f _ {\mathcal {G}} (v) d \mu (v) \\ = \alpha \sigma^ {2} \int_ {- \infty} ^ {w} (w ^ {2} + y ^ {2} - 2 w y) \phi (y) d \mu (y) + \sigma^ {2} \int_ {w} ^ {\infty} (w ^ {2} + y ^ {2} - 2 w y) \phi (y) d \mu (y) \\ = \alpha \sigma^ {2} \int_ {- \infty} ^ {w} (w ^ {2} + y ^ {2} - 2 w y) \phi (y) d \mu (y) - \sigma^ {2} \int_ {- \infty} ^ {w} (w ^ {2} + y ^ {2} - 2 w y) \phi (y) d \mu (y) \\ + \sigma^ {2} \int_ {- \infty} ^ {\infty} (w ^ {2} + y ^ {2} - 2 w y) \phi (y) d \mu (y) \\ = (\alpha - 1) \sigma^ {2} \left(w ^ {2} \Phi (w) + \Phi (w) - w \phi (w) + 2 w \phi (w)\right) ] + \sigma^ {2} \left(w ^ {2} + 1\right) \\ = (\alpha - 1) \sigma^ {2} \left((w ^ {2} + 1) \Phi (w) + w \phi (w)\right) + \sigma^ {2} \left(w ^ {2} + 1\right). \\ \end{array}
$$
Applicability of Theorem C.1. We verify that any instance  $\mathcal{I} = (\Omega, f_{\mathcal{G}}, \ell, \alpha, \tau)$  of PrimalOpt defined by  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{G}}$  as a Gaussian density, a finite  $\alpha$ , and  $\tau$  satisfies the assumptions in Theorem C.1. Since  $\Omega = \mathbb{R}$ , (A0) holds for any finite  $\tau$ . (A1) and (A2) hold due to the choice of the loss function. (A3) holds since  $f_{\mathcal{G}}$  has a finite variance. (A4) holds with, e.g.,  $R = \sigma^2$ . In Lemma H.1, we compute
where  $w = \frac{x - m}{\sigma}$ . From this expression, it follows that  $I(x)$  is differentiable and
$$
\frac {\partial^ {2} I (x)}{\partial^ {2} x} = 2 + 2 (\alpha - 1) \Phi (w).
$$
Since  $\alpha \geq 1$  and  $\Phi(\cdot)$  is non-negative, it follows that  $I(x)$  is strongly convex and, hence, has a unique global minimum. Therefore, (A5) holds. Since assumptions (A0)-(A5) hold, we invoke Theorem C.1 to deduce the form of  $f^{\star}$ .
Figure 3 plots the mean and the variance of the output density  $f^{\star}$  as a function of the parameter  $\tau$  when the input density  $f_{\mathcal{D}}$  is the standard normal density. Figure 4 plots the mean and the variance of the output density as a function of the parameter  $\alpha$  in this setting.