SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem H.2 (Expression for  $f^{\star}$  when  $\alpha = 1$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{D}}, \ell, 1, \tau)$  of PrimalOpt where  $\Omega = \mathbb{R}$ ,  $\ell(x, v) := (x - v)^2$ , and  $f_{\mathcal{G}}$  is the Gaussian density with mean  $m$  and variance  $\sigma^2$ . Let  $f^{\star}$  be the optimal solution of  $\mathcal{I}$ . Then  $f^{\star}$  is a Gaussian with mean  $m$  and variance  $\frac{1}{2\pi} e^{2\tau - 1}$ .
(b)
Figure 4: Mean and variance of the output density  $f$ , i.e.,  $\mathbb{E}_{x\sim f}[x]$  and  $\mathrm{Var}_{x\sim f}[x]$ , as a function of  $\alpha$  when  $f_{\mathcal{D}}$  is the standard normal density and  $\tau$  is fixed to the entropy of  $f_{\mathcal{D}}$ .
Thus, for  $\alpha = 1$ , increasing  $\tau$  does not change the mean, but increases the variance of the output density.
Proof. For  $\alpha = 1$ , Lemma H.1 implies that
$$
I (x) = (x - m) ^ {2} + \sigma^ {2}.
$$
As shown earlier, by Theorem C.1,  $f^{\star}$  has the following form
$$
f ^ {\star} (x) \propto e ^ {\frac {- (x - m) ^ {2} + \sigma^ {2}}{\gamma^ {\star}}} \propto e ^ {\frac {- (x - m) ^ {2}}{\gamma^ {\star}}}.
$$
where the proportionality constant and  $\gamma^{\star}$  are determined by  $\int_{\mathbb{R}}f^{\star}(x)d\mu (x) = 1$  and  $\mathrm{Ent}(f^{\star}) = \tau$ .  $f^{\star}$  is a Gaussian density with mean  $m$  and variance  $\frac{\gamma^{\star}}{2}$  and, hence,  $\mathrm{Ent}(f^{\star}) = \frac{1}{2} +\frac{1}{2}\ln (\pi \gamma^{\star})$  [148]. Since  $\mathrm{Ent}(f^{\star}) = \tau$ , the previous equality implies that  $\gamma^{\star} = \frac{1}{\pi} e^{2\tau -1}$ . It follows that  $f^{\star}(x) = \frac{1}{\sqrt{2e^{2\tau - 1}}}\exp \left(-\pi \frac{(x - m)^2}{e^{2\tau - 1}}\right)$  which is the Gaussian density with mean  $m$  and variance  $\frac{1}{2\pi} e^{2\tau -1}$ .
Shift in mean with increase in  $\alpha$ . We consider the effect of the parameter  $\alpha$  on the mean of the output density. We consider an instance  $\mathcal{I}_1 = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau)$  where  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{D}}$  is the normal density  $N(\mu, \sigma^2)$ ,  $\ell(v, x) = (x - v)^2$ ,  $\tau = \operatorname{Ent}(f_{\mathcal{D}}) = \ln(\sigma \sqrt{2\pi e})$  and  $\alpha = 1$ . We know from the proof of Theorem H.2 that the output density  $f^{\star}$  is the same as  $f_{\mathcal{D}}$  and hence, has mean 0.
Now consider an instance  $\mathcal{I}_2$ , that corresponds to the disadvantaged group and has the same parameters as that of  $\mathcal{I}_1$ , except that the parameter  $\alpha$  is larger than 1. Let  $f_{\alpha}^{\star}$  denote the corresponding output density. We know from Theorem C.16 that the output density is proportional to  $e^{-I_{\alpha}(x) / \gamma_{\alpha}^{\star}}$ , where  $\gamma_{\alpha}^{\star}$  is the optimal dual variable for the entropy constraint and Lemma H.1 shows that  $I_{\alpha}(x)$  is given by:
$$
I _ {\alpha} (x) = (\alpha - 1) \sigma^ {2} \left((w ^ {2} + 1) \Phi (w) + w \phi (w)\right) + \sigma^ {2} \left(w ^ {2} + 1\right),
$$
where  $w = (x - \mu) / \sigma$ . For sake of brevity, let  $g(w)$  denote  $(w^{2} + 1)\Phi (w) + w\phi (w)$ , and assume w.l.o.g. that  $\mu = 0$  and  $\sigma = 1$ , i.e., the input density is standard normal. Thus,  $I_{\alpha}(x)$  can be written as  $(\alpha -1)g(x) + (x^{2} + 1)$ . When  $\alpha$  is large, the first term here dominates the second term as long as  $\Phi (x)\gg 1 / \alpha$ . Since  $\Phi (x)\geq 1 / 2$  for all  $x\geq 0$ ,  $I_{\alpha}(x)$  is relatively large for all  $x\geq 0$ , and hence,  $f_{\alpha}^{\star}(x)$  goes to 0 for  $x\in [0,\infty)$  as  $\alpha$  increases. In fact,  $g(x)$  is much larger than  $1 / \alpha$  when  $x$  is larger than  $-\Omega (\sqrt{\ln(\alpha)})$  and, hence,  $f_{\alpha}^{\star}$  should be small for such values of  $x$  as well. Therefore, we expect the mean of the output density  $f_{\alpha}^{\star}$  to go to  $-\infty$  as  $\alpha$  goes to  $\infty$ . Further, the mean of  $f_{\alpha}^{\star}$  should decrease at a logarithmic rate with respect to  $\alpha$ . We verify these observations numerically in Figure 5.
Deriving the implicit variance model. We show that the implicit variance model can be derived from our optimization framework. In particular, we prove the following result:
Theorem H.3. Consider an instance of the implicit variance model given by parameters  $\mu, \sigma, \sigma_0$ . Consider instances  $I_1 = (\Omega, f_{\mathcal{G}}, \ell, \alpha, \tau_1)$  and  $I_2 = (\Omega, f_{\mathcal{G}}, \ell, \alpha, \tau_2)$  of (OptProg), where  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{G}}$  is the normal density  $N(\mu, \sigma_0^2)$ ,  $\alpha = 1$ ,  $\ell(v, x) = (x - v)^2$ ,  $\tau_1 = \frac{1}{2} (1 + \ln(2\pi \sigma_0^2))$  and
(b)
Figure 5: Figure (a) plots the input density, i.e., standard normal density, and the output density when the parameter  $\alpha = 1000$ . Figure (b) plots the mean of the output density as a function of the parameter  $\alpha$  (on a log scale).
$\tau_{2} = \frac{1}{2} (1 + \ln (2\pi (\sigma_{0}^{2} + \sigma^{2})))$  . Then the output density of (OptProg) on  $I_{1}$  is  $N(\mu ,\sigma_0^2)$  and the output density of (OptProg) on  $I_{2}$  is  $N(\mu ,\sigma_0^2 +\sigma^2)$
Proof. The result follows from Theorem H.2. For the instance  $\mathcal{I}_1$ , Theorem H.2 shows that the output density of OptProg is the normal density with mean  $\mu$  and variance
$$
\frac {1}{2 \pi} e ^ {2 \tau_ {1} - 1} = \sigma_ {0} ^ {2}.
$$
Similarly, it follows that the output density for the instance  $\mathcal{I}_2$  is normal with mean  $\mu$  and variance
$$
\frac {1}{2 \pi} e ^ {2 \tau_ {2} - 1} = \sigma_ {0} ^ {2} + \sigma^ {2}.
$$
I Pareto density
The Pareto density is defined as follows over  $\Omega = [1,\infty)$  and has a parameter  $\beta > 1$ :
$$
f _ {\mathcal {P}} (x) := \frac {\beta}{x ^ {\beta + 1}}, \quad x \in [ 1, \infty).
$$
The mean of this density is  $\frac{\beta}{\beta - 1}$ . Thus, the mean is finite only when  $\beta > 1$ . Its differential entropy is  $1 + \frac{1}{\beta} + \ln \frac{1}{\beta}$  [148]. We consider the loss function  $\ell(x, v) \coloneqq \ln x - \ln v$ . First, we compute the expression for  $I(x)$  which we use to verify the applicability of Theorem C.1 with the above parameters.
Lemma I.1 (Expression for  $I_{\alpha}(x)$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{P}}, \ell, \alpha, \tau)$  of PrimalOpt where  $\Omega = [1, \infty)$ ,  $\ell(x, v) \coloneqq \ln x - \ln v$ , and  $f_{\mathcal{P}}$  is the Pareto density with parameter  $\beta > 1$ . Then
$$
I (x) = \alpha \ln x + \frac {\alpha - 1}{\beta x ^ {\beta}} - \frac {\alpha}{\beta}.
$$
Proof. We use integration by parts to derive the following expression for  $I(x)$ :
$$
\begin{array}{l} I (x) = \alpha \int_ {1} ^ {x} (\ln x - \ln v) \frac {\beta}{v ^ {\beta + 1}} d \mu (v) + \int_ {x} ^ {\infty} (\ln x - \ln v) \frac {\beta}{v ^ {\beta + 1}} d \mu (v) \\ = \alpha \left[ \frac {\ln v - \ln x}{v ^ {\beta}} \right] _ {1} ^ {x} - \alpha \int_ {1} ^ {x} \frac {1}{v ^ {\beta + 1}} d \mu (v) + \left[ \frac {\ln v - \ln x}{v ^ {\beta}} \right] _ {x} ^ {\infty} + \int_ {x} ^ {\infty} \frac {1}{v ^ {\beta + 1}} d \mu (v) \\ = \alpha \ln x + \frac {\alpha}{\beta} \left(\frac {1}{x ^ {\beta}} - 1\right) - \frac {1}{\beta x ^ {\beta}}. \\ \end{array}
$$
(b)
Figure 6: Mean and variance of the output density  $f$ , i.e.,  $\mathbb{E}_{x \sim f}[x]$  and  $\mathrm{Var}_{x \sim f}[x]$ , as a function of  $\tau$  when  $f_{\mathcal{D}}$  is the Pareto distribution with parameter 3 and  $\alpha$  is fixed to 2.
Applicability of Theorem C.1. Next, we verify that assumptions (A0)-(A5) hold. Since  $\Omega = [1,\infty)$ , (A0) holds for any finite  $\tau$ . (A1) and (A2) hold due to the choice of the loss function. (A3) can be shown to hold since  $f_{\mathcal{P}}$  is a Pareto density: To see this note that for any finite  $x \geq 1$
$$
\begin{array}{l} \int_ {\Omega} | \ell (x, v) | f _ {\mathcal {P}} (v) d \mu (v) = \int_ {1} ^ {x} (\ln (x) + \ln (v)) f _ {\mathcal {P}} (v) d \mu (v) + \int_ {x} ^ {\infty} (\ln (v) - \ln (x)) f _ {\mathcal {P}} (v) d \mu (v) \\ \leq \int_ {1} ^ {\infty} (\ln (x) + \ln (v)) f _ {\mathcal {P}} (v) d \mu (v) \\ = \ln (x) + \int_ {1} ^ {\infty} \ln (v) f _ {\mathcal {P}} (v) d \mu (v) \\ = \ln (x) + \left[ - \frac {\ln (v)}{v ^ {\beta}} \right] _ {1} ^ {\infty} + \int_ {1} ^ {\infty} \frac {1}{v ^ {\beta + 1}} d \mu (v) \\ = \ln (x) + \left[ - \frac {v ^ {- \beta}}{\beta} \right] _ {1} ^ {\infty} \\ = \ln (x) + \frac {1}{\beta} \\ <   \infty . \\ \end{array}
$$
Thus, (A3) holds. (A4) holds with, e.g.,  $R = 2$ . By Lemma I.1,
Thus,  $I(x)$  is differentiable at each  $x\in \Omega$  . Moreover, for all  $x\in \Omega$
$$
\frac {\partial I (x)}{\partial x} = \frac {\alpha}{x} - \frac {\alpha - 1}{x ^ {\beta + 1}} = \frac {1}{x} \left(\alpha - \frac {\alpha - 1}{x ^ {\beta}}\right) ^ {x \geq 1} > 0.
$$
Since the derivative is positive for all  $x \in \Omega = [1, \infty)$ , it follows that  $I(x)$  has a unique global minimum at  $x = 1$ . Therefore, (A5) holds. Since assumptions (A0)-(A5) hold, we invoke Theorem C.1 to deduce the form of  $f^{\star}$ . Figure 6 plots the mean and the variance of the output density  $f^{\star}$  as a function of the parameter  $\tau$  when the input density  $f_{\mathcal{D}}$  is the standard normal density. Figure 7 plots the mean and the variance of the output density as a function of the parameter  $\alpha$  in this setting.
Theorem I.2 (Expression for  $f^{\star}$  with  $\alpha = 1$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{P}}, \ell, 1, \tau)$  of PrimalOpt where  $\Omega = [1, \infty)$ ,  $\ell := \ln x - \ln v$ , and  $f_{\mathcal{P}}$  is the Pareto density with parameter  $\beta > 1$ . Let  $f^{\star}$  be the optimal solution of instance  $\mathcal{I}$ .  $f^{\star}$  is a Pareto density with parameter  $\beta_{\tau}$  satisfying the following condition:
$$
1 + \frac {1}{\beta_ {\tau}} - \ln \beta_ {\tau} = \tau .
$$
(b)
Figure 7: Mean and variance of the output density  $f$ , i.e.,  $\mathbb{E}_{x\sim f}[x]$  and  $\mathrm{Var}_{x\sim f}[x]$ , as a function of  $\alpha$  when  $f_{\mathcal{D}}$  is the Pareto distribution with parameter 3 and  $\tau$  is fixed to the entropy of  $f_{\mathcal{D}}$ .
Let  $m_{\tau}$  and  $\sigma_{\tau}^{2}$  be the mean and variance of  $f_{\tau}^{\star}$  as a function of  $\tau$ . It holds that  $m_{\tau}$  is monotonically increasing in  $\tau$  and  $\sigma_{\tau}^{2}$  is either infinite or monotonically increasing in  $\tau$ .
Proof. For  $\alpha = 1$ , Lemma I.1 implies that
$$
I (x) = \ln (x) - \frac {1}{\beta}.
$$
As shown earlier, one can invoke Theorem C.1 for instance  $\mathcal{I}$  for any finite  $\tau$ , which implies that  $f^{\star}$  has the following form
$$
f ^ {\star} (x) \propto e ^ {\frac {- \ln (x) + \frac {1}{\beta}}{\gamma^ {\star}}} \propto x ^ {- \frac {1}{\gamma^ {\star}}}.
$$
where the proportionality constant and  $\gamma^{\star}$  are determined by  $\int_{\Omega}f^{\star}(x)d\mu (x) = 1$  and  $\mathrm{Ent}(f^{\star}) = \tau$ .  $f^{\star}$  is a Pareto density with parameter  $\beta_{\tau} = \frac{1}{\gamma^{\star}} -1$  and, hence, has a differential entropy of  $1 + \frac{1}{\beta_{\tau}} +\ln \frac{1}{\beta_{\tau}}$  [148]. This combined with the condition  $\mathrm{Ent}(f^{\star}) = \tau$  implies that  $f^{\star}$  is the Pareto density with  $\beta_{\tau}$  satisfying
$$
1 + \frac {1}{\beta_ {\tau}} + \ln \frac {1}{\beta_ {\tau}} = \tau .
$$
Since  $1 + \frac{1}{\beta_{\tau}} + \ln \frac{1}{\beta_{\tau}}$  is a decreasing function of  $\beta_{\tau}$ , it follows that increasing  $\tau$  monotonically decreases  $\beta_{\tau}$ . Note that  $\beta_{\tau} > 1$  for any finite  $\tau$ . The mean and variance of  $f^{\star}$  are  $m_{\tau} = \frac{\beta_{\tau}}{\beta_{\tau} - 1}$  and  $\sigma_{\tau}^{2} = \frac{\beta_{\tau}}{(\beta_{\tau} - 1)^{2}(\beta_{\tau} - 2)}$  respectively. Since  $m_{\tau}$  is a monotonically decreasing function of  $\beta_{\tau}$  (for  $\beta_{\tau} > 1$ ) and  $\beta_{\tau}$  is a monotonically decreasing function of  $\tau$ , it follows that  $m_{\tau}$  is a monotonically increasing function of  $\beta_{\tau}$ . The variance  $\sigma_{\tau}^{2}$  is finite when  $\beta_{\tau} > 2$ . Moreover, if  $\beta_{\tau} > 2$ , then  $\sigma_{\tau}^{2}$  is a monotonically decreasing function of  $\beta_{\tau}$ . Since  $\beta_{\tau}$  is a monotonically decreasing function of  $\tau$ , it follows that  $\sigma_{\tau}^{2}$  is either infinite or a monotonically increasing function of  $\beta_{\tau}$ .
Reduction in mean with increase in  $\alpha$ . We show how our framework can capture a similar phenomenon as the multiplicative-bias model of [90]. Recall that in the multiplicative-bias model, the estimated utility of the disadvantaged group is scaled down by a factor  $\rho > 1$ .
Fix a parameter  $\beta > 1$ . Consider an instance  $\mathcal{I}_1$  given by the parameters  $I_1 = (\Omega = [1, \infty), f_{\mathcal{D}}, \ell, \alpha, \tau)$  where  $f_{\mathcal{D}}$  is the Pareto density with parameter  $\beta$ ,  $\tau = \mathrm{Ent}(f_{\mathcal{D}}) = 1 + \frac{1}{\beta} + \ln \frac{1}{\beta}$ ,  $\alpha = 1$  and  $\ell(x, v) = \ln x - \ln v$ . As shown in Theorem I.2, the output density  $f^{\star}$  is the same as  $f_{\mathcal{D}}$ . The proof of this result also shows that the output density  $f^{\star}(x)$  is proportional to  $e^{-I(x) / \gamma^{\star}}$ , where  $I(x) = \ln x - \frac{1}{\beta}$  and  $\gamma^{\star}$  is the optimal dual variable for the corresponding entropy constraint.
The disadvantaged group is modeled by an instance  $\mathcal{I}_2$  which has the same parameters as that of  $\mathcal{I}_1$  except that the parameter  $\alpha$  is larger than 1. In this case, Theorem I.2 shows that the optimal density  $f_{\alpha}^{\star}(x)$  is proportional to  $e^{-I_{\alpha}(x) / \gamma_{\alpha}^{\star}}$ , where  $I_{\alpha}(x) = \alpha \ln x + \frac{\alpha - 1}{\beta x^{\beta}} - \frac{\alpha}{\beta}$  and  $\gamma_{\alpha}^{\star}$  is the corresponding
(b)
Figure 8: Figure (a) plots the input Pareto density with parameter  $\beta = 1.5$  and the corresponding limiting output density. Figure (b) plots the mean of the input Pareto density and the corresponding limiting output density as a function of the parameter  $\beta$ .
optimal dual variable. For large  $\alpha$ ,  $I_{\alpha}(x) \sim \alpha(I(x) + \frac{1}{\beta x^{\beta}})$ . Hence, the output density (for large  $\alpha$ ) is given by
$$
f _ {\alpha} ^ {\star} (x) = K \cdot e ^ {- \frac {\alpha}{\gamma_ {\alpha} ^ {\star}} \left(I (x) + \frac {1}{\beta x ^ {\beta}}\right)},
$$
where  $K$  is a normalization constant. The normalization constant  $K$  and the ratio  $\gamma^{\star} / \alpha$  are given by two constraints: (i) integral of  $f_{\alpha}^{\star}$  over the domain  $\Omega$  should be 1, and (ii) the entropy of  $f_{\alpha}^{\star}$  should be equal to  $\tau = \mathsf{Ent}(f_{\mathcal{D}})$ . This shows that the ratio  $\gamma_{\alpha}^{\star} / \alpha$  tends to a constant for large  $\alpha$ , and hence, the output density converges to the density given by
$$
g ^ {\star} (x) = K e ^ {- C \left(\ln x + \frac {1}{\beta_ {x} \beta}\right)}.
$$
We plot the output density  $g^{\star}(x)$  for  $\beta = 1.5$  in Figure 8(a). In Figure 8(b), we observe that for all considered values of the parameter  $\beta$ , the mean of the limiting output density  $g^{\star}$  always remains below that of the input density  $f_{\mathcal{D}}$ . It is also worth noting that for large  $\beta$ , the gap between the mean of the input density and that of the (limiting) output density diminishes. Intuitively, this happens because as  $\beta$  increases, the mean of the input (Pareto) density gets closer to 1 (recall that the mean of a Pareto density with parameter  $\beta$  is equal to  $\frac{\beta}{\beta - 1}$ ). Now the output density also places more mass closer to 1, but gets restricted because of two conditions: (i) the entropy of the output density must be the same as that of the corresponding input density, and (ii) it cannot place any probability mass on values below 1. Hence, there is not much "room" for the output density to place extra probability mass on values close to 1 (as compared to the corresponding Pareto density). Hence its mean cannot go much below that of the corresponding Pareto density.
J Exponential density
The exponential density is defined as follows over  $\Omega = [0,\infty)$  and has a parameter  $\lambda > 0$ :
$$
f _ {\operatorname {E x p}} (x) := \lambda e ^ {- \lambda x}, \quad x \in [ 0, \infty).
$$
$\lambda$  is referred to as the "rate" parameter. The mean of the exponential density is  $\frac{1}{\lambda}$ . The differential entropy of  $f_{\mathrm{Exp}}$  is  $1 - \ln \lambda$  [148]. We consider the loss function  $\ell(x, v) \coloneqq x - v$ . First, we compute the expression of  $I(x)$  which we use to verify the applicability of Theorem C.1 with the above parameters.
Lemma J.1 (Expression for  $I(x)$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathrm{Exp}}, \ell, \alpha, \tau)$  of PrimalOpt where  $\Omega = [0, \infty)$ ,  $f_{\mathrm{Exp}}$  is the Exponential density with rate parameter  $\lambda$ , and  $\ell(x, v) := x - v$ . Then
$$
I (x) = \frac {1}{\lambda} (\alpha (\lambda x - 1) + (\alpha - 1) e ^ {- \lambda x}).
$$
Proof. The desired integral is
$$
\lambda \alpha \int_ {0} ^ {x} (x - v) e ^ {- \lambda v} d \mu (v) + \lambda \int_ {x} ^ {\infty} (x - v) e ^ {- \lambda v} d \mu (v).
$$
We do a change of variable, with  $y \coloneqq x - v$  above, to get
$$
\begin{array}{l} \lambda \alpha \int_ {0} ^ {x} y e ^ {\lambda (y - x)} d \mu (y) + \lambda \int_ {- \infty} ^ {0} y e ^ {\lambda (y - x)} d \mu (y) = \lambda \alpha e ^ {- \lambda x} \left[ e ^ {\lambda y} \frac {\lambda y - 1}{\lambda^ {2}} \right] _ {0} ^ {x} + \lambda e ^ {- \lambda x} \left[ e ^ {\lambda y} \frac {\lambda y - 1}{\lambda^ {2}} \right] _ {- \infty} ^ {0} \\ = \frac {\alpha (\lambda x - 1)}{\lambda} + \frac {\alpha e ^ {- \lambda x}}{\lambda} - \frac {e ^ {- \lambda x}}{\lambda} \\ = \frac {1}{\lambda} (\alpha (\lambda x - 1) + (\alpha - 1) e ^ {- \lambda x}). \\ \end{array}
$$
Applicability of Theorem C.1. We show that assumptions (A0)-(A5) hold. Since  $\Omega = [0,\infty)$ , (A0) holds for any finite  $\tau$ . (A1) and (A2) hold due to the choice of the loss function. (A3) can be shown to hold since  $f_{\mathrm{Exp}}$  is an Exponential density: To see this note that for any finite  $x\in \Omega$
$$
\begin{array}{l} \int_ {\Omega} | \ell (x, v) | f _ {\mathrm {E x p}} (v) d \mu (v) = \int_ {1} ^ {x} (x - v) f _ {\mathrm {E x p}} (v) d \mu (v) + \int_ {x} ^ {\infty} (v - x) f _ {\mathrm {E x p}} (v) d \mu (v) \\ \leq \int_ {1} ^ {\infty} (| x | + | v |) f _ {\operatorname {E x p}} (v) d \mu (v) \\ = | x | + \frac {1}{\lambda} \\ <   \infty . \\ \end{array}
$$
Thus, (A3) holds. (A4) holds with, e.g.,  $R = \frac{1}{\lambda}$ . By Lemma J.1,
$$
I (x) = \frac {1}{\lambda} \left(\alpha (\lambda x - 1) + (\alpha - 1) e ^ {- \lambda x}\right).
$$
From this expression, it follows that  $I(x)$  is differentiable at each  $x\in \Omega$ . Moreover, for all  $x\in \Omega$
$$
\frac {\partial^ {2} I (x)}{\partial^ {2} x} = \lambda (\alpha - 1) e ^ {- \lambda x}.
$$
For any  $\alpha > 1$ , this derivative is positive for all  $x \in \Omega$  and, hence,  $I(x)$  strictly convex whenever  $\alpha > 1$  and, thus, it has a unique global minimum. If  $\alpha = 1$ , then  $I(x) = x - \frac{1}{\lambda}$ . This function has a unique global minimum at  $x = 0$  over  $\Omega = [0, \infty)$ . Combining with the  $\alpha > 1$  case, it follows that (A5) holds. Since assumptions (A0)-(A5) hold, one can invoke Theorem C.1 to deduce the form of  $f^{\star}$ .
Theorem J.2 (Expression for  $f^{\star}$  when  $\alpha = 1$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathrm{Exp}}, \ell, 1, \tau)$  of PrimalOpt where  $\Omega = [0, \infty)$ ,  $\ell(x, v) := x - v$ , and  $f_{\mathrm{Exp}}$  is the Exponential with rate parameter  $\lambda$ . Let  $f^{\star}$  be the optimal solution of  $\mathcal{I}$ . Then  $f^{\star}$  is the Exponential density with mean  $e^{\tau - 1}$
Thus, for  $\alpha = 1$ , increasing  $\tau$  increases the rate parameter of the output density.
Proof. Since  $\alpha = 1$ , Lemma J.1 implies that
$$
I (x) = x - \frac {1}{\lambda}.
$$
$$
f ^ {\star} (x) \propto \exp \left(\frac {- x - \frac {1}{\lambda}}{\gamma^ {\star}}\right) \propto \exp \left(\frac {- x}{\gamma^ {\star}}\right).
$$
where the proportionality constant and  $\gamma^{\star}$  are determined by  $\int_{\Omega}f^{\star}(x)d\mu (x) = 1$  and  $\mathsf{Ent}(f^{\star}) = \tau$ . Since  $f^{\star}$  is an Exponential density with rate parameter  $\frac{1}{\gamma^{\star}}$ , its entropy is  $\mathsf{Ent}(f^{\star}) = 1 + \ln (\gamma^{\star})$  [148]. Since  $\mathsf{Ent}(f^{\star}) = \tau$ , the previous equality implies that  $\gamma^{\star} = e^{\tau -1}$ . It follows that  $f^{\star}(x) = e^{1 - \tau}\cdot \exp \left(-\frac{x}{\exp(\tau - 1)}\right)$  which is the Exponential density with mean  $e^{\tau -1}$ .
K Laplace density
The Laplace density is defined as follows over  $\Omega = \mathbb{R}$  and has parameters  $a\in \mathbb{R}$  and  $b > 0$
$$
f _ {\mathcal {L}} (x) := \frac {1}{2 b} e ^ {- \frac {1}{b} | x - a |}, \quad x \in \mathbb {R}.
$$
$a$  is referred to as the "location" parameter and  $b$  as "diversity." The differential entropy of  $f_{\mathcal{L}}$  is  $1 + \ln (2b)$  [148]. We consider the loss function to be  $\ell (x,v)\coloneqq |x - a| - |v - a|$  for  $a\in \mathbb{R}$ . First, we compute the expression for  $I(x)$  that we use to show that Theorem C.1 is applicable with the above parameters.
Lemma K.1 (Expression for  $I(x)$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{L}}, \ell, \alpha, \tau)$  of PrimalOpt where  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{L}}$  is the Laplace density with parameters  $a \in \mathbb{R}$  and  $b > 0$ , and  $\ell(x, v) \coloneqq |x - a| - |v - a|$ . Then
$$
I (x) = \left\{ \begin{array}{l l} \alpha b (w - 1) + b \frac {(\alpha - 1)}{2} e ^ {- w} & \text {i f} x \geq a \\ - b (w + 1) + b \frac {(1 - \alpha)}{2} e ^ {w} & \text {o t h e r w i s e} \end{array} \right.,
$$
where  $w\coloneqq \frac{x - a}{b}$
$$
\frac {\alpha}{2 b} \int_ {- \infty} ^ {x} (| x - a | - | v - a |) e ^ {- | v - a | / b} d \mu (v) + \frac {1}{2 b} \int_ {x} ^ {\infty} (| x - a | - | v - a |) e ^ {- | v - a | / b} d \mu (v).
$$
We perform a change of variables with  $w \coloneqq \frac{x - a}{b}$  and  $y = \frac{v - a}{b}$ . Then the above integral becomes
$$
\frac {\alpha b}{2} \int_ {- \infty} ^ {w} (| w | - | y |) e ^ {- | y |} d \mu (y) + \frac {b}{2} \int_ {w} ^ {\infty} (| w | - | y |) e ^ {- | y |} d \mu (y).
$$
Now two cases arise (i)  $w \geq 0$ , or (ii)  $w \leq 0$ . First, consider the case when  $w \geq 0$ . Then the above integral becomes:
$$
\begin{array}{l} \frac {\alpha}{2} \int_ {- \infty} ^ {0} (w + y) e ^ {y} d \mu (y) + \frac {\alpha}{2} \int_ {0} ^ {w} (w - y) e ^ {- y} d \mu (y) + \frac {1}{2} \int_ {w} ^ {\infty} (w - y) e ^ {- y} d \mu (y) \\ = \frac {\alpha (w - 1)}{2} + \frac {\alpha (w + e ^ {- w} - 1)}{2} - \frac {e ^ {- w}}{2} \\ = \alpha b (w - 1) + \frac {(\alpha - 1) b}{2} e ^ {- w}. \\ \end{array}
$$
In the second case, we get
$$
\begin{array}{l} \frac {\alpha}{2} \int_ {- \infty} ^ {w} (- w + y) e ^ {y} d \mu (y) + \frac {1}{2} \int_ {w} ^ {0} (- w + y) e ^ {y} d \mu (y) + \frac {1}{2} \int_ {0} ^ {\infty} (- w - y) e ^ {- y} d \mu (y) \\ = - \frac {\alpha e ^ {w}}{2} + \frac {e ^ {w} - w - 1}{2} - \frac {1 + w}{2} \\ = - (w + 1) b + \frac {(1 - \alpha) e ^ {w} b}{2}. \\ \end{array}
$$
Applicability of Theorem C.1. We show that for any finite  $a \in \mathbb{R}$  and  $b > 0$ , assumptions (A0)-(A5) hold for the instance  $\mathcal{I} = (\Omega, f_{\mathcal{L}}, \ell, \alpha, \tau)$  of PrimalOpt where  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{L}}$  is the Laplace density with parameters  $a \in \mathbb{R}$  and  $b > 0$ , and  $\ell(x, v) = |x - a| - |v - a|$ . Since  $\Omega = \mathbb{R}$ , (A0) holds for any