SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This requirement is not automatically satisfied and typically necessitates modifications to current non-private HPO methods. To address this challenge, we propose a general recipe to modify any non-private method. The idea is quite straightforward: throughout the algorithm, we maintain a non-private version of the distribution density  $\pi^{(j)}$ . When sampling from the space  $\Lambda$ , we perform a projection from  $\pi^{(j)}$  to the space consisting of bounded densities. Specifically, we define the space of essentially bounded density functions by  $S_{C,c} = \{f \in \Lambda^{\mathbb{R}^+} : \operatorname{ess}\sup f \leq \frac{C}{\mu(\Lambda)}, \operatorname{ess}\inf f \geq \frac{c}{\mu(\Lambda)}, \int_{\alpha \in \Lambda} f(\alpha) \mathrm{d}\alpha = 1\}$ . For such a space to be non-empty, we require that  $c \leq 1 \leq C$ , where  $\mu$  is the measure on  $\Lambda$ . This condition is well-defined as we assume  $\mu(\Lambda) < \infty$ .
To privatize  $\pi^{(j)}$  at the  $j$ -th iteration, we project it into the space  $S_{C,c}$ , by solving the following convex functional programming problem:
$$
\min  _ {f} \| f - \pi^ {(j)} \| _ {2}, \tag {3.1}
$$
$$
\begin{array}{l} \text {s . t .} f \in S _ {C, c}. \end{array}
$$
Note that this is a convex program since  $S_{C,c}$  is convex and closed. We denote the output from this optimization problem by  $\mathcal{P}_{S_{C,c}}(\pi^{(j)})$ . Theoretically, problem (3.1) allows the hyperparameter space  $\Lambda$  to be general measurable space with arbitrary topological structure. However, empirically, practitioners need to discretize  $\Lambda$  to some extent to make the convex optimization computationally feasible. Compared to the previous work, our formulation provides the most general characterization of the problem and allows practitioners to adaptively and iteratively choose a proper discretization as needed. Framework 1 tolerates a much finer level of discretization than the previous method, as the performance of latter degrades fast when the number of candidates increases. We also provide examples using CVX to solve this problem in Section 4.2. In Appendix C, we discuss about its practical implementation, and the connection to information projection.
4 Application: DP-HyPO with Gaussian Process
In this section, we provide an instantiation of DP-HyPO using Gaussian process (GP) [38]. GPs are popular non-parametric Bayesian models frequently employed for hyperparameter optimization. At the meta-level, GPs are trained to generate surrogate models by establishing a probability distribution over the performance measure  $q$ . While traditional GP implementations are not private, we leverage the approach introduced in Section 3.2 to design a private version that adheres to the bounded density constraint.
We provide the algorithmic description in Section 4.1 and the empirical evaluation in Section 4.2.
4.1 Algorithm Description
The following Algorithm  $(\mathcal{A}\mathcal{G}\mathcal{P})$  is a private version of Gaussian process for hyperparameter tuning. In Algorithm 3, we utilize GP to construct a surrogate model that generates probability distributions
Algorithm 3 DP-HyPO with Gaussian process  $\mathcal{A}\mathcal{G}\mathcal{P}(D,\theta ,\gamma ,\tau ,\beta ,\Lambda ,C,c)$
Initialize  $\pi^{(0)} = \mathrm{Unif}(\Lambda)$  , and  $A = \{\}$    
Draw  $T\sim \mathrm{NegBin}(\theta ,\gamma)$    
for  $t = 0$  to  $T - 1$  do Truncate the density of current  $\pi^{(t)}$  to be bounded into the range of  $[c,C]$  by projecting to  $S_{C,c}$ $\tilde{\pi}^{(t)} = \mathcal{P}_{S_{C,c}}(\pi^{(t)})$  Sample  $(x,q)\sim Q(D,\tilde{\pi}^{(j)})$  , and update  $A = A\cup \{(x,q)\}$  Update mean estimation and variance estimation of the Gaussian process  $\mu_{\lambda},\sigma_{\lambda}^{2}$  , and get the score as  $s_\lambda = \mu_\lambda +\tau \sigma_\lambda$  Update true (untruncated) posterior  $\pi^{(t + 1)}$  with softmax, by  $\pi^{(t + 1)}(\lambda) = \frac{\exp(\beta\cdot s_{\lambda})}{\int_{\lambda^{\prime}\in\Lambda}\exp(\beta\cdot s_{\lambda}^{\prime})}$    
end for   
Output  $(x,q)$  from  $A$  with the highest  $q$
for the performance measure  $q$ . By estimating the mean and variance, we assign a "score" to each hyperparameter  $\lambda$ , known as the estimated upper confidence bound (UCB). The weight factor  $\tau$  controls the balance between exploration and exploitation, where larger weights prioritize exploration by assigning higher scores to hyperparameters with greater uncertainty.
To transform these scores into a sampling distribution, we apply the softmax function across all hyperparameters, incorporating the parameter  $\beta$  as the inverse temperature. A higher value of  $\beta$  signifies increased confidence in the learned scores for each hyperparameter.
4.2 Empirical Evaluations
We now evaluate the performance of our GP-based DP-HyPO (referred to as "GP") in various settings. Since DP-HyPO is the first adaptive private hyperparameter optimization method of its kind, we compare it to the special case of Uniform DP-HyPO (Algorithm 2), referred to as "Uniform", as proposed in [24, 32]. In this demonstration, we consider two pragmatic privacy configurations: the white-box setting and the black-box setting, contingent on whether adaptive HPO algorithms incur extra privacy cost. In the white-box scenario (Section 4.2.1 and 4.2.2), we conduct experiments involving training deep learning models on both the MNIST dataset and CIFAR-10 dataset. Conversely, when considering the black-box setting (Section 4.2.3), our attention shifts to a real-world Federated Learning (FL) task from the industry. These scenarios provide meaningful insights into the effectiveness and applicability of our GP-based DP-HyPO approach.
4.2.1 MNIST Simulation
We begin with the white-box scenario, in which the data curator aims to provide overall protection to the published model. In this context, to accommodate adaptive HPO algorithms, it becomes necessary to reduce the budget allocated to the base algorithm.
In this section, we consider the MNIST dataset, where we employ DP-SGD to train a standard CNN. The base algorithms in this case are different DP-SGD models with varying hyperparameters, and we evaluate each base algorithm based on its accuracy. Our objective is to identify the best hyperparameters that produce the most optimal model within a given total privacy budget.
Specifically, we consider two variable hyperparameters: the learning rate  $\eta$  and clipping norm  $R$ , while keeping the other parameters fixed. We ensure that both the GP algorithm and the Uniform algorithm operate under the same total privacy budget, guaranteeing a fair comparison.
Due to constraints on computational resources, we conduct a semi-real simulation using the MNIST dataset. For both base algorithms (with different noise multipliers), we cache the mean accuracy of 5 independently trained models for each discretized hyperparameter and treat that as a proxy for the "actual accuracy" of the hyperparameter. Each time we sample the accuracy of a hyperparameter, we add a Gaussian noise with a standard deviation of 0.1 to the cached mean. We evaluate the performance of the output model based on the "actual accuracy" corresponding to the selected hyperparameter. Further details on the simulation and parameter configuration can be found in Appendix E.1.
In the left panel of Figure 1, we demonstrated the comparison of performance of the Uniform and GP methods with total privacy budget  $\varepsilon = 15^3$  and  $\delta = 1e - 5$ . The accuracy reported is the actual accuracy of the output hyperparameter. From the figure, we see that when  $T$  is very small ( $T < 8$ ), GP method is slightly worse than Uniform method as GP spends  $\log (C / c)$  budget less than Uniform method for each base algorithm (the cost of adaptivity). However, we see that after a short period of exploration, GP consistently outperform Uniform, mostly due to the power of being adaptive. The superiority of GP is further demonstrated in Table 1, aggregating over geometric distribution.
4.2.2 CIFAR-10 Simulation
When examining the results from MNIST, a legitimate critique arises: our DP-Hypo exhibits only marginal superiority over its uniform counterpart, which questions the assertion that adaptivity holds significant value. Our conjecture is that the hyperparameter landscape of MNIST is relatively uncomplicated, which limits the potential benefits of adaptive algorithms.
To test the hypothesis, we conduct experiments on the CIFAR-10 dataset, with a setup closely mirroring the previous experiment: we employ the same CNN model for training, and optimize the same set of hyperparameters, which are the learning rate  $\eta$  and clipping norm  $R$ . The primary difference lies in how we generate the hyperparameter landscape. Given that a single run on CIFAR-10 is considerably more time-consuming than on MNIST, conducting multiple runs for every hyperparameter combination is unfeasible. To address this challenge, we leverage BoTorch [3],
3The  $\varepsilon$  values are seemingly very large. Nonetheless, the reported privacy budget encompasses the overall cost of the entire HPO, which is typically overlooked in the existing literature. Given that HPO roughly incurs three times the privacy cost of the base algorithm, an  $\varepsilon$  as high as 15 could be reported as only 5 in many other works.
Figure 1: Left: The accuracy of the output hyperparameter in MNIST semi-real simulation, with  $\varepsilon = 15$ ,  $\delta = 0.00001$ . Middle: The accuracy of the output hyperparameter in CIFAR-10, with  $\varepsilon = 12$ ,  $\delta = 0.00001$ . Right: The loss of the output hyperparameter in FL. Error bars stands for  $95\%$  confidence. Curves for GP are calculated by averaging 400 independent runs, and curves for Uniform are calculated by averaging 10000 independent runs. For a clearer demonstration, we compare the performance for each fixed value of  $T$ , and recognize that the actual performance is a weighted average across different values of  $T$ .
an open-sourced library for HPO, to generate the landscape. Since we operate in the white-box setting, where the base algorithms have distinct privacy budgets for the uniform and adaptive scenarios, we execute 50 runs and generate the landscape for each case, including the mean  $(\mu_{\lambda})$  and standard error  $(\sigma_{\lambda})$  of accuracy for each hyperparameter combination  $\lambda$ . When the algorithm (GP or Uniform) visits a specific  $\lambda$ , our oracle returns a noisy score  $q(\lambda)$  drawn from a normal distribution of  $N(\mu_{\lambda}, \sigma_{\lambda})$ . A more detailed description of our landscapes and parameter configuration can be found in Appendix E.2.
In the middle of Figure 1, we showcase a performance comparison between the Uniform and GP methods with a total privacy budget of  $\varepsilon = 12$  and  $\delta = 1e - 5$ . Clearly, GP consistently outperforms the Uniform method, with the largest performance gap occurring when the number of runs is around 10.
4.2.3 Federated Learning
In this section, we move to the black-box setting, where the privacy budget allocated to the base algorithm remains fixed, while we allow extra privacy budget for HPO. That being said, the adaptivity can be achieved without compromising the utility of the base algorithm.
We explore another real-world scenario: a Federated Learning (FL) task conducted on a proprietary dataset  ${}^{4}$  from industry. Our aim is to determine the optimal learning rates for the central server (using AdaGrad) and the individual users (using SGD). To simulate this scenario, we once again rely on the landscape generated by BoTorch [3], as shown in Figure 3 in Appendix E.3.
Under the assumption that base algorithms are black-box models with fixed privacy costs, we proceed with HPO while varying the degree of adaptivity. The experiment results are visualized in the right panel of Figure 1, and Table 2 presents the aggregated performance data.
We consistently observe that GP outperforms Uniform in the black-box setting. Furthermore, our findings suggest that allocating a larger privacy budget to the GP method facilitates the acquisition of adaptive information, resulting in improved performance in HPO. This highlights the flexibility of GP in utilizing privacy resources effectively.
Geometric(γ)	0.001	0.002	0.003	0.005	0.01	0.02	0.025	0.03
GP	0.946	0.948	0.948	0.947	0.943	0.937	0.934	0.932
Uniform	0.943	0.945	0.945	0.944	0.940	0.935	0.932	0.929
Table 1: Accuracy of MNIST using Geometric Distribution with various different values of  $\gamma$  for Uniform and GP methods. Each number is the mean of 200 runs.
<sup>4</sup>We have to respect confidentiality constraints that limit our ability to provide extensive details about this dataset.
Geometric(γ)	0.001	0.002	0.003	0.005	0.01	0.02	0.025	0.03
GP (C = 1.25)	0.00853	0.0088	0.00906	0.00958	0.0108	0.0129	0.0138	0.0146
GP (C = 1.33)	0.00821	0.00847	0.00872	0.00921	0.0104	0.0123	0.0132	0.0140
GP (C = 1.5)	0.00822	0.00848	0.00872	0.00920	0.0103	0.0123	0.0131	0.0130
Uniform	0.0104	0.0106	0.0109	0.0113	0.0123	0.0141	0.0149	0.0156
Table 2: Loss of FL using Geometric Distribution with various different values of  $\gamma$  for Uniform and GP methods with different choice of  $C$  and  $c = 1/C$ . Each number is the mean of 200 runs.
In conclusion, this paper presents a novel framework, DP-HyPO. As the first adaptive HPO framework with sharp DP guarantees, DP-HyPO effectively bridges the gap between private and non-private HPO. Our work encompasses the random search method by [24, 32] as a special case, while also granting practitioners the ability to adaptively learn better sampling distributions based on previous runs. Importantly, DP-HyPO enables the conversion of any non-private adaptive HPO algorithm into a private one. Our framework proves to be a powerful tool for professionals seeking optimal model performance and robust DP guarantees.
The DP-HyPO framework presents two interesting future directions. One prospect involves an alternative HPO specification which is practically more favorable. Considering the extensive literature on HPO, there is a significant potential to improve the empirical performance by leveraging more advanced HPO methods. Secondly, there is an interest in establishing a theoretical utility guarantee for DP-HyPO. By leveraging similar proof methodologies to those in Theorem 3.3 in [24], it is feasible to provide basic utility guarantees for the general DP-HyPO, or for some specific configurations within DP-HyPO.
The authors would like to thank Max Balandat for his thoughtful comments and insights that helped us improve the paper.
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Martin S Andersen, Joachim Dahl, Lieven Vandenberghe, et al. Cvxopt: A python package for convex optimization. Available at cvxopt.org, 54, 2013.
[3] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems, 33:21524-21538, 2020.
[4] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE, 2014.
[5] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in neural information processing systems, 24, 2011.
[6] Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, volume 267, 2019.
[7] Kamalika Chaudhuri and Staal A Vinterbo. A stability-based validation procedure for differentially private machine learning. Advances in Neural Information Processing Systems, 26, 2013.
A Proofs of the technical results
A.1 Proof of Main Results
First, we define Rényi divergence as follows.
Definition A.1 (Rényi Divergences). Let  $P$  and  $Q$  be probability distributions on a common space  $\Omega$ . Assume that  $P$  is absolutely continuous with respect to  $Q$  - i.e., for all measurable  $E \subset \Omega$ , if  $Q(E) = 0$ , then  $P(E) = 0$ . Let  $P(x)$  and  $Q(x)$  denote the densities of  $P$  and  $Q$  respectively. The KL divergence from  $P$  to  $Q$  is defined as
$$
\mathrm {D} _ {1} (P \| Q) := \underset {X \leftarrow P} {\mathbb {E}} \left[ \log \left(\frac {P (X)}{Q (X)}\right) \right] = \int_ {\Omega} P (x) \log \left(\frac {P (x)}{Q (x)}\right) \mathrm {d} x.
$$
The max divergence from  $P$  to  $Q$  is defined as
$$
\mathrm {D} _ {\infty} (P \| Q) := \sup  \left\{\log \left(\frac {P (E)}{Q (E)}\right): P (E) > 0 \right\}.
$$
For  $\alpha \in (1,\infty)$ , the Rényi divergence from  $P$  to  $Q$  of order  $\alpha$  is defined as
$$
\begin{array}{l} \mathrm {D} _ {\alpha} (P \| Q) := \frac {1}{\alpha - 1} \log \left(\underset {X \leftarrow P} {\mathbb {E}} \left[ \left(\frac {P (X)}{Q (X)}\right) ^ {\alpha - 1} \right]\right) \\ = \frac {1}{\alpha - 1} \log \left(\underset {X \leftarrow Q} {\mathbb {E}} \left[ \left(\frac {P (X)}{Q (X)}\right) ^ {\alpha} \right]\right) \\ = \frac {1}{\alpha - 1} \log \left(\int_ {Q} P (x) ^ {\alpha} Q (x) ^ {1 - \alpha} \mathrm {d} x\right). \\ \end{array}
$$
We now present the definition of Rényi DP (RDP) in [27].
Definition A.2 (Rényi Differential Privacy). A randomized algorithm  $M: \mathcal{X}^n \to \mathcal{Y}$  is  $(\alpha, \varepsilon)$ -Rényi differentially private if, for all neighboring pairs of inputs  $D, D' \in \mathcal{X}^n$ ,  $\mathrm{D}_{\alpha}(M(x) \| M(x')) \leq \varepsilon$ .
We define some additional notations for the sake of the proofs. In algorithm 1, for any  $1 \leq j \leq T$ , and neighboring dataset  $D$  and  $D'$ , we define the following notations for any  $y = (x,q) \in \mathcal{V}$ , the totally ordered range set.
$$
P _ {j} (y) = \mathbb {P} _ {\tilde {y} \sim Q (D, \pi^ {(j)})} (\tilde {y} = y) \quad \text {a n d} \quad P _ {j} ^ {\prime} (y) = \mathbb {P} _ {\tilde {y} \sim Q \left(D ^ {\prime}, \pi^ {\prime (j)}\right)} (\tilde {y} = y)
$$
$$
P _ {j} (\leq y) = \mathbb {P} _ {\tilde {y} \sim Q (D, \pi^ {(j)})} (\tilde {y} \leq y) \quad \text {a n d} \quad P _ {j} ^ {\prime} (\leq y) = \mathbb {P} _ {\tilde {y} \sim Q \left(D ^ {\prime}, \pi^ {\prime (j)}\right)} (\tilde {y} \leq y)
$$
$$
P _ {j} (<   y) = \mathbb {P} _ {\tilde {y} \sim Q (D, \pi^ {(j)})} (\tilde {y} <   y) \quad \text {a n d} \quad P _ {j} ^ {\prime} (<   y) = \mathbb {P} _ {\tilde {y} \sim Q \left(D ^ {\prime}, \pi^ {\prime (j)}\right)} (\tilde {y} <   y).
$$
By these definitions, we have  $P_{j}(\leq y) = P_{j}(<y) + P_{j}(y)$ , and  $P_{j}'(\leq y) = P_{j}'(<y) + P_{j}'(y)$ . And additionally, we have
$$
\begin{array}{l} \frac {P _ {j} (y)}{P _ {j} ^ {\prime} (y)} = \frac {\int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D) = y) \pi^ {(j)} (\lambda) d \lambda}{\int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) \pi^ {\prime (j)} (\lambda) d \lambda} \leq \sup _ {\lambda \in \Lambda} \frac {\mathbb {P} (M _ {\lambda} (D) = y) \pi^ {(j)} (\lambda)}{\mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) \pi^ {\prime (j)} (\lambda)} \\ \leq \frac {C}{c} \cdot \sup  _ {\lambda \in \Lambda} \frac {\mathbb {P} \left(M _ {\lambda} (D) = y\right)}{\mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) = y\right)}. \tag {A.1} \\ \end{array}
$$
Here, the first inequality follows from the simple property of integration, and the second inequality follows from the fact that  $\pi^{(j)}$  has bounded density between  $c$  and  $C$ . Similarly, we have
$$
\frac {P _ {j} (\leq y)}{P _ {j} ^ {\prime} (\leq y)} \leq \frac {C}{c} \cdot \sup  _ {\lambda \in \Lambda} \frac {\mathbb {P} \left(M _ {\lambda} (D) \leq y\right)}{\mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) \leq y\right)}, \tag {A.2}
$$
$$
\frac {P _ {j} (<   y)}{P _ {j} ^ {\prime} (<   y)} \leq \frac {C}{c} \cdot \sup  _ {\lambda \in \Lambda} \frac {\mathbb {P} \left(M _ {\lambda} (D) <   y\right)}{\mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) <   y\right)}. \tag {A.3}
$$
Note that  $D$  and  $D^{\prime}$  are neighboring datasets, and  $M_{\lambda}$  satisfies some DP guarantees. So the ratio  $\frac{\mathbb{P}(M_{\lambda}(D)\in E)}{\mathbb{P}(M_{\lambda}(D^{\prime})\in E)}$  for any event  $E$  can be bounded.
For simplicity, we define the inner product of a distribution  $\pi$  with the vector  $M(D) = (\mathbb{P}(M_{\lambda}(D) = y):\lambda \in \Lambda)$  as
$$
\pi \cdot M (D) := \int_ {\lambda \in \Lambda} \mathbb {P} \left(M _ {\lambda} (D) = y\right) \pi (\lambda) d \lambda . \tag {A.4}
$$
Now, we define additional notations to bound the probabilities. Recall  $S_{C,s}$  is given by  $\{f\in \Lambda^{\mathbb{R}^{+}}:$  ess sup  $f\leq C$ , ess inf  $f\geq c,\int_{\alpha \in \Lambda}f(\alpha)d\alpha = 1.\}$ . It is straightforward to see this is a compact set as it is the intersection of three compact sets. We define
$$
P ^ {+} (y) := \sup  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} \left(M _ {\lambda} (D) = y\right) \pi^ {(j)} (\lambda) d \lambda = \pi^ {+} \cdot M (D), \tag {A.5}
$$
where  $\pi^{+}$  is the distribution that achieves the supremce in the compact set  $S_{C,c}$ . Similarly, we define  $P^{\prime -}(y)$  for  $D^{\prime}$  as given by
$$
P ^ {\prime -} (y) := \inf  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) = y\right) \cdot \pi^ {\prime (j)} (\lambda) d \lambda = \pi^ {\prime -} \cdot M. \tag {A.6}
$$
Similarly, we can define  $P^{\prime +}(y)$  and  $P^{-}(y)$  accordingly. From the definition, we know that
$$
P ^ {-} (y) \leq P _ {j} (y) \leq P ^ {+} (y) \quad \text {a n d} \quad P ^ {\prime -} (y) \leq P _ {j} ^ {\prime} (y) \leq P ^ {\prime +} (y). \tag {A.7}
$$
$$
\frac {P ^ {+} (y)}{P ^ {\prime -} (y)} = \frac {\pi^ {*} \cdot M (D)}{\pi^ {\prime -} \cdot M \left(D ^ {\prime}\right)} \leq \sup  _ {\lambda} \frac {\mathbb {P} \left(M _ {\lambda} (D) = y\right)}{\mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) = y\right)} \cdot \frac {C}{c}. \tag {A.8}
$$
It is similar to define
$$
P ^ {+} (\leq y) := \sup  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D) \leq y) \quad \text {a n d} \quad P ^ {\prime +} (\leq y) := \sup  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D ^ {\prime}) \leq y)
$$
$$
P ^ {-} (\leq y) := \inf  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D) \leq y) \quad \text {a n d} \quad P ^ {\prime -} (\leq y) := \inf  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D ^ {\prime}) \leq y)
$$
$$
P ^ {+} (<   y) := \sup  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D) <   y) \quad \text {a n d} \quad P ^ {\prime +} (<   y) := \sup  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D ^ {\prime}) <   y)
$$
$$
P ^ {-} (<   y) := \inf  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D) <   y) \quad \text {a n d} \quad P ^ {\prime -} (<   y) := \inf  _ {\pi \in S _ {C, c}} \int_ {\lambda \in \Lambda} \mathbb {P} (M _ {\lambda} (D ^ {\prime}) <   y).
$$
Following the exact same proof, we have
$$
P ^ {-} (\leq y) \leq P _ {j} (\leq y) \leq P ^ {+} (\leq y) \quad \text {a n d} \quad P ^ {\prime -} (\leq y) \leq P _ {j} ^ {\prime} (\leq y) \leq P ^ {\prime +} (\leq y) \tag {A.9}
$$
$$
P ^ {-} (<   y) \leq P _ {j} (<   y) \leq P ^ {+} (<   y) \quad \text {a n d} \quad P ^ {\prime -} (<   y) \leq P _ {j} ^ {\prime} (<   y) \leq P ^ {\prime +} (<   y) \tag {A.10}
$$