SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now establish some basic properties of our attack. The first is that, like the baseline attack, it actually achieves its target false positive rate. Unlike the baseline attack, this is no longer immediate, but can be derived from properties of the pinball loss:
Theorem 1. Fix a distribution  $\mathcal{D} \in \Delta(\mathcal{X} \times \mathcal{Y})$  over labeled examples and a model  $f$ . Suppose that the marginal distribution over  $s(x, y)$  for  $(x, y) \sim \mathcal{D}$  is continuous. Let  $\mathcal{H}$  be any class of models that is closed under additive shifts — i.e., such that for each  $q \in \mathcal{H}$  and  $\Delta \in \mathbb{R}$ , then we also have  $q' \in \mathcal{H}$  for  $q'(x) = q(x) + \Delta$ . Then for the membership inference attack  $\mathcal{A}_q$  produced by our method,  $FPR(\mathcal{A}_q) = \alpha$ .
We defer the proof to Appendix C.1. Thus by varying  $\alpha$ , we can use our attack to sweep out a curve trading off our target false positive rate with our (empirically measured) true positive rate, just as we can for the baseline attack  $\mathcal{A}_{\tau}$ .
Is this guarantee stronger than the baseline attack, and if so, in what sense? To give one perspective on this, it will be helpful to define group conditional quantile consistency, which is related to multicalibration, a concept originating from the fairness in machine learning literature [Hébert-Johnson et al., 2018, Gupta et al., 2022, Bastani et al., 2022, Jung et al., 2023, Noarov and Roth, 2023].
Definition 4. Fix a collection  $\mathcal{G}$  of group indicator functions  $g:\mathcal{X}\to \{0,1\}$  and a model  $q:\mathcal{X}\rightarrow \mathbb{R}$ .  $q$  satisfies group conditional quantile consistency with respect to a distribution  $\mathcal{P}\in \Delta (\mathcal{X}\times \mathbb{R})$ , a target quantile  $1 - \alpha$ , and the collection of groups  $\mathcal{G}$  if for every  $g\in \mathcal{G}$ :
$$
\Pr_ {(x, s) \sim \mathcal {P}} [ q (x) \leq s | g (x) = 1 ] = 1 - \alpha
$$
Group conditional quantile consistency asks that our quantile predictions be correct not just marginally over the data, but also (simultaneously) conditionally on membership in a large number of potentially intersecting groups. If we optimize pinball loss over a richer set of models (that are closed under shifts by a class of group indicator functions, rather than just constant functions), then our attack will achieve its target false positive rate even when conditioning on membership in each of the groups specified by the functions  $g \in \mathcal{G}$ , rather than just marginally. This is a stronger guarantee, as a marginal false positive rate on its own need not hold subject to additional conditioning events.
Theorem 2. Fix a distribution  $\mathcal{D} \in \Delta(\mathcal{X} \times \mathcal{Y})$  over labeled examples and a model  $f$ . Fix a collection of group indicator functions  $\mathcal{G}$  and a class of models  $\mathcal{H}$  such that:
1.  $\mathcal{H}$  is closed under shifts from  $\mathcal{G}$ : for every  $h\in \mathcal{H}$ ,  $g\in \mathcal{G}$ , and  $\eta \in \mathbb{R}$ , the function  $h^\prime (x) = h(x) + \eta g(x)$  is such that  $h^\prime \in \mathcal{H}$ .
2. The conditional distribution over  $s(x,y)$  for  $(x,y) \sim \mathcal{D}$ , conditional on  $g(x) = 1$  is continuous for all  $g \in \mathcal{G}$ .
Then for the membership inference attack  $\mathcal{A}_q$  produced by our method, its false positive rate is  $1 - \alpha$  conditional on membership in each group  $g\in \mathcal{G}$ :  $\operatorname*{Pr}_{(x,y)\sim \mathcal{D}}[\mathcal{A}_q(x,y) = \bot |g(x) = 1] = 1 - \alpha$ .
We defer the proof to Appendix C.2.
We present two sets of experiments on two different data domains, including images and tabular data. Here, we mainly focus on attacking widely-used classification models in these two domains, however, our theoretical claims generalize to other data domains as well.
We evaluate the effectiveness of our proposed approach on four image classification datasets: CIFAR-10 [Krizhevsky et al., 2009], a standard image classification dataset with 10 target classes; CIFAR-100 [Krizhevsky et al., 2009] another image classification dataset with 100 target classes; ImageNet-1k [Russakovsky et al., 2015], a substantially larger image classification task with 1000 target classes; and CINIC-10 [Darlow et al., 2018], an extension of CIFAR-10 that additionally uses images from ImageNet-1k corresponding to the original 10 target classes. To provide a realistic evaluation, we ensure our base models use common, well-performing architectures and follow standard guidelines for hyperparameter selection [He et al., 2015], including data augmentation, learning rate schedule, and l2 regularization(weight decay). For CIFAR-10 and CIFAR-100, target (classification) models include ResNet-10, ResNet-18, ResNet-34, and ResNet-50 [He et al., 2015]. For ImageNet-1k and CINIC-10, the target model is a ResNet-50. In all experiments,  $50\%$  of the dataset is used for training the target model, and, following the common standards, the resolution of the target model is  $32\times 32$  for CIFAR and CINIC datasets, and  $224\times 224$  for the ImageNet-1k dataset. The accuracy of each target model is presented in Appendix B.
To perform our membership inference attack, we train a single quantile regression model following our proposal in Eq.(1). One of the advantages of our attack is that it is model-agnostic: since it does not require the knowledge of the model architecture of the target or the knowledge of the training algorithm, we use the same model architecture for our quantile regression model in all settings: a pretrained ConvNext-Tiny model [Liu et al., 2022]. On CINIC-10 we additionally experimented with a ResNet-50 model as our quantile model architecture.
For the scoring function, we use the hinge score proposed in Carlini et al. [2022]  $s_{hinge}(x,y) = z_y(x) - \max_{y' \neq y} z_{y'}(x)$ . This scoring rule is closely related to the logit function of the model's confidence  $s(x,y) = \log (f_y(x)) - \log (1 - f_y(x))$  for models with high label confidence, and has
been empirically shown to be approximately normally distributed. On the largest dataset, ImageNet-1k, we directly train a quantile regression model to predict (log-spaced) quantiles at  $\alpha \in [0.9996, 1]$ . An alternative is to learn a parameterized model so that, for each sample, the model predicts the mean  $\mu(x)$  and the log of the standard deviation  $\log \sigma(x)$ , and the quantiles of a sample can be generated from the Gaussian distribution  $\mathcal{N}(\mu(x), (e^{\log \sigma(x)})^2)$ . Due to the fact that CIFAR and CINIC datasets are much smaller (25000 samples available for training on CIFAR), rather than directly learning quantiles, we opt to learn the parametrized models with Gaussian parameters on these datasets.
All images are processed at  $224 \times 224$  resolution by the quantile model, which is trained on the remaining  $50\%$  of the training samples that were not used to train the target model. Since there is a smaller body of literature on stable hyperparameters for regression models, we use Ray Tune [Liaw et al., 2018] for hyperparameter tuning (tuning is used to minimize validation pinball loss in a held out dataset). The compute budget for our attack was approximately 30 GPU minutes per quantile regression attack (4 hours including hyperparameter optimization) on CIFAR-10/CIFAR-100, 18 minutes (4 hours 40 minutes including hyperparameter optimization) on CINIC-10, and 16 hours (128 hours including hyperparameter optimization) on ImageNet-1k. Final hyperparameters were found to be consistent across all tasks sharing an architecture; more information is provided in Appendix D
Figures 1 and 2 shows ROC curves of our proposed approach on ImageNet-1k and CINIC-10 respectively; FPR is computed on a held-out dataset that was not used to train the target or the quantile regression model. Both the marginal quantile approach from Yeom et al. [2018] and the shadow model approach LIRA from Carlini et al. [2022] are also shown for reference. In these experiments, our quantile regression approach dominates the shadow model approach at all comparison points for ImageNet-1k, and is roughly comparable to 4 shadow models on CINIC-10. Appendix E shows the same comparison against all CIFAR-10/100 target models (ResNet-10, ResNet18, ResNet34, ResNet-50). In these latter experiments, our attack outperforms the marginal baseline but falls short of the shadow model approach. We note that in this case, the shadow models actually produce thresholds that have lower pinball loss than our quantile regression algorithm, suggesting that on the smaller CIFAR datasets, our optimization heuristic was unable to sufficiently minimize test pinball loss. We find this observation to be consistent across experiments, that the attack with smaller pinball loss on public data is generally the attack with the best TPR at that FPR level  $\alpha$
Figure 2: Comparing the true positive rate vs. false positive rate of our membership inference attack (with parametric Gaussian loss) on the CINIC-10 dataset (an extension of the CIFAR-10 dataset with 270,000 images, 4.5 times that of CIFAR-10) with the marginal baseline proposed by Yeom et al. [2018] and the state-of-the-art LiRA proposed in Carlini et al. [2022] evaluated at 2, 4, 6, and 8 shadow models. Both the model under attack and our quantile regression model use a ResNet-50 architecture. Both LiRA and our method are able to reliably identify train samples at very low false positive rates.
Table 1: Precision of all membership inference attack at  $1\%$  and  $0.1\%$  false positive rates on ResNet-50 architectures. Our attack consistently dominates the marginal baseline and produces excellent results on ImageNet-1k compared to shadow model approaches. We do not beat the shadow model approach on the smaller CIFAR datasets, potentially owing to their small dataset size. On CINIC-10, a larger dataset than CIFAR, we obtain comparable results to 4 shadow models. Additional results for the remaining architectures are presented in Appendix E
Precision @ 1% FPR	Precision @ 0.1% FPR
Method	C-10	C-100	CIN10	IN-1k	C-10	C-100	CIN10	IN-1k
Marginal	48.56%	58.81%	49.02%	47.62%	60.94%	65.75%	45.76%	46.81%
LIRA (n=2)	78.55%	95.21%	55.43%	62.70%	83.18%	98.65%	54.07%	56.04%
LIRA (n=4)	80.52%	95.87%	78.71%	89.11%	91.48%	98.94%	86.99%	95.18%
LIRA (n=6)	83.19%	96.20%	88.75%	93.74%	93.17%	99.02%	96.40%	98.38%
LIRA (n=8)	83.00%	96.07%	91.86%	94.57%	93.70%	98.98%	97.94%	98.73%
Ours	62.95%	79.57%	76.67%	97.45%	64.48%	85.41%	85.46%	99.64%
Table 1 shows precision of the proposed membership inference attack at  $1\%$  and  $0.1\%$  false positive rate on the ResNet-50 target networks (additional results shown in Appendix E). Our attack robustly predicts membership in the private dataset with high precision even at low FPR. The proposed approach works on all target architectures and datasets, but works particularly well on the more complex and data intensive ImageNet-1k task.
Since LIRA produces an explicit score distribution  $\mathcal{N}(s(x,y); \mu(x,y), \sigma(x,y))$  based on the shadow model's predictions, we can compare all 3 methods (Ours, LIRA, marginal baseline) in terms of pinball loss on public data ( $x \sim \mathcal{D}$ ). We find that the attack with the smallest pinball loss on public data is the better membership inference attack across all datasets. This shows that a strong quantile predictor on public data is a strong membership inference attack; which validates the core premise of our approach in Section 3.
A possible explanation for the relative lack of success of directly learning a quantile regression model on CIFAR datasets could be the relatively low number of available samples (25000). In such low data scenarios, training shadow models is also computationally affordable. The opposite holds true for the much larger ImageNet1k dataset, on which the computational cost of training a single target model, much less multiple shadow models far exceeds the cost of a single quantile regression model.
4.2 Tabular Classification Experiments
In addition to experiments on image datasets, we here demonstrate the effectiveness of our membership inference attack on tabular datasets, including large datasets from derived from the US Census' American Community Survey (ACS) [Ding et al., 2021] and small ones from OpenML  $^{4}$  [Grinsztajn et al., 2022]. Gradient boosting with decision trees is widely-used for classification tasks with tabular data, so in our experiments, to achieve a reasonable performance, we train a gradient boosting model with 5-fold cross validation for hyperparameter tuning on the private portion of the data. For our attack model, gradient boosting with regression trees is applied, but now with regression targets as mentioned above. Hyperparameters for regression tasks are also tuned using a public portion of the data to avoid overfitting. In our experiments, catboost is used for model training, and Optuna [Akiba et al., 2019] is used for hyperparameter tuning.
Table 2 shows that our attack, which involves learning a single regression model, performs on par with the LiRA attack, which requires learning at least 16 models on some tasks and more on other tasks. Since each model, including our regression model and a shadow model in LiRA, has the same latency in terms of hyperparameter tuning and model training, our attack requires significantly less compute (equivalent to a single shadow model), and it reduces a successful attack from training many models to only one model. Figure 3 shows ROC curves of our proposed approach on OpenML.
We have introduced a new family of membership inference attacks that are competitive with the state of the art (and in our ImageNet experiments, substantially and uniformly better), while requiring
<sup>4</sup>https://www.openml.org
Table 2: Precision @ 1% FPR on OpenML datasets and 0.5% on ACS datasets. Evaluating at different FPR levels is due to the fact that OpenML datasets have around 5,000 samples, and ACS datasets have around 20,000 samples.  $n$  here denotes the number of shadow models trained for LIRA.
Precision @ 1% FPR (OpenML)	Precision @ 0.5% FPR (ACS NY)
Task ID	361057	361064	361067	361070	Coverage	Income	Travel	Mobility
LIRA (n=16)	70.33%	85.44%	85.52%	73.33%	66.07%	50.71%	66.07%	72.74%
LIRA (n=32)	76.22%	88.52%	89.62%	78.35%	66.28%	52.53%	67.52%	68.84%
LIRA (n=64)	82.73%	90.31%	89.46%	79.57%	69.23%	50.24%	65.64%	65.26%
Ours	83.35%	88.05%	87.35%	86.54%	67.31%	56.35%	63.98%	85.27%
(a) OpenML Datasets
(b) ACS Task on NY data
Figure 3: Comparing the true positive rate vs. false positive rate of our membership inference attack against LIRA evaluated at 16, 32, and 64 shadow models on OpenML and ACS datasets. A single quantile regression model can produce similar results as multiple shadow models at a fraction of the compute cost for gradient boosting models.
substantially fewer computational resources and less knowledge of the target model. Moreover, we have identified pinball loss as a key target objective: uniformly across all of our experiments, the methods that produce thresholds minimizing pinball loss are the most effective attacks. Together, this brings membership inference closer to practicality on large commercial models. This serves to highlight a growing risk to privacy—but also provides a more efficient means to audit models by subjecting them to our attacks. We hope that our methods encourage and enable a more widespread practice of auditing models for privacy violations by subjecting them to membership inference attacks before deployment.
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
O. Bastani, V. Gupta, C. Jung, G. Noarov, R. Ramalingam, and A. Roth. Practical adversarial multivalid conformal prediction. In Neural Information Processing Systems (NeurIPS), 2022.
J. Bergstra, D. Yamins, and D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115-123. PMLR, 2013.
G. Noarov and A. Roth. The scope of multicalibration: Characterizing multicalibration via property elicitation. International Conference on Machine Learning (ICML), 2023.
A. Roth. Uncertain: Modern topics in uncertainty estimation. https://www.cis.upenn.edu/ aaroth/uncertainty-notes.pdf, 2022.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.
A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages 5558-5567. PMLR, 2019.
A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes. M-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
L. Song and P. Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium, volume 1, page 4, 2021.
L. Watson, C. Guo, G. Cormode, and A. Sablayrolles. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440, 2021.
Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and T. Goldstein. Canary in a coalmine: Better membership inference with ensembled adversarial queries. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=b7SBTEBFnC.
J. Ye, A. Maddi, S. K. Murakonda, and R. Shokri. Enhanced membership inference attacks against machine learning models. CoRR, abs/2111.09679, 2021. URL https://arxiv.org/abs/2111.09679.
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
A Shadow Model Architecture Mismatch
Here we explore how a shadow model membership inference attack is affected by the lack of knowledge of the target model. For this, we vary the shadow model's and target model's architecture between 6 different configurations: 3 CNN models with 32 filters each and varying number of pooling layers between 1 and 3, 4 Wide Residual Networks Zagoruyko and Komodakis [2016] of depth 28 and varying width ([1, 2, 5, 10]). We train a single target model and 4 shadow models; all models are trained with SGD with momentum and random augmentations. The results are shown in Figure 4, where small architecture mismatches (i.e., same model family of target and test architecture) generally degrade performance very little, but larger architecture mismatches can cause significant performance degradation.
(a) CIFAR-10
Figure 4: Comparing the precision at  $0.1\%$  false positive rate of the LIRA attack on CIFAR-10 and CIFAR-100 on mismatched target and shadow model architectures. Red circles are used to denote scenarios where target and shadow model architectures match.
B Target Model Accuracies
Here we summarize the accuracy of all target model architectures.
Architecture
Dataset	ResNet-10 Acc	ResNet-18 Acc	ResNet-34 Acc	ResNet-50 Acc
CIFAR-10	90.3%	90.5%	90.7%	91.0%
CIFAR-100	66.7%	68.6%	68.3%	68.6%
ImageNet-1k	-	-	-	67.5%
Table 3: Accuracy of target classifiers. All target classifiers are trained on  $50\%$  of the usual training data split using the training setup described in He et al. [2015]. Due to the reduced amount of training data, these target networks have lower accuracies than those initially reported in He et al. [2015].
Proof. To establish this theorem, the following lemma will be useful, which informally states that if we can shift a model's quantile predictions to get its "marginal coverage rate" to be closer to the target  $1 - \alpha$ , then we will also decrease the pinball loss of that model:
Lemma 3 (Jung et al. [2023], Roth [2022]). Fix any continuous distribution  $\mathcal{P} \in \Delta \mathbb{R}$  with density bounded by  $\rho$ . Suppose  $\hat{q}$  is a model such that  $\operatorname{Pr}_{y \sim \mathcal{P}}[y \leq \hat{q}(x)] = 1 - \alpha'$ , and let  $\Delta \in \mathbb{R}$  be such that  $\operatorname{Pr}_{y \sim \mathcal{P}}[y \leq \hat{q}(x) + \Delta] = 1 - \alpha$ . Let  $q(x) = \hat{q}(x) + \Delta$ . Then:
$$
\underset {y \sim \mathcal {P}} {\mathbb {E}} \left[ P B _ {1 - \alpha} (\hat {q} (x), y) \right] - \underset {y \sim \mathcal {P}} {\mathbb {E}} \left[ P B _ {1 - \alpha} (q (x), y) \right] \geq \frac {(\alpha - \alpha^ {\prime}) ^ {2}}{2 \rho}
$$
Recall that  $q$  is chosen such that:
$$
q\in \arg \min_{q^{\prime}\in \mathcal{H}}\underset {(x,y)\sim \mathcal{D}}{\mathbb{E}}\left[\mathrm{PB}_{1 - \alpha}(q(x),s(x,y))\right]
$$
For point of contradiction, suppose that  $\mathrm{FPR}(\mathcal{A}_q) = \alpha'$  for some  $\alpha' \neq \alpha$ . Expanding out the definition of the false positive rate, we have that:
$$
\begin{array}{l} \alpha^ {\prime} \quad = \quad \operatorname {F P R} (\mathcal {A} _ {q}) \\ = \Pr_ {(x, y) \sim \mathcal {D}} [ \mathcal {A} _ {q} (x, y) = \bot ] \\ = \operatorname * {P r} _ {(x, y) \sim \mathcal {D}} [ s (x, y) \geq q (x) ] \\ = 1 - \Pr_ {(x, y) \sim \mathcal {D}} [ s (x, y) \leq q (x) ] \\ \end{array}
$$
So  $\operatorname{Pr}_{(x,y) \sim \mathcal{D}}[s(x,y) \leq q(x)] = 1 - \alpha'$ . Let  $\Delta \in \mathbb{R}$  be such that  $\operatorname{Pr}_{(x,y) \sim \mathcal{D}}[s(x,y) \leq q(x) + \Delta] = 1 - \alpha$ . Note that such a  $\Delta$  is guaranteed to exist by continuity of the distribution on  $s(x,y)$ . Let  $q'(x) = q(x) + \Delta$ . By Lemma 3,
$$
\underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] <   \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q (x), s (x, y)\right) \right]
$$
But because  $\mathcal{H}$  is closed under additive shifts, we also have that  $q' \in \mathcal{H}$ . Together, these contradict the optimality of  $q$  as measured by pinball loss, which completes the proof.
Proof. Recall that  $q$  is chosen such that:
For point of contradiction, suppose that there is some  $g \in \mathcal{G}$  and some  $\alpha' \neq \alpha$  such that  $\mathcal{A}_q$ 's false positive rate conditional on  $g(x) = 1$  is  $\alpha'$ . Let  $\mathcal{D}_g$  be the conditional distribution on  $\mathcal{D}$  conditional on  $g(x) = 1$ , and let  $\mathcal{D}_{\bar{g}}$  be the conditional distribution on  $\mathcal{D}$  conditional on  $g(x) = 0$ . Expanding out definitions, we have that:
$$
\begin{array}{l} \alpha^{\prime}\quad = \operatorname *{Pr}_{(x,y)\sim \mathcal{D}}[\mathcal{A}_{q}(x,y) = \bot |g(x) = 1] \\ = \Pr_ {(x, y) \sim \mathcal {D} _ {g}} [ s (x, y) \geq q (x) ] \\ = 1 - \Pr_ {(x, y) \sim \mathcal {D} _ {g}} [ s (x, y) \leq q (x) ] \\ \end{array}
$$
So  $\operatorname{Pr}_{(x,y) \sim \mathcal{D}_g}[s(x,y) \leq q(x)] = 1 - \alpha'$ . Let  $\eta \in \mathbb{R}$  be such that  $\operatorname{Pr}_{(x,y) \sim \mathcal{D}_g}[s(x,y) \leq q(x) + \eta] = 1 - \alpha$ . Note that such an  $\eta$  is guaranteed to exist by continuity of the distribution on  $s(x,y)$  conditional on  $g(x) = 1$ . Let  $q'(x) = q(x) + \eta g(x)$ . By Lemma 3,
$$
\underset {(x, y) \sim \mathcal {D} _ {g}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] <   \underset {(x, y) \sim \mathcal {D} _ {g}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q (x), s (x, y)\right) \right]
$$