SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, note that Theorem 4.1 offers computationally tractable approximations to the W-DRO adversarial training objectives (7) and (8). In Appendix D we propose two possible training methods but do not evaluate their performance and otherwise leave this topic to future research.
5 Performance Bounds
Understanding how a DNN classifier will perform outside the training data set is of key importance. We leverage the DRO sensitivity results now to obtain a lower bound on  $A_{\delta}$ . We then use results on convergence of empirical measures in Fournier and Guillin (2015) to translate our lower bound into guarantees on out-of-sample performance.
Bounds on Adversarial Accuracy. We propose the following metric of robustness:
$$
\mathcal {R} _ {\delta} := \frac {A _ {\delta}}{A} \in [ 0, 1 ].
$$
Previous works mostly focus on the maximum distortion a neural network can withhold to retain certain adversarial performance, see Hein and Andriushchenko (2017), Weng et al. (2018b) for
To stress the Wasserstein attack and the particular loss function we may write, e.g.,  $W$ -PGD-ReDLR.
local robustness and Bastani et al. (2016) for global robustness. However, there is no immediate connection between such a maximum distortion and the adversarial accuracy, especially in face of a distributionally adversarial attack. In contrast, since  $A = A_{0}$  is known, computing  $\mathcal{R}_{\delta}$  is equivalent to computing  $A_{\delta}$ . We choose to focus on the relative loss of accuracy as it provides a convenient normalization:  $0 \leqslant \mathcal{R}_{\delta} \leqslant 1$ .  $\mathcal{R}_{\delta} = 1$  corresponds to a very robust architecture which performs as well under attacks as it does on clean test data, while  $\mathcal{R}_{\delta} = 0$  corresponds to an architecture which loses all of its predictive power under an adversarial attack. Together the couple  $(A, A_{\delta})$  thus summarizes the performance of a given classifier. However, computing  $A_{\delta}$  is difficult and time-consuming. Below, we develop a simple and efficient method to calculate theoretical guaranteed bounds on  $\mathcal{R}$  and thus also on  $A_{\delta}$ .
Assumption 5.1. We assume that for any  $Q \in B_{\delta}(P)$
(i)  $0 < Q(S) < 1$
(ii)  $\mathcal{W}_p(Q(\cdot |S),P(\cdot |S)) + \mathcal{W}_p(Q(\cdot |S^c),P(\cdot |S^c)) = o(\delta)$ , where  $S^c = (\mathcal{X}\times \mathcal{Y})\backslash S$  and the conditional distribution is given by  $Q(E|S) = Q(E\cap S) / Q(S)$ .
The first condition stipulates non-degeneracy: the classifier does not perform perfectly but retains some accuracy under attacks. The second condition says the classes are well-separated: for  $\delta$  small enough an admissible attack can rarely succeed.
We write the adversarial loss condition on the correctly classified images and misclassified images as
$$
C(\delta) = \sup_{Q\in B_{\delta}(P)}\mathbf{E}_{Q}[J_{\theta}(x,y)|S]\quad \text{and}\quad W(\delta) = \sup_{Q\in B_{\delta}(P)}\mathbf{E}_{Q}[J_{\theta}(x,y)|S^{c}].
$$
We note that an upper bound on  $\mathcal{R}_{\delta}$  is given by any adversarial attack. In particular,
$$
\mathcal {R} _ {\delta} \leqslant \mathcal {R} _ {\delta} ^ {u} := Q _ {\delta} (S) / A. \tag {12}
$$
Theorem 5.1. Under Assumptions 4.1 and 5.1, we have an asymptotic lower bound as  $\delta \to 0$
$$
\mathcal {R} _ {\delta} \geqslant \frac {W (0) - V (\delta)}{W (0) - V (0)} + o (\delta) = \widetilde {\mathcal {R}} _ {\delta} ^ {l} + o (\delta) = \overline {{\mathcal {R}}} _ {\delta} ^ {l} + o (\delta), \tag {13}
$$
where the first order approximations are given by
$$
\widetilde {\mathcal {R}} _ {\delta} ^ {l} = \frac {W (0) - \mathbf {E} _ {Q _ {\delta}} [ J _ {\theta} (x , y) ]}{W (0) - V (0)} \quad a n d \quad \overline {{\mathcal {R}}} _ {\delta} ^ {l} = \frac {W (0) - V (0) - \delta \Upsilon}{W (0) - V (0)}. \tag {14}
$$
The equality between the lower bound and the two first-order approximations  $\widetilde{\mathcal{R}}_{\delta}^{l}$  and  $\overline{\mathcal{R}}_{\delta}^{l}$  follows from Theorem 4.1. Consequently,  $\mathcal{R}_{\delta}^{l} := \min \{\widetilde{\mathcal{R}}_{\delta}^{l}, \overline{\mathcal{R}}_{\delta}^{l}\}$  allows us to estimate the model robustness without performing any sophisticated adversarial attack. Our experiments, detailed below, show the bound is reliable for small  $\delta$  and is orders of magnitude faster to compute than  $\mathcal{R}_{\delta}$  even in the classical case of pointwise attacks. The proof is reported in Appendix A. Its key ingredient is the following tower-like property.
Proposition 5.2. Under Assumptions 4.1 and 5.1, we have
$$
V (\delta) = \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \big [ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \big ] + o (\delta).
$$
Bounds on Out-of-Sample Performance. Our results on distributionally adversarial robustness translate into bounds for performance of the trained DNN on unseen data. We rely on the results of Fournier and Guillin (2015) and refer to Lee and Raginsky (2018) for analogous applications to finite sample guarantees and to Gao (2022) for further results and discussion.
We fix  $1 < p < n/2$  and let  $N = |\mathcal{D}_{tr}|$ ,  $M = |\mathcal{D}_{tt}|$ . If sampling of data from  $P$  is described on a probability space  $(\Omega, \mathcal{F}, \mathbb{P})$  then  $\hat{P}$  is a random measure on this space and, by ergodic theorem,  $\mathbb{P}$ -a.s., it converges weakly to  $P$  as  $N \to \infty$ . In fact,  $\mathcal{W}_p(\hat{P}, P)$  converges to zero  $\mathbb{P}$ -a.s. Crucially the rates of convergence were obtained in Dereich et al. (2013), Fournier and Guillin (2015) and yield
$$
\mathbb {E} \left[ \mathcal {W} _ {p} (\hat {P}, P) \right] \leqslant K N ^ {- \frac {1}{n}} \quad \text {a n d} \quad \mathbb {P} \left(\mathcal {W} _ {p} (\hat {P}, P) \geqslant \varepsilon\right) \leqslant K \exp \left(- K N \varepsilon^ {n}\right), \tag {15}
$$
where  $K$  is a constant depending on  $p$  and  $n$  which can be computed explicitly, see for example Guo and Obloj (2019, Appendix). This, with triangle inequality and Theorem 5.1, gives
Corollary 5.3. Under Assumptions 4.1 and 5.1 on measure  $\hat{P}$ , with probability at least  $1 - 2K\exp(-K\varepsilon^n\min\{M,N\})$  it holds that
$$
\check {A} = \check {P} (S) \geqslant \hat {A} \hat {\mathcal {R}} _ {2 \varepsilon} ^ {l} + o (\varepsilon).
$$
Next results provide a finer statistical guarantee on the out-of-sample performance for robust (W-DRO) training. Its proof is reported in Appendix B.
Theorem 5.4. Under Assumption 4.1, with probability at least  $1 - K\exp (-KN\varepsilon^{n})$  we have
$$
V (\delta) \leqslant \hat {V} (\delta) + \varepsilon \sup  _ {Q \in B _ {\delta} ^ {\star} (\hat {P})} \left(\mathbf {E} _ {Q} \| \nabla_ {x} J _ {\theta} (x, y) \| _ {s} ^ {q}\right) ^ {1 / q} + o (\varepsilon) \leqslant \hat {V} (\delta) + \mathrm {L} \varepsilon
$$
where  $B_{\delta}^{\star}(\hat{P}) = \arg \max_{Q\in B_{\delta}(\hat{P})}\mathbf{E}_Q[J_\theta (x,y)]$  and constant  $K$  only depends on  $p$  and  $n$ .
Our lower bound estimate in Theorem 5.1 can be restated as
$$
\Delta \hat {A} _ {\delta} := \hat {A} - \hat {A} _ {\delta} \leqslant \frac {\hat {V} (\delta) - \hat {V} (0)}{\widehat {W} (0) - \hat {C} (0)} + o (\delta).
$$
We now use Theorem 5.4 to bound  $\Delta A(\delta)$ , the shortfall of the adversarial accuracy under  $P$ , using quantities evaluated under  $\hat{P}$ .
Corollary 5.5. Under Assumptions 4.1 and 5.1, with probability at least  $1 - K\exp (-KN\delta^n)$  it holds that
$$
\Delta A _ {\delta} (P) \leqslant \frac {\widehat {V} (\delta) - \widehat {V} (0)}{\widehat {W} (0) - \widehat {C} (0)} + \frac {2 \mathrm {L} \delta}{\widehat {W} (0) - \widehat {C} (0)} + o (\delta).
$$
We remark that the above results are easily extended to the out-of-sample performance on the test set, via the triangle inequality  $\mathcal{W}_p(\hat{P},\check{P})\leqslant \mathcal{W}_p(\hat{P},P) + \mathcal{W}_p(P,\check{P})$ . By using complexity measures such as entropy integral (Lee and Raginsky, 2018), Rademacher complexity (Gao, 2022, Gao et al., 2022) a further analysis can be undertaken for
$$
\inf  _ {\theta \in \Theta} \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} [ J _ {\theta} (x, y) ] \quad \text {a n d} \quad \inf  _ {\theta \in \Theta} \sup  _ {Q \in B _ {\delta} (\hat {P})} \mathbf {E} _ {Q} [ J _ {\theta} (x, y) ]. \tag {16}
$$
In particular, a dimension-free estimate of out-of-sample performance is obtained in (Gao, 2022) under a Lipschitz framework with light-tail reference measures.
Experimental Setting. We conduct experiments on a high performance computing server equipped with 49 GPU nodes. The algorithms are implemented in Python. Experiments are conducted on CIFAR-10, CIFAR-100, and ImageNet datasets. Numerical results are consistent across different datasets, and we present the results on CIFAR-10 in body paragraph only for the sake of brevity. Results on CIFAR-100 and ImageNet are reported in Appendix F.
CIFAR-10 (Krizhevsky, 2009) comprises 60,000 color images across 10 mutually exclusive classes, with 6,000 images per class. Each image contains  $32 \times 32$  pixels in 3 color channels. We normalize the input feature as a vector  $x \in [0,1]^{3 \times 32 \times 32}$ . The dataset is further divided into training and test sets, containing 50,000 and 10,000 images respectively. We evaluate the robustness of neural networks on the test set only.
We consider four threat models  $(\mathcal{W}_p, l_r)$  with  $p, r \in \{2, \infty\}$  with different range of attack budget  $\delta$  depending on the relative strength of the attack. E.g., roughly speaking, if an  $l_{\infty}$ -attack modifies one third of the pixels of an image with strength 4/255, then it corresponds to an  $l_2$ -attack with strength 1/2. When clear from the context, we drop the  $\delta$  subscript.
We take top neural networks from RobustBench (Croce et al., 2021), a lively maintained repository that records benchmark robust neural networks on CIFAR-10 against pointwise attacks. For pointwise threat models  $(\mathcal{W}_{\infty},l_{r})$ , RobustBench reports  $A_{\delta}$  obtained using AutoAttack (Croce and Hein, 2020) for  $l_{\infty},\delta = 8 / 255$  and  $l_{2},\delta = 1 / 2$ , see Appendix H. However, due to high computational
cost of AutoAttack, we apply PGD-50 based on CE and DLR losses as a substitute to obtain the reference adversarial accuracy for attacks with relatively small budgets  $\delta = 2/255$ ,  $4/255$  for  $l_{\infty}$  and  $\delta = 1/8$ ,  $1/4$  for  $l_{2}$ . For distributional threat models  $(\mathcal{W}_{2}, l_{r})$ , there is no existing benchmark attacking method. Therefore, W-PGD attack (11) based on ReDLR loss is implemented to obtain the reference adversarial accuracy  $A_{\delta}$ . All PGD attacks are run with 50 iteration steps and take between 1 and 12 hours to run on a single GPU environment. Bounds  $\mathcal{R}^{l}$ ,  $\mathcal{R}^{u}$  compute ca. 50 times faster.
Table 1: Comparison of adversarial accuracy of neural networks on RobustBench under different empirical attacks. Set attack budget  $\delta  = 8/{255}$  for  ${l}_{\infty }$  threat models and  $\delta  = 1/2$  for  ${l}_{2}$  threat models.
Methods	W∞	W2
AutoAttack	W-PGD-CE	W-PGD-DLR	W-PGD-ReDLR
l∞	57.66%	61.32%	79.00%	45.46%
l2	75.78%	74.62%	78.69%	61.69%
Distributionally Adversarial Attack. We report in Table 1 the average accuracy of top neural networks on RobustBench against pointwise and distributional attacks under different loss functions. The predicted drop in accuracy between a pointwise, i.e.,  $\infty$ -W-DRO attack and a distributional 2-W-DRO attack is only realized using the ReDLR loss.
In Figure 2, we compare the adversarial accuracy of robust networks on RobustBench against pointwise threat models and distributional threat models. We notice a significant drop of the adversarial accuracy even for those neural networks robust against pointwise threat models.
Figure 2: Shortfall of WD-adversarial accuracy with different metrics  $l_{\infty}$  (left) and  $l_{2}$  (right).
Bounds on Adversarial Accuracy. We report in Table 2 the computation time of our proposed bounds  $\mathcal{R}_{\delta}^{l} = \min \{\widetilde{\mathcal{R}}_{\delta}^{l},\overline{\mathcal{R}}_{\delta}^{l}\}$  in (14) and  $\mathcal{R}_{\delta}^{u}$  in (12) with the computation time of  $\mathcal{R}_{\delta}$  obtained from AutoAttack. Computing our proposed bounds  $\mathcal{R}^l,\mathcal{R}^u$  is orders of magnitude faster than performing an attack to estimate  $\mathcal{R}$ . This also holds for distributional threat attacks.
Table 2: Computation times of  $(\mathcal{W}_{\infty}, l_{\infty})$ ,  $\delta = 8/255$  attack for one mini-batch of size 100, in seconds. We compute  $\mathcal{R}$  by AutoAttack and average the computation time over models on RobustBench grouped by their architecture.
PreActResNet-18	ResNet-18	ResNet-50	WRN-28-10	WRN-34-10	WRN-70-16
R	197	175	271	401	456	2369
Rl&Ru	0.52	0.49	0.17	0.55	0.53	1.46
To illustrate the applications of Theorem 5.1, we plot the bounds  $\mathcal{R}^l$  and  $\mathcal{R}^u$  against  $\mathcal{R}$  for neural networks on RobustBench. The results are plotted in Figure 3 and showcase the applicability of our bounds across different architectures. $^6$  Note that smaller  $\delta$  values are suitable for the stronger  $\mathcal{W}_2$ -distributional attack. For pointwise threat models (top row) we compute the bounds using CE loss. For distributional threat models (bottom row), reference adversarial accuracy is obtained from a
<sup>6</sup>We use all 60 available networks on RobustBench (model zoo) for  $l_{\infty}$  and all 20 available networks for  $l_{2}$ .
W-PGD-ReDLR attack and, accordingly, we use ReDLR loss to compute  $\mathcal{R}^u$  and  $\mathcal{R}^l$ . In this case, the width of the gap between our upper and lower bounds varies significantly for different DNNs. To improve the bounds, instead of  $\mathcal{R}^l$ , we could estimate  $V(\delta)$  and use the lower bound in (13). This offers a trade-off between computational time and accuracy which is explored further in Appendix E.
Figure 3:  $\mathcal{R}^u$  &  $\mathcal{R}^l$  versus  $\mathcal{R}$ . Top row:  $\mathcal{W}_{\infty}$ -attack with bounds computed based on CE loss across neural networks on RobustBench. Bottom row:  $\mathcal{W}_2$ -attack for the same sets of neural networks with bounds computed using ReDLR loss.
Limitations. Our theoretical results are asymptotic and their validity is confined to the linear approximation regime. We believe that the empirical results we presented from all the leaderboard models on RobustBench across different attack types provide an overwhelming evidence that our results are valid and relevant for the range of attack budget  $\delta$  considered in AA settings. However, as  $\delta$  increases we are likely to go outside of the linear approximation regime, see Figure 1. In Appendix E we plot the results for pointwise attack with  $\delta = 8/255$  where some of neural networks have a lower bound  $\mathcal{R}^l$  greater than the reference  $\mathcal{R}$ . We do not have theoretical results to provide guarantees on the range of applicability but, in Appendix E, discuss a possible rule of thumb solution.
Future Work. We believe our research opens up many avenues for future work. These include: developing stronger attacks under distributional threat models, testing the performance of the two training algorithms derived here and investigating further sensitivity-based ones, as well as analyzing the relation between the values and optimizers in (16), verifying empirical performance of our out-of-sample results, including Corollary 5.5, and extending these to out-of-distribution performance.
Broader Impact. Our work contributes to the understanding of robustness of DNN classifiers. We believe it can help users in designing and testing DNN architectures. It also offers a wider viewpoint on the question of robustness and naturally links the questions of adversarial attacks, out-of-sample performance, out-of-distribution performance and Knightian uncertainty. We provide computationally efficient tools to evaluate robustness of DNNs. However, our results are asymptotic and hence valid for small attacks and we acknowledge the risk that some users may try to apply the methods outside of their applicable regimes. Finally, in principle, our work could also enhance understanding of malicious agents aiming to identify and attack vulnerable DNN-based classifiers.
The authors are grateful to Johannes Wiesel for his most helpful comments and suggestions in the earlier stages of this project. JO gratefully acknowledges the support from St John's College, Oxford. YJ's research is supported by the EPSRC Centre for Doctoral Training in Mathematics of Random Systems: Analysis, Modelling and Simulation (EP/S023925/1). XB and GH's work, part of their research internship with JO, was supported by the Mathematical Institute and, respectively, St John's College and St Anne's College, Oxford.
V. Sehwag, S. Wang, P. Mittal, and S. Jana. HYDRA: pruning adversarially robust neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 19655-19666. Curran Associates, Inc., 2020.
Z. Shen, J. Liu, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui. Towards out-of-distribution generalization: a survey, Aug. 2021.
A. Sinha, H. Namkoong, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
M. Staib and S. Jegelka. Distributionally robust deep learning as a generalization of adversarial training. In NIPS workshop on Machine Learning and Computer Security, volume 3, page 4, 2017.
V. Tjeng, K. Y. Xiao, and R. Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019.
F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: attacks and defenses. In International Conference on Learning Representations, 2018.
R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in Neural Information Processing Systems, 31, 2018.
Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2020.
Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, and S. Yan. Better diffusion models further improve adversarial training. In Proceedings of the 40th International Conference on Machine Learning, 2023.
L. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, L. Daniel, D. Boning, and I. Dhillon. Towards fast computation of certified robustness for ReLU networks. In Proceedings of the 35th International Conference on Machine Learning, pages 5276-5285. PMLR, July 2018a.
T. W. Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su, Y. Gao, C.-J. Hsieh, and L. Daniel. Evaluating the robustness of neural networks: an extreme value theory approach. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2018b.
E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the 35th International Conference on Machine Learning, pages 5286-5295. PMLR, July 2018.
D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. In Advances in Neural Information Processing Systems, volume 33, pages 2958-2969. Curran Associates, Inc., 2020.
Y. Xing, Q. Song, and G. Cheng. Why do artificially generated data help adversarial robustness. In Advances in Neural Information Processing Systems, Oct. 2022.
H. Zhang, Y. Yu, J. Jiao, E. Xing, L. E. Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, pages 7472-7482. PMLR, May 2019.
A Bounds on Adversarial Accuracy
Recall that Proposition 5.2 states the following tower-like property
$$
V (\delta) = \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] = o (\delta),
$$
Proof of Proposition 5.2. One direction follows directly from the usual tower property of conditional expectation:
$$
\begin{array}{l} V (\delta) = \sup _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} [ J (x, y) ] = \sup _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} [ \mathbf {E} _ {Q} [ J (x, y) | \sigma (S) ] ] \\ \leqslant \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right]. \\ \end{array}
$$
For the other direction, note that
$$
Q (E | S) = Q (E \cap S) / Q (S) \quad \text {a n d} \quad Q (E | S ^ {c}) = Q (E \cap S ^ {c}) / Q (S ^ {c}),
$$
are well-defined for any Borel  $E$  by Assumption 5.1. Take an arbitrary  $\varepsilon >0$  and  $Q_{c},Q_{w}\in B_{\delta}(P)$  such that
$$
\mathbf {E} _ {Q _ {c}} [ J (x, y) | S ] \geqslant C (\delta) - \varepsilon \quad \text {a n d} \quad \mathbf {E} _ {Q _ {w}} [ J (x, y) | S ^ {c} ] \geqslant W (\delta) - \varepsilon .
$$
We further take  $Q_{\star} \in B_{\delta}(P)$  such that
$$
\mathbf {E} _ {Q _ {\star}} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] \geqslant \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] - \varepsilon ,
$$
and write distribution  $\tilde{Q}$  given by
$$
\widetilde {Q} (E) = Q _ {c} (E | S) Q _ {\star} (S) + Q _ {w} (E | S ^ {c}) Q _ {\star} \left(S ^ {c}\right).
$$
These give us
$$
\begin{array}{l} \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] \leqslant \mathbf {E} _ {Q _ {\star}} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] + \varepsilon \\ \leqslant \mathbf {E} _ {Q _ {*}} \left[ \mathbf {E} _ {Q _ {c}} \left[ J _ {\theta} (x, y) | S \right] \mathbb {1} _ {S} + \mathbf {E} _ {Q _ {w}} \left[ J _ {\theta} (x, y) | S ^ {c} \right] \mathbb {1} _ {S ^ {c}} \right] + 3 \varepsilon \\ = \mathbf {E} _ {\widetilde {Q}} [ J _ {\theta} (x, y) ] + 3 \varepsilon . \\ \end{array}
$$
Recall Assumption 5.1 (ii) gives for any  $Q \in B_{\delta}(P)$
$$
\mathcal {W} _ {p} (Q (\cdot | S), P (\cdot | S)) + \mathcal {W} _ {p} (Q (\cdot | S ^ {c}), P (\cdot | S ^ {c})) = o (\delta).
$$
Now we take  $\pi_c \in \Pi(Q_\star(\cdot|S), Q_c(\cdot|S))$  such that  $\mathbf{E}_{\pi_c}[d(X,Y)^p] = \mathcal{W}_p(Q_\star(\cdot|S), Q_c(\cdot|S))^p$ , and similarly  $\pi_w \in \Pi(Q_\star(\cdot|S^c), Q_w(\cdot|S^c))$ . Then by definition of  $\tilde{Q}$ , we have  $\pi = Q_\star(S)\pi_c + Q_\star(S^c)\pi_w \in \Pi(Q_\star, \tilde{Q})$ . Moreover, we derive
$$
\begin{array}{l} \mathcal {W} _ {p} (Q _ {\star}, \tilde {Q}) ^ {p} \leqslant \mathbf {E} _ {\pi} [ d (X, Y) ^ {p} ] = Q _ {\star} (S) \mathbf {E} _ {\pi_ {c}} [ d (X, Y) ^ {p} ] + Q _ {\star} (S ^ {c}) \mathbf {E} _ {\pi_ {w}} [ d (X, Y) ^ {p} ] \\ = Q _ {\star} (S) \mathcal {W} _ {p} \left(Q _ {\star} (\cdot | S), Q _ {c} (\cdot | S)\right) ^ {p} + Q _ {\star} \left(S ^ {c}\right) \mathcal {W} _ {p} \left(Q _ {\star} (\cdot | S ^ {c}), Q _ {w} (\cdot | S ^ {c})\right) ^ {p} \\ = o \left(\delta^ {p}\right), \\ \end{array}
$$
which implies  $\mathcal{W}_p(P, \widetilde{Q}) = \delta + o(\delta)$  by triangle inequality. Hence, by L-Lipschitzness of  $J_{\theta}$  and (Bartl et al., 2021, Appendix Corollary 7.5) we obtain
$$
\sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] \leqslant V (\delta + o (\delta)) + 3 \varepsilon \leqslant V (\delta) + o (\delta) + 3 \varepsilon .
$$
Finally, by taking  $\varepsilon \to 0$ , we deduce
$$
V (\delta) = \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ C (\delta) \mathbb {1} _ {S} + W (\delta) \mathbb {1} _ {S ^ {c}} \right] + o (\delta)
$$
which concludes the proof of Proposition 5.2.
Now, we present the proof of the lower bound estimate in Theorem 5.1,