SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on Lemma 3, the proof of Theorem 1 is given.
Proof of Theorem 1. To begin with, it can be noticed that  $\mathbb{E}_{p_{+}(\boldsymbol{x})}[\ell(g(\boldsymbol{x}), +1)] = \mathbb{E}_{p_{+}(\boldsymbol{x}^{\prime})}[\ell(g(\boldsymbol{x}^{\prime}), +1)]$  and  $\mathbb{E}_{p_{-}(\boldsymbol{x})}[\ell(g(\boldsymbol{x}), -1)] = \mathbb{E}_{p_{-}(\boldsymbol{x}^{\prime})}[\ell(g(\boldsymbol{x}^{\prime}), -1)]$ . Then, by summing up all the equations from Eq. (18) to Eq. (21), we can get the following equation:
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \mathcal {L} _ {+} (g (\boldsymbol {x}), g (\boldsymbol {x} ^ {\prime})) + \mathcal {L} _ {-} (g (\boldsymbol {x}), g (\boldsymbol {x} ^ {\prime})) ] \\ = 2 \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ] + 2 \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ] \\ \end{array}
$$
After dividing each side of the equation above by 2, we can obtain Theorem 1.
B Analysis on Variance of Risk Estimator
B.1 Proof of Lemma 1
Based on Lemma 3, it can be observed that
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \mathcal {L} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) ] = \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} \left[ \left(\pi_ {+} - c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})\right) \ell (g (\boldsymbol {x}), + 1) + \left(\pi_ {-} - c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})\right) \ell (g (\boldsymbol {x} ^ {\prime}), - 1) \right] \\ = \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ] + \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x} ^ {\prime})} [ \ell (g (\boldsymbol {x} ^ {\prime}), - 1) ] \\ = \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ] + \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ] \\ = R (g) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \mathcal {L} (\boldsymbol {x} ^ {\prime}, \boldsymbol {x}) ] = \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\pi_ {+} + c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x} ^ {\prime}), + 1) + (\pi_ {-} + c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x}), - 1) ] \\ = \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ] + \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x} ^ {\prime})} [ \ell (g (\boldsymbol {x} ^ {\prime}), + 1) ] \\ = \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ] + \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ] \\ = R (g). \\ \end{array}
$$
Therefore, for an arbitrary weight  $\alpha \in [0,1]$
$$
\begin{array}{l} R (g) = \alpha R (g) + (1 - \alpha) R (g) \\ = \alpha \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \mathcal {L} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) ] + (1 - \alpha) \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \mathcal {L} (\boldsymbol {x} ^ {\prime}, \boldsymbol {x}) ], \\ \end{array}
$$
which indicates that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} (\alpha \mathcal {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}\right) + (1 - \alpha) \mathcal {L} \left(\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i}\right))
$$
is also an unbiased risk estimator and concludes the proof.
In this subsection, we show that Eq. (8) in the main paper achieves the minimum variance of
$$
S (g; \alpha) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\alpha \mathcal {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}\right) + (1 - \alpha) \mathcal {L} \left(\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i}\right)\right)
$$
w.r.t. any  $\alpha \in [0,1]$ . To begin with, we introduce the following notations:
$$
\mu_ {1} \triangleq \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime})) ^ {2} ] = \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i})) ^ {2} ],
$$
$$
\mu_ {2} \triangleq \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}) \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i}) ].
$$
Furthermore, according to Lemma 1 in the main paper, we have
$$
\mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ S (g; \alpha) ] = R (g).
$$
Then, we provide the proof of Theorem 2 as follows.
Proof of Theorem 2.
$$
\begin{array}{l} \operatorname {V a r} (S (g; \alpha)) = \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (S (g; \alpha) - R (g)) ^ {2} ] \\ = \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ S (g; \alpha) ^ {2} ] - R (g) ^ {2} \\ = \alpha^ {2} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime})) ^ {2} ] + (1 - \alpha) ^ {2} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i})) ^ {2} ] \\ + 2 \alpha (1 - \alpha) \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}) \sum_ {i = 1} ^ {n} \mathcal {L} (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i}) ] - R (g) ^ {2} \\ = \mu_ {1} \alpha^ {2} + \mu_ {1} (1 - \alpha) ^ {2} + 2 \mu_ {2} \alpha (1 - \alpha) - R (g) ^ {2} \\ = (2 \mu_ {1} - 2 \mu_ {2}) (\alpha - \frac {1}{2}) ^ {2} + \frac {1}{2} (\mu_ {1} + \mu_ {2}) - R (g) ^ {2}. \\ \end{array}
$$
Besides, it can be observed that
$$
2 \mu_ {1} - 2 \mu_ {2} = \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\frac {1}{n} \sum_ {i = 1} ^ {n} (\mathcal {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}) - \mathcal {L} (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i})) ^ {2} ] \geq 0.
$$
Therefore,  $\operatorname{Var}(S(g; \alpha))$  achieves the minimum value when  $\alpha = 1/2$ , which concludes the proof.
C Proof of Theorem 3
To begin with, we give the definition of Rademacher complexity.
Definition 2 (Rademacher complexity). Let  $\mathcal{X}_n = \{\pmb{x}_1, \dots, \pmb{x}_n\}$  denote  $n$  i.i.d. random variables drawn from a probability distribution with density  $p(\pmb{x})$ ,  $\mathcal{G} = \{g : \mathcal{X} \mapsto \mathbb{R}\}$  denote a class of measurable functions, and  $\sigma = (\sigma_1, \sigma_2, \dots, \sigma_n)$  denote Rademacher variables taking values from  $\{+1, -1\}$  uniformly. Then, the (expected) Rademacher complexity of  $\mathcal{G}$  is defined as
$$
\mathfrak {R} _ {n} (\mathcal {G}) = \mathbb {E} _ {\mathcal {X} _ {n}} \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {g \in \mathcal {G}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} g \left(\boldsymbol {x} _ {i}\right) \right]. \tag {22}
$$
Let  $\mathcal{D}_n\stackrel {\mathrm{i.i.d.}}{\sim}p(\boldsymbol {x},\boldsymbol{x}^{\prime})$  denote  $n$  pairs of ConfDiff data and  $\mathcal{L}_{\mathrm{CD}}(g;\boldsymbol {x}_i,\boldsymbol{x}_i^{\prime}) = (\mathcal{L}(\boldsymbol {x},\boldsymbol{x}^{\prime}) + \mathcal{L}(\boldsymbol{x}^{\prime},\boldsymbol {x})) / 2$  then we introduce the following lemma.
Lemma 4.
$$
\bar {\Re} _ {n} (\mathcal {L} _ {\mathrm {C D}} \circ \mathcal {G}) \leq 2 L _ {\ell} \Re_ {n} (\mathcal {G}),
$$
where  $\mathcal{L}_{\mathrm{CD}}\circ \mathcal{G} = \{\mathcal{L}_{\mathrm{CD}}\circ g|g\in \mathcal{G}\}$  and  $\tilde{\Re}_n(\cdot)$  is the Rademacher complexity over ConfDiff data pairs  $\mathcal{D}_n$  of size  $n$ .
$$
\begin{array}{l} \bar {\mathfrak {R}} _ {n} \left(\mathcal {L} _ {\mathrm {C D}} \circ \mathcal {G}\right) = \mathbb {E} _ {\mathcal {D} _ {n}} \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {g \in \mathcal {G}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \mathcal {L} _ {\mathrm {C D}} \left(g; \boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {\mathcal {D} _ {n}} \mathbb {E} _ {\boldsymbol {\sigma}} [ \sup  _ {g \in \mathcal {G}} \frac {1}{2 n} \sum_ {i = 1} ^ {n} \sigma_ {i} ((\pi_ {+} - c _ {i}) \ell (g (\boldsymbol {x} _ {i}), + 1) + (\pi_ {-} - c _ {i}) \ell (g (\boldsymbol {x} _ {i} ^ {\prime}), - 1) \\ + \left(\pi_ {+} + c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i} ^ {\prime}\right), + 1\right) + \left(\pi_ {-} + c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i}\right), - 1)\right) ]. \\ \end{array}
$$
Then, we can induce that
$$
\begin{array}{l} \| \nabla \mathcal {L} _ {\mathrm {C D}} (g; \boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}) \| _ {2} \\ = \| \nabla \left(\frac {\left(\pi_ {+} - c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i}\right) , + 1\right) + \left(\pi_ {-} - c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i} ^ {\prime}\right) , - 1\right)}{2} \right. \\ + \frac {\left(\pi_ {+} + c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i} ^ {\prime}\right) , + 1\right) + \left(\pi_ {-} + c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i}\right) , - 1\right)}{2}) \| _ {2} \\ \leq \| \nabla (\frac {(\pi_ {+} - c _ {i}) \ell (g (\boldsymbol {x} _ {i}) , + 1)}{2}) \| _ {2} + \| \nabla (\frac {(\pi_ {-} - c _ {i}) \ell (g (\boldsymbol {x} _ {i} ^ {\prime}) , - 1)}{2}) \| _ {2} \\ + \| \nabla (\frac {(\pi_ {+} + c _ {i}) \ell (g (\boldsymbol {x} _ {i} ^ {\prime}) , + 1)}{2}) \| _ {2} + \| \nabla (\frac {(\pi_ {-} + c _ {i}) \ell (g (\boldsymbol {x} _ {i}) , - 1)}{2}) \| _ {2} \\ \leq \frac {\left| \pi_ {+} - c _ {i} \right| L _ {\ell}}{2} + \frac {\left| \pi_ {-} - c _ {i} \right| L _ {\ell}}{2} + \frac {\left| \pi_ {+} + c _ {i} \right| L _ {\ell}}{2} + \frac {\left| \pi_ {-} + c _ {i} \right| L _ {\ell}}{2}. \tag {23} \\ \end{array}
$$
Suppose  $\pi_{+} \geq \pi_{-}$ , the value of RHS of Eq. (23) can be determined as follows: when  $c_{i} \in [-1, -\pi_{+})$ , the value is  $-2c_{i}L_{\ell}$ ; when  $c_{i} \in [-\pi_{+}, -\pi_{-})$ , the value is  $(\pi_{+} - c_{i})L_{\ell}$ ; when  $c_{i} \in [-\pi_{-}, \pi_{-})$ , the value is  $L_{\ell}$ ; when  $c_{i} \in [\pi_{-}, \pi_{+})$ , the value is  $(\pi_{+} + c_{i})L_{\ell}$ ; when  $c_{i} \in [\pi_{+}, 1]$ , the value is  $2c_{i}L_{\ell}$ . To sum up, when  $\pi_{+} \geq \pi_{-}$ , the value of RHS of Eq. (23) is less than  $2L_{\ell}$ . When  $\pi_{+} \leq \pi_{-}$ , we can
deduce that the value of RHS of Eq. (23) is less than  $2L_{\ell}$  in the same way. Therefore,
$$
\begin{array}{l} \bar {\mathfrak {R}} _ {n} (\mathcal {L} _ {\mathrm {C D}} \circ \mathcal {G}) \leq 2 L _ {\ell} \mathbb {E} _ {\mathcal {D} _ {n}} \mathbb {E} _ {\boldsymbol {\sigma}} [ \sup _ {g \in \mathcal {G}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} g (\boldsymbol {x} _ {i}) ] \\ = 2 L _ {\ell} \mathbb {E} _ {\mathcal {X} _ {n}} \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {g \in \mathcal {G}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} g (\boldsymbol {x} _ {i}) \right] \\ = 2 L _ {\ell} \Re_ {n} (\mathcal {G}), \\ \end{array}
$$
After that, we introduce the following lemma.
Lemma 5. The inequality below hold with probability at least  $1 - \delta$ :
$$
\sup _ {g \in \mathcal {G}} | R (g) - \widehat {R} _ {\mathrm {C D}} (g) | \leq 4 L _ {\ell} \Re_ {n} (\mathcal {G}) + 2 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}}.
$$
Proof. To begin with, we introduce  $\Phi = \sup_{g\in \mathcal{G}}(R(g) - \widehat{R}_{\mathrm{CD}}(g))$  and  $\bar{\Phi} = \sup_{g\in \mathcal{G}}(R(g) - \widehat{R}_{\mathrm{CD}}(g))$ , where  $\widehat{R}_{\mathrm{CD}}(g)$  and  $\widehat{R}_{\mathrm{CD}}(g)$  denote the empirical risk over two sets of training examples with exactly one different point  $\{(x_i,x_i'),c_i\}$  and  $\{(\bar{x}_i,\bar{x}_i'),c(\bar{x}_i,\bar{x}_i')\}$  respectively. Then we have
$$
\begin{array}{l} \bar {\Phi} - \Phi \leq \sup  _ {g \in \mathcal {G}} (\widehat {R} _ {\mathrm {C D}} (g) - \widehat {\bar {R}} _ {\mathrm {C D}} (g)) \\ \leq \sup _ {g \in \mathcal {G}} (\frac {\mathcal {L} _ {\mathrm {C D}} (g ; \boldsymbol {x} _ {i} , \boldsymbol {x} _ {i} ^ {\prime}) - \mathcal {L} _ {\mathrm {C D}} (g ; \bar {\boldsymbol {x}} _ {i} , \bar {\boldsymbol {x}} _ {i} ^ {\prime})}{n}) \\ \leq \frac {2 C _ {\ell}}{n}. \\ \end{array}
$$
Accordingly,  $\Phi -\bar{\Phi}$  can be bounded in the same way. The following inequalities hold with probability at least  $1 - \delta /2$  by applying McDiarmid's inequality:
$$
\sup _ {g \in \mathcal {G}} (R (g) - \widehat {R} _ {\mathrm {C D}} (g)) \leq \mathbb {E} _ {\mathcal {D} _ {n}} [ \sup _ {g \in \mathcal {G}} (R (g) - \widehat {R} _ {\mathrm {C D}} (g)) ] + 2 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}},
$$
Furthermore, we can bound  $\mathbb{E}_{\mathcal{D}_n}[\sup_{g\in \mathcal{G}}(R(g) - \widehat{R}_{\mathrm{CD}}(g))]$  with Rademacher complexity. It is a routine work to show by symmetrization [60] that
$$
\mathbb {E} _ {\mathcal {D} _ {n}} [ \sup _ {g \in \mathcal {G}} (R (g) - \widehat {R} _ {\mathrm {C D}} (g)) ] \leq 2 \bar {\Re} _ {n} (\mathcal {L} _ {\mathrm {C D}} \circ \mathcal {G}) \leq 4 L _ {\ell} \Re_ {n} (\mathcal {G}),
$$
where the second inequality is from Lemma 4. Accordingly,  $\sup_{g\in \mathcal{G}}(\widehat{R}_{\mathrm{CD}}(g) - R(g))$  has the same bound. By using the union bound, the following inequality holds with probability at least  $1 - \delta$ :
$$
\sup  _ {g \in \mathcal {G}} | R (g) - \widehat {R} _ {\mathrm {C D}} (g) | \leq 4 L _ {\ell} \Re_ {n} (\mathcal {G}) + 2 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}},
$$
Finally, the proof of Theorem 3 is provided.
$$
\begin{array}{l} R \left(\widehat {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right) = \left(R \left(\widehat {g} _ {\mathrm {C D}}\right) - \widehat {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right)\right) + \left(\widehat {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right) - \widehat {R} _ {\mathrm {C D}} \left(g ^ {*}\right)\right) + \left(\widehat {R} _ {\mathrm {C D}} \left(g ^ {*}\right) - R \left(g ^ {*}\right)\right) \\ \leq \left(R (\widehat {g} _ {\mathrm {C D}}) - \widehat {R} _ {\mathrm {C D}} (\widehat {g} _ {\mathrm {C D}})\right) + \left(\widehat {R} _ {\mathrm {C D}} (g ^ {*}) - R (g ^ {*})\right) \\ \leq \left| R \left(\widehat {g} _ {\mathrm {C D}}\right) - \widehat {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right) \right| + \left| \widehat {R} _ {\mathrm {C D}} \left(g ^ {*}\right) - R \left(g ^ {*}\right) \right| \\ \leq 2\sup_{g\in \mathcal{G}}|R(g) - \widehat{R}_{\mathrm{CD}}(g)| \\ \leq 8 L _ {\ell} \Re_ {n} (\mathcal {G}) + 4 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}}. \\ \end{array}
$$
The first inequality is derived because  $\widehat{g}_{\mathrm{CD}}$  is the minimizer of  $\widehat{R}_{\mathrm{CD}}(g)$ . The last inequality is derived according to Lemma 5, which concludes the proof.
D Proof of Theorem 4
To begin with, we provide the following inequality:
Then, we deduce the following inequality:
$$
\begin{array}{l} R \left(\bar {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right) = \left(R \left(\bar {g} _ {\mathrm {C D}}\right) - \widehat {R} _ {\mathrm {C D}} \left(\bar {g} _ {\mathrm {C D}}\right)\right) + \left(\widehat {R} _ {\mathrm {C D}} \left(\bar {g} _ {\mathrm {C D}}\right) - \bar {R} _ {\mathrm {C D}} \left(\bar {g} _ {\mathrm {C D}}\right)\right) + \left(\bar {R} _ {\mathrm {C D}} \left(\bar {g} _ {\mathrm {C D}}\right) - \bar {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right)\right) \\ + \left(\bar {R} _ {\mathrm {C D}} (\widehat {g} _ {\mathrm {C D}}) - \widehat {R} _ {\mathrm {C D}} (\widehat {g} _ {\mathrm {C D}})\right) + \left(\widehat {R} _ {\mathrm {C D}} (\widehat {g} _ {\mathrm {C D}}) - R (\widehat {g} _ {\mathrm {C D}})\right) + \left(R (\widehat {g} _ {\mathrm {C D}}) - R (g ^ {*})\right) \\ \leq 2 \sup  _ {g \in \mathcal {G}} | R (g) - \widehat {R} _ {\mathrm {C D}} (g) | + 2 \sup  _ {g \in \mathcal {G}} | \bar {R} _ {\mathrm {C D}} (g) - \widehat {R} _ {\mathrm {C D}} (g) | + \left(R (\widehat {g} _ {\mathrm {C D}}) - R (g ^ {*})\right) \\ \leq 4 \sup  _ {g \in \mathcal {G}} | R (g) - \widehat {R} _ {\mathrm {C D}} (g) | + 2 \sup  _ {g \in \mathcal {G}} | \bar {R} _ {\mathrm {C D}} (g) - \widehat {R} _ {\mathrm {C D}} (g) | \\ \leq 1 6 L _ {\ell} \Re_ {n} (\mathcal {G}) + 8 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}} + \frac {4 C _ {\ell} \sum_ {i = 1} ^ {n} | \bar {c} _ {i} - c _ {i} |}{n} + 4 C _ {\ell} | \bar {\pi} _ {+} - \pi_ {+} |. \\ \end{array}
$$
The first inequality is derived because  $\bar{g}_{\mathrm{CD}}$  is the minimizer of  $\bar{R}(g)$ . The second and third inequality are derived according to the proof of Theorem 3 and Lemma 5 respectively.
E Proof of Theorem 5
To begin with, let  $\mathfrak{D}_n^+ (g) = \{\mathcal{D}_n|\widehat{A} (g)\geq 0\cap \widehat{B} (g)\geq 0\cap \widehat{C} (g)\geq 0\cap \widehat{D} (g)\geq 0\}$  and  $\mathfrak{D}_n^- (g) =$ $\{\mathcal{D}_n|\widehat{A} (g)\leq 0\cup \widehat{B} (g)\leq 0\cup \widehat{C} (g)\leq 0\cup \widehat{D} (g)\leq 0\}$ . Before giving the proof of Theorem 5, we give the following lemma based on the assumptions in Section 3.
Lemma 6. The probability measure of  $\mathfrak{D}_n^-(g)$  can be bounded as follows:
$$
\mathbb {P} \left(\mathfrak {D} _ {n} ^ {-} (g)\right) \leq \exp \left(\frac {- 2 a ^ {2} n}{C _ {\ell} ^ {2}}\right) + \exp \left(\frac {- 2 b ^ {2} n}{C _ {\ell} ^ {2}}\right) + \exp \left(\frac {- 2 c ^ {2} n}{C _ {\ell} ^ {2}}\right) + \exp \left(\frac {- 2 d ^ {2} n}{C _ {\ell} ^ {2}}\right). \tag {24}
$$
Proof. It can be observed that
$$
\begin{array}{l} p \left(\mathcal {D} _ {n}\right) = p \left(\boldsymbol {x} _ {1}, \boldsymbol {x} _ {1} ^ {\prime}\right) \dots p \left(\boldsymbol {x} _ {n}, \boldsymbol {x} _ {n} ^ {\prime}\right) \\ = p \left(\boldsymbol {x} _ {1}\right) \dots p \left(\boldsymbol {x} _ {n} ^ {\prime}\right) p \left(\boldsymbol {x} _ {1}\right) \dots p \left(\boldsymbol {x} _ {n} ^ {\prime}\right). \\ \end{array}
$$
Therefore, the probability measure  $\mathbb{P}(\mathfrak{D}_n^- (g))$  can be defined as follows:
$$
\begin{array}{l} \mathbb {P} \left(\mathfrak {D} _ {n} ^ {-} (g)\right) = \int_ {\mathcal {D} _ {n} \in \mathfrak {D} _ {n} ^ {-} (g)} p \left(\mathcal {D} _ {n}\right) \mathrm {d} \mathcal {D} _ {n} \\ = \int_ {\mathcal {D} _ {n} \in \mathfrak {D} _ {n} ^ {-} (g)} p (\mathcal {D} _ {n})   \mathrm {d} \boldsymbol {x} _ {1} \dots \mathrm {d} \boldsymbol {x} _ {n}   \mathrm {d} \boldsymbol {x} _ {1} ^ {\prime} \dots \mathrm {d} \boldsymbol {x} _ {n} ^ {\prime}. \\ \end{array}
$$
When exactly one ConfDiff data pair in  $S_{n}$  is replaced, the change of  $\widehat{A}(g), \widehat{B}(g), \widehat{C}(g)$  and  $\widehat{D}(g)$  will be no more than  $C_{\ell}/n$ . By applying McDiarmid's inequality, we can obtain the following inequalities:
$$
\mathbb {P} (\mathbb {E} [ \widehat {A} (g) ] - \widehat {A} (g) \geq a) \leq \exp \left(\frac {- 2 a ^ {2} n}{C _ {\ell} ^ {2}}\right),
$$
$$
\mathbb {P} (\mathbb {E} [ \widehat {B} (g) ] - \widehat {B} (g) \geq b) \leq \exp \left(\frac {- 2 b ^ {2} n}{C _ {\ell} ^ {2}}\right),
$$
$$
\mathbb {P} (\mathbb {E} [ \widehat {C} (g) ] - \widehat {C} (g) \geq c) \leq \exp \left(\frac {- 2 c ^ {2} n}{C _ {\ell} ^ {2}}\right),
$$
$$
\mathbb {P} (\mathbb {E} [ \widehat {D} (g) ] - \widehat {D} (g) \geq d) \leq \exp \left(\frac {- 2 d ^ {2} n}{C _ {\ell} ^ {2}}\right).
$$
$$
\begin{array}{l} \mathbb {P} \left(\mathfrak {D} _ {n} ^ {-} (g)\right) \leq \mathbb {P} (\widehat {A} (g) \leq 0) + \mathbb {P} (\widehat {B} (g) \leq 0) + \mathbb {P} (\widehat {C} (g) \leq 0) + \mathbb {P} (\widehat {D} (g) \leq 0) \\ \leq \mathbb {P} (\widehat {A} (g) \leq \mathbb {E} [ \widehat {A} (g) ] - a) + \mathbb {P} (\widehat {B} (g) \leq \mathbb {E} [ \widehat {B} (g) ] - b) \\ + \mathbb {P} (\widehat {C} (g) \leq \mathbb {E} [ \widehat {C} (g) ] - c) + \mathbb {P} (\widehat {D} (g) \leq \mathbb {E} [ \widehat {D} (g) ] - d) \\ = \mathbb {P} (\mathbb {E} [ \widehat {A} (g) ] - \widehat {A} (g) \geq a) + \mathbb {P} (\mathbb {E} [ \widehat {B} (g) ] - \widehat {B} (g) \geq b) \\ + \mathbb {P} (\mathbb {E} [ \widehat {C} (g) ] - \widehat {C} (g) \geq c) + \mathbb {P} (\mathbb {E} [ \widehat {D} (g) ] - \widehat {D} (g) \geq d) \\ \leq \exp \left(\frac {- 2 a ^ {2} n}{C _ {\ell} ^ {2}}\right) + \exp \left(\frac {- 2 b ^ {2} n}{C _ {\ell} ^ {2}}\right) + \exp \left(\frac {- 2 c ^ {2} n}{C _ {\ell} ^ {2}}\right) + \exp \left(\frac {- 2 d ^ {2} n}{C _ {\ell} ^ {2}}\right), \\ \end{array}
$$
Then, the proof of Theorem 5 is given.
Proof of Theorem 5. To begin with, we prove the first inequality in Theorem 5.
$$
\begin{array}{l} \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] - R (g) \\ = \mathbb {E} \left[ \widetilde {R} _ {\mathrm {C D}} (g) - \widehat {R} _ {\mathrm {C D}} (g) \right] \\ = \int_ {\mathcal {D} _ {n} \in \mathfrak {D} _ {n} ^ {+} (g)} (\widetilde {R} _ {\mathrm {C D}} (g) - \widehat {R} _ {\mathrm {C D}} (g)) p \left(\mathcal {D} _ {n}\right) \mathrm {d} \mathcal {D} _ {n} \\ + \int_ {\mathcal {D} _ {n} \in \mathfrak {D} _ {n} ^ {-} (g)} (\widetilde {R} _ {\mathrm {C D}} (g) - \widehat {R} _ {\mathrm {C D}} (g)) p (\mathcal {D} _ {n}) \mathrm {d} \mathcal {D} _ {n} \\ = \int_ {\mathcal {D} _ {n} \in \mathfrak {D} _ {n} ^ {-} (g)} (\widetilde {R} _ {\mathrm {C D}} (g) - \widehat {R} _ {\mathrm {C D}} (g)) p (\mathcal {D} _ {n})   \mathrm {d} \mathcal {D} _ {n} \geq 0, \\ \end{array}
$$
where the last inequality is derived because  $\widetilde{R}_{\mathrm{CD}}(g)$  is an upper bound of  $\widehat{R}_{\mathrm{CD}}(g)$ . Furthermore,
Similar to the proof of Theorem 3, we can obtain
$$
\left| \pi_ {+} - c _ {i} \right| + \left| \pi_ {-} - c _ {i} \right| + \left| \pi_ {+} + c _ {i} \right| + \left| \pi_ {-} + c _ {i} \right| \leq 4.
$$
$$
\mathbb {E} \left[ \widetilde {R} _ {\mathrm {C D}} (g) \right] - R (g) \leq 2 \left(L _ {f} + 1\right) C _ {\ell} \Delta ,
$$
which concludes the proof of the first inequality in Theorem 5. Before giving the proof of the second inequality, we give the upper bound of  $|\widetilde{R}_{\mathrm{CD}}(g) - \mathbb{E}[\widetilde{R}_{\mathrm{CD}}(g)]|$ . When exactly one ConfDiff data pair in  $\mathcal{D}_n$  is replaced, the change of  $\widetilde{R}_{\mathrm{CD}}(g)$  is no more than  $2C_{\ell}L_{f} / n$ . By applying McDiarmid's inequality, we have the following inequalities with probability at least  $1 - \delta / 2$ :
$$
\widetilde {R} _ {\mathrm {C D}} (g) - \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] \leq 2 C _ {\ell} L _ {f} \sqrt {\frac {\ln 2 / \delta}{2 n}},
$$
$$
\mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] - \widetilde {R} _ {\mathrm {C D}} (g) \leq 2 C _ {\ell} L _ {f} \sqrt {\frac {\ln 2 / \delta}{2 n}}.
$$
Therefore, with probability at least  $1 - \delta$ , we have
$$
| \widetilde {R} _ {\mathrm {C D}} (g) - \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] | \leq 2 C _ {\ell} L _ {f} \sqrt {\frac {\ln 2 / \delta}{2 n}}.
$$
$$
\begin{array}{l} | \widetilde {R} _ {\mathrm {C D}} (g) - R (g) | = | \widetilde {R} _ {\mathrm {C D}} (g) - \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] + \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] - R (g) | \\ \leq \left| \widetilde {R} _ {\mathrm {C D}} (g) - \mathbb {E} \left[ \widetilde {R} _ {\mathrm {C D}} (g) \right] \right| + \left| \mathbb {E} \left[ \widetilde {R} _ {\mathrm {C D}} (g) \right] - R (g) \right| \\ = | \widetilde {R} _ {\mathrm {C D}} (g) - \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] | + \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] - R (g) \\ \leq 2 C _ {\ell} L _ {f} \sqrt {\frac {\ln 2 / \delta}{2 n}} + 2 (L _ {f} + 1) C _ {\ell} \Delta , \\ \end{array}
$$
with probability at least  $1 - \delta$ , which concludes the proof.
F Proof of Theorem 6
With probability at least  $1 - \delta$ , we have