SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\Delta = \exp (-2a^{2}n / C_{\ell}^{2}) + \exp (-2b^{2}n / C_{\ell}^{2}) + \exp (-2c^{2}n / C_{\ell}^{2}) + \exp (-2d^{2}n / C_{\ell}^{2})$ . Furthermore, with probability at least  $1 - \delta$ , we have
$$
\left| \widetilde {R} _ {\mathrm {C D}} (g) - R (g) \right| \leq 2 C _ {\ell} L _ {f} \sqrt {\frac {\ln 2 / \delta}{2 n}} + 2 \left(L _ {f} + 1\right) C _ {\ell} \Delta . \tag {14}
$$
Theorem 5 demonstrates that  $\widetilde{R}_{\mathrm{CD}}(g) \to R(g)$  in  $\mathcal{O}_p(1 / \sqrt{n})$ , which means that  $\widetilde{R}_{\mathrm{CD}}(g)$  is biased yet consistent. The estimation error bound of  $\widetilde{g}_{\mathrm{CD}}$  is analyzed in Theorem 6.
Theorem 6. Based on the assumptions above, for any  $\delta >0$ , the following inequality holds with probability at least  $1 - \delta$ :
$$
R \left(\widetilde {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right) \leq 8 L _ {\ell} \Re_ {n} (\mathcal {G}) + 4 C _ {\ell} \left(L _ {f} + 1\right) \sqrt {\frac {\ln 2 / \delta}{2 n}} + 4 \left(L _ {f} + 1\right) C _ {\ell} \Delta . \tag {15}
$$
Theorem 6 elucidates that as  $n\to \infty$ ,  $R(\widetilde{g}_{\mathrm{CD}})\rightarrow R(g^{*})$ , since  $\mathcal{R}_n(\mathcal{G})\to 0$  for all parametric models with a bounded norm [60] and  $\Delta \to 0$ . Furthermore, the estimation error bound converges in  $\mathcal{O}_p(1 / \sqrt{n})$ , which is the optimal parametric rate for empirical risk minimization without additional assumptions [59].
In this section, we verify the effectiveness of our proposed approaches experimentally.
We conducted experiments on benchmark data sets, including MNIST [61], Kuzushiji-MNIST [62], Fashion-MNIST [63], and CIFAR-10 [64]. In addition, four UCI data sets [65] were used, including Optdigits, USPS, Pundigits, and Letter. Since the data sets were originally designed for multi-class classification, we manually partitioned them into binary classes. For CIFAR-10, we used ResNet-34 [66] as the model architecture. For other data sets, we used a multilayer perceptron (MLP) with three hidden layers of width 300 equipped with the ReLU [67] activation function and batch normalization [68]. The logistic loss is utilized to instantiate the loss function  $\ell$ .
It is worth noting that confidence difference is given by labelers in real-world applications, while it was generated synthetically in this paper to facilitate comprehensive experimental analysis. We firstly trained a probabilistic classifier via logistic regression with ordinarily labeled data and the same neural network architecture. Then, we sampled unlabeled data in pairs at random, and generated the class posterior probabilities by inputting them into the probabilistic classifier. After that, we
Table 1: Classification accuracy (mean±std) of each method on benchmark data sets with different class priors, where the best performance (excluding Oracle) is shown in bold.
Class Prior	Method	MNIST	Kuzushiji	Fashion	CIFAR-10
π+ = 0.2	Pcomp-Unbiased	0.761±0.017	0.637±0.052	0.737±0.050	0.776±0.023
Pcomp-ReLU	0.800±0.000	0.800±0.000	0.800±0.000	0.800±0.000
Pcomp-ABS	0.800±0.000	0.800±0.000	0.800±0.000	0.800±0.000
Pcomp-Teacher	0.965±0.010	0.871±0.046	0.853±0.017	0.836±0.019
Oracle-Hard	0.990±0.000	0.939±0.001	0.979±0.001	0.894±0.003
Oracle-Soft	0.989±0.001	0.939±0.004	0.979±0.001	0.893±0.003
ConfDiff-Unbiased	0.789±0.041	0.672±0.053	0.855±0.024	0.789±0.025
ConfDiff-ReLU	0.968±0.003	0.860±0.017	0.964±0.004	0.844±0.020
ConfDiff-ABS	0.975±0.003	0.898±0.003	0.965±0.002	0.862±0.015
π+ = 0.5	Pcomp-Unbiased	0.712±0.020	0.578±0.036	0.723±0.042	0.703±0.042
Pcomp-ReLU	0.502±0.003	0.502±0.004	0.500±0.000	0.602±0.032
Pcomp-ABS	0.842±0.012	0.727±0.006	0.851±0.012	0.583±0.018
Pcomp-Teacher	0.893±0.014	0.782±0.046	0.903±0.016	0.779±0.016
Oracle-Hard	0.986±0.000	0.929±0.002	0.976±0.001	0.871±0.003
Oracle-Soft	0.985±0.001	0.928±0.002	0.978±0.001	0.877±0.002
ConfDiff-Unbiased	0.911±0.046	0.712±0.046	0.896±0.036	0.720±0.024
ConfDiff-ReLU	0.944±0.011	0.805±0.015	0.960±0.003	0.830±0.007
ConfDiff-ABS	0.964±0.001	0.867±0.006	0.967±0.001	0.843±0.004
π+ = 0.8	Pcomp-Unbiased	0.799±0.005	0.671±0.029	0.813±0.029	0.737±0.022
Pcomp-ReLU	0.910±0.031	0.775±0.022	0.897±0.023	0.851±0.010
Pcomp-ABS	0.854±0.027	0.838±0.026	0.921±0.017	0.849±0.007
Pcomp-Teacher	0.943±0.026	0.814±0.027	0.936±0.014	0.821±0.003
Oracle-Hard	0.991±0.001	0.942±0.003	0.979±0.000	0.897±0.002
Oracle-Soft	0.990±0.002	0.945±0.003	0.980±0.001	0.904±0.009
ConfDiff-Unbiased	0.792±0.017	0.758±0.033	0.810±0.035	0.794±0.012
ConfDiff-ReLU	0.970±0.004	0.886±0.009	0.970±0.002	0.851±0.012
ConfDiff-ABS	0.983±0.002	0.915±0.001	0.975±0.002	0.874±0.011
generated confidence difference for each pair of sampled data according to Definition 1. To verify the effectiveness of our approaches under different class prior settings, we set  $\pi_{+} \in \{0.2, 0.5, 0.8\}$  for all the data sets. Besides, we assumed that the class prior  $\pi_{+}$  was known for all the compared methods. We repeated the sampling-and-training procedure for five times, and the mean accuracy as well as the standard deviation were recorded.
We adopted the following variants of our proposed approaches: 1) ConfDiff-Unbiased, which denotes the method working by minimizing the unbiased risk estimator; 2) ConfDiff-ReLU, which denotes the method working by minimizing the corrected risk estimator with the ReLU function as the risk correction function; 3) ConfDiff-ABS, which denotes the method working by minimizing the corrected risk estimator with the absolute value function as the risk correction function. We compared our proposed approaches with several Pcomp methods [43], including Pcomp-Unbiased, Pcomp-ReLU, Pcomp-ABS, and Pcomp-Teacher. We also recorded the experimental results of supervised learning methods, including Oracle-Hard having access to ground-truth hard labels and Oracle-Soft having access to pointwise positive confidence. All the experiments were conducted on NVIDIA GeForce RTX 3090. The number of training epochs was set to 200 and we obtained the testing accuracy by averaging the results in the last 10 epochs. All the methods were implemented in PyTorch [69]. We used the Adam optimizer [70]. To ensure fair comparisons, We set the same hyperparameter values for all the compared approaches, where the details can be found in Appendix H.
Benchmark data sets. Table 1 reports detailed experimental results for all the compared methods on four benchmark data sets. Based on Table 1, we can draw the following conclusions: a) On all the cases of benchmark data sets, our proposed ConfDiff-ABS method achieves superior performance against all of the other compared approaches significantly, which validates the effectiveness of our approach in utilizing supervision information of confidence difference; b) Pcomp-Teacher achieves
Table 2: Classification accuracy (mean±std) of each method on UCI data sets with different class priors, where the best performance (excluding Oracle) is shown in bold.
Class Prior	Method	Optdigits	USPS	Pendigits	Letter
π+ = 0.2	Pcomp-Unbiased	0.771±0.016	0.721±0.046	0.743±0.057	0.757±0.028
Pcomp-ReLU	0.800±0.000	0.800±0.000	0.800±0.000	0.800±0.000
Pcomp-ABS	0.800±0.001	0.800±0.000	0.800±0.000	0.800±0.000
Pcomp-Teacher	0.901±0.023	0.894±0.023	0.928±0.019	0.883±0.006
Oracle-Hard	0.990±0.002	0.984±0.002	0.997±0.001	0.978±0.003
Oracle-Soft	0.990±0.003	0.984±0.004	0.998±0.001	0.971±0.007
ConfDiff-Unbiased	0.831±0.078	0.840±0.078	0.865±0.079	0.732±0.053
ConfDiff-ReLU	0.953±0.014	0.957±0.007	0.987±0.003	0.929±0.008
ConfDiff-ABS	0.963±0.009	0.960±0.005	0.988±0.002	0.942±0.007
π+ = 0.5	Pcomp-Unbiased	0.651±0.112	0.671±0.090	0.748±0.038	0.632±0.019
Pcomp-ReLU	0.630±0.076	0.554±0.048	0.514±0.019	0.525±0.023
Pcomp-ABS	0.787±0.031	0.814±0.018	0.793±0.017	0.748±0.031
Pcomp-Teacher	0.890±0.009	0.860±0.012	0.883±0.018	0.864±0.024
Oracle-Hard	0.988±0.003	0.980±0.003	0.997±0.001	0.975±0.001
Oracle-Soft	0.987±0.003	0.980±0.003	0.997±0.001	0.967±0.006
ConfDiff-Unbiased	0.917±0.006	0.936±0.010	0.945±0.052	0.755±0.041
ConfDiff-ReLU	0.921±0.011	0.945±0.009	0.981±0.004	0.895±0.006
ConfDiff-ABS	0.962±0.006	0.959±0.004	0.988±0.003	0.925±0.003
π+ = 0.8	Pcomp-Unbiased	0.765±0.023	0.746±0.012	0.743±0.026	0.694±0.031
Pcomp-ReLU	0.902±0.017	0.891±0.024	0.913±0.023	0.827±0.025
Pcomp-ABS	0.894±0.019	0.879±0.009	0.911±0.009	0.870±0.006
Pcomp-Teacher	0.918±0.007	0.933±0.023	0.903±0.008	0.872±0.011
Oracle-Hard	0.987±0.003	0.983±0.002	0.997±0.001	0.976±0.004
Oracle-Soft	0.986±0.003	0.985±0.004	0.998±0.001	0.965±0.010
ConfDiff-Unbiased	0.886±0.037	0.803±0.042	0.892±0.096	0.748±0.015
ConfDiff-ReLU	0.949±0.007	0.958±0.008	0.986±0.003	0.927±0.008
ConfDiff-ABS	0.964±0.005	0.964±0.003	0.987±0.002	0.945±0.007
superior performance against all of the other Pcomp approaches by a large margin. The excellent performance benefits from the effectiveness of consistency regularization for weakly supervised learning problems [1, 17, 71]; c) It is worth noting that the classification results of ConfDiff-ReLU and ConfDiff-ABS have smaller variances than ConfDiff-Unbiased. It demonstrates that the risk correction method can enhance the stability and robustness for ConfDiff classification.
UCI data sets. Table 2 reports detailed experimental results on four UCI data sets as well. From Table 2, we can observe that: a) On all the UCI data sets under different class prior probability settings, our proposed ConfDiff-ABS method achieves the best performance among all the compared approaches with significant superiority, which verifies the effectiveness of our proposed approaches again; b) The performance of our proposed approaches is more stable than the compared Pcomp approaches under different class prior probability settings, demonstrating the superiority of our methods in dealing with various kinds of data distributions.
4.3 Experiments on a Real-world Recommender System Data Set
We also conducted experiments on a recommender system data set to demonstrate our approach's usefulness and promising applications in real-world scenarios. We used the KuaiRec [72] data set, a real-world recommender system data set collected from a well-known short-video mobile app. In this data set, user-item interactions are represented by watching ratios, i.e., the ratios of watching time to the entire length of videos. Such statistics could reveal the confidence of preference, and we regarded them as pointwise positive confidence. We generated pairwise confidence difference between pairs of items for a given user. We adopted the NCF [73] model as our backbone.
Table 3: Experimental results on the KuaiRec data set.
Method	HR	NDCG
BPR	0.464	0.256
MRL	0.476	0.271
Oracle-Hard	0.469	0.283
Oracle-Soft	0.534	0.380
Pcomp-Teacher	0.179	0.066
ConfDiff-ABS	0.570	0.372
(a) Kuzushiji
(b) Fashion
(c) USPS
(d) Letter
(e) Optdigits
(f)Pendigits
(g) MNIST
Figure 2: Classification performance of ConfDiff-ReLU and ConfDiff-ABS given a fraction of training data as well as Pcomp-Teacher given  $100\%$  of training data  $(\pi_{+} = 0.5)$ .
(h) CIFAR-10
Details of the experimental setup and the data set can be found in Appendix H. We employed 5 compared methods, including BPR [74], Margin Ranking Loss (MRL), Oracle-Hard, Oracle-Soft, and Pcomp-Teacher. Table 3 reports the hit ratio (HR) and normalized discounted cumulative gain (NDCG) results. Our approach performs comparably against Oracle in terms of NDCG and even performs better in terms of HR. On the contrary, Pcomp-Teacher does not perform well on this data set. It validates the effectiveness of our approach in exploiting the supervision information of the confidence difference.
4.4 Performance with Fewer Training Data
We conducted experiments by changing the fraction of training data for ConfDiff-ReLU and ConfDiff-ABS (100% indicated that all the ConfDiff data were used for training). For comparison, we used 100% of training data for Pcomp-Teacher during the training process. Figure 2 shows the results with  $\pi_{+} = 0.5$ , and more experimental results can be found in Appendix I. We can observe that the classification performance of our proposed approaches is still advantageous given a fraction of training data. Our approaches can achieve superior or comparable performance even when only 10% of training data are used. It elucidates that leveraging confidence difference may be more effective than increasing the number of training examples.
4.5 Analysis on Robustness
In this subsection, we investigate the influence of an inaccurate class prior probability and noisy confidence difference on the generalization performance of the proposed approaches. Specifically, let  $\bar{\pi}_{+} = \epsilon \pi_{+}$  denote the corrupted class prior probability with  $\epsilon$  being a real number around 1. Let  $\bar{c}_i = \epsilon_i' c_i$  denote the noisy confidence difference where  $\epsilon_i'$  is sampled from a normal distribution  $\mathcal{N}(1, \sigma^2)$ . Figure 3 shows the classification performance of our proposed approaches on MNIST and Pendigits ( $\pi_{+} = 0.5$ ) with different  $\epsilon$  and  $\sigma$ . It is demonstrated that the performance degenerates with  $\epsilon = 0.8$  or  $\epsilon = 1.2$  on some data sets, which indicates that it is important to estimate the class prior accurately.
In this paper, we dived into a novel weakly supervised learning setting where only unlabeled data pairs equipped with confidence difference were given. To solve the problem, an unbiased risk estimator was derived to perform empirical risk minimization. An estimation error bound was established to show that the optimal parametric convergence rate can be achieved. Furthermore, a risk correction approach was introduced to alleviate overfitting issues. Extensive experimental results validated the superiority of our proposed approaches. In future, it would be promising to apply our approaches in real-world scenarios and multi-class classification settings.
(a) ConfDiff-Unbiased
(b) ConfDiff-ReLU
(c) ConfDiff-ABS
(d) ConfDiff-Unbiased
(e) ConfDiff-ReLU
(f) ConfDiff-ABS
Figure 3: Classification accuracy on MNIST (the first row) and Pendigits (the second row) with  $\pi_{+} = 0.5$  given an inaccurate class prior probability and noisy confidence difference.
The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. The research was partially supported by the SGU MEXT Scholarship, the Junior Research Associate (JRA) program of RIKEN, Microsoft Research Asia, the National Science Foundation of China (62176055), and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
[1] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A. Raffel. MixMatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32, pages 5050-5060, 2019.
[2] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 2006.
[3] Xiaojin Zhu and Andrew B. Goldberg. Introduction to semi-supervised learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 3(1):1-130, 2009.
[4] Yu-Feng Li and Zhi-Hua Zhou. Towards making unlabeled data never hurt. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1):175-188, 2015.
[5] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems 31, pages 8536-8546, 2018.
[6] Junnan Li, Caiming Xiong, and Steven C. H. Hoi. MoPro: Webly supervised learning with momentum prototypes. In Proceedings of the 9th International Conference on Learning Representations, 2021.
[7] Giorgio Patrini, Alessandro Rozza, Aditya K. Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 1944–1952, 2017.
Before giving the proof of Theorem 1, we begin with the following lemmas:
Lemma 2. The confidence difference  $c(\pmb{x}, \pmb{x}^{\prime})$  can be equivalently expressed as
$$
\begin{array}{l} c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \frac {\pi_ {+} p (\boldsymbol {x}) p _ {+} \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {+} p _ {+} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)}{p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)} (16) \\ = \frac {\pi_ {-} p _ {-} \left(\boldsymbol {x}\right) p \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {-} p (\boldsymbol {x}) p _ {-} \left(\boldsymbol {x} ^ {\prime}\right)}{p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)} (17) \\ \end{array}
$$
Proof. On one hand,
$$
\begin{array}{l} c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = p (y ^ {\prime} = + 1 | \boldsymbol {x} ^ {\prime}) - p (y = + 1 | \boldsymbol {x}) \\ = \frac {p \left(\boldsymbol {x} ^ {\prime} , y ^ {\prime} = + 1\right)}{p \left(\boldsymbol {x} ^ {\prime}\right)} - \frac {p (\boldsymbol {x} , y = + 1)}{p (\boldsymbol {x})} \\ = \frac {\pi_ {+} p _ {+} \left(\boldsymbol {x} ^ {\prime}\right)}{p \left(\boldsymbol {x} ^ {\prime}\right)} - \frac {\pi_ {+} p _ {+} (\boldsymbol {x})}{p (\boldsymbol {x})} \\ = \frac {\pi_ {+} p (\boldsymbol {x}) p _ {+} \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {+} p _ {+} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)}{p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)}. \\ \end{array}
$$
$$
\begin{array}{l} c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = p (y ^ {\prime} = + 1 | \boldsymbol {x} ^ {\prime}) - p (y = + 1 | \boldsymbol {x}) \\ = (1 - p \left(y ^ {\prime} = 0 | \boldsymbol {x} ^ {\prime}\right)) - (1 - p (y = 0 | \boldsymbol {x})) \\ = p (y = 0 | \boldsymbol {x}) - p \left(y ^ {\prime} = 0 | \boldsymbol {x} ^ {\prime}\right) \\ = \frac {p (\boldsymbol {x} , y = 0)}{p (\boldsymbol {x})} - \frac {p \left(\boldsymbol {x} ^ {\prime} , y = 0\right)}{p \left(\boldsymbol {x} ^ {\prime}\right)} \\ = \frac {\pi_ {-} p _ {-} (\boldsymbol {x})}{p (\boldsymbol {x})} - \frac {\pi_ {-} p _ {-} \left(\boldsymbol {x} ^ {\prime}\right)}{p \left(\boldsymbol {x} ^ {\prime}\right)} \\ = \frac {\pi_ {-} p _ {-} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {-} p (\boldsymbol {x}) p _ {-} \left(\boldsymbol {x} ^ {\prime}\right)}{p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)}, \\ \end{array}
$$
Lemma 3. The following equations hold:
$$
\mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\pi_ {+} - c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x}), + 1) ] = \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ], \tag {18}
$$
$$
\mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\pi_ {-} + c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x}), - 1) ] = \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ], \tag {19}
$$
$$
\mathbb {E} _ {p \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)} \left[ \left(\pi_ {+} + c \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)\right) \ell \left(g \left(\boldsymbol {x} ^ {\prime}\right), + 1\right) \right] = \pi_ {+} \mathbb {E} _ {p _ {+} \left(\boldsymbol {x} ^ {\prime}\right)} \left[ \ell \left(g \left(\boldsymbol {x} ^ {\prime}\right), + 1\right) \right], \tag {20}
$$
$$
\mathbb {E} _ {p \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)} \left[ \left(\pi_ {-} - c \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)\right) \ell \left(g \left(\boldsymbol {x} ^ {\prime}\right), - 1\right) \right] = \pi_ {-} \mathbb {E} _ {p _ {-} \left(\boldsymbol {x} ^ {\prime}\right)} \left[ \ell \left(g \left(\boldsymbol {x} ^ {\prime}\right), - 1\right) \right]. \tag {21}
$$
Proof. Firstly, the proof of Eq. (18) is given:
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\pi_ {+} - c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x}), + 1) ] \\ = \int \int \frac {\pi_ {+} p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {+} p (\boldsymbol {x}) p _ {+} \left(\boldsymbol {x} ^ {\prime}\right) + \pi_ {+} p _ {+} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)}{p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)} \ell (g (\boldsymbol {x}), + 1) p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) d \boldsymbol {x} d \boldsymbol {x} ^ {\prime} \\ = \int \int (\pi_ {+} p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {+} p (\boldsymbol {x}) p _ {+} \left(\boldsymbol {x} ^ {\prime}\right) + \pi_ {+} p _ {+} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} d \boldsymbol {x} ^ {\prime} \\ = \int \pi_ {+} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} \int p \left(\boldsymbol {x} ^ {\prime}\right) d \boldsymbol {x} ^ {\prime} - \int \pi_ {+} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} \int p _ {+} \left(\boldsymbol {x} ^ {\prime}\right) d \boldsymbol {x} ^ {\prime} \\ + \int \pi_ {+} p _ {+} (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} \int p (\boldsymbol {x} ^ {\prime}) d \boldsymbol {x} ^ {\prime} \\ = \int \pi_ {+} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} - \int \pi_ {+} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} + \int \pi_ {+} p _ {+} (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} \\ = \int \pi_ {+} p _ {+} (\boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) d \boldsymbol {x} \\ = \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ]. \\ \end{array}
$$
After that, the proof of Eq. (19) is given:
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} [ (\pi_ {-} + c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x}), - 1) ] \\ = \int \int \frac {\pi_ {-} p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) + \pi_ {-} p _ {-} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {-} p (\boldsymbol {x}) p _ {-} \left(\boldsymbol {x} ^ {\prime}\right)}{p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right)} \ell (g (\boldsymbol {x}), - 1) p (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) d \boldsymbol {x} d \boldsymbol {x} ^ {\prime} \\ = \int \int (\pi_ {-} p (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) + \pi_ {-} p _ {-} (\boldsymbol {x}) p \left(\boldsymbol {x} ^ {\prime}\right) - \pi_ {-} p (\boldsymbol {x}) p _ {-} \left(\boldsymbol {x} ^ {\prime}\right)) \ell (g (\boldsymbol {x}), - 1) d \boldsymbol {x} d \boldsymbol {x} ^ {\prime} \\ = \int \pi_ {-} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) \mathrm {d} \boldsymbol {x} \int p \left(\boldsymbol {x} ^ {\prime}\right) \mathrm {d} \boldsymbol {x} ^ {\prime} + \int \pi_ {-} p _ {-} (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) \mathrm {d} \boldsymbol {x} \int p \left(\boldsymbol {x} ^ {\prime}\right) \mathrm {d} \boldsymbol {x} ^ {\prime} \\ - \int \pi_ {-} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) d \boldsymbol {x} \int p _ {-} \left(\boldsymbol {x} ^ {\prime}\right) d \boldsymbol {x} ^ {\prime} \\ = \int \pi_ {-} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) d \boldsymbol {x} + \int \pi_ {-} p _ {-} (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) d \boldsymbol {x} - \int \pi_ {-} p (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) d \boldsymbol {x} \\ = \int \pi_ {-} p _ {-} (\boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) d \boldsymbol {x} \\ = \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ]. \\ \end{array}
$$
It can be noticed that  $c(\pmb{x}, \pmb{x}') = -c(\pmb{x}', \pmb{x})$  and  $p(\pmb{x}, \pmb{x}') = p(\pmb{x}', \pmb{x})$ . Therefore, it can be deduced naturally that  $\mathbb{E}_{p(\pmb{x}, \pmb{x}')}[(\pi_{+} - c(\pmb{x}, \pmb{x}'))\ell(g(\pmb{x}), +1)] = \mathbb{E}_{p(\pmb{x}', \pmb{x})}[(\pi_{+} + c(\pmb{x}', \pmb{x}))\ell(g(\pmb{x}), +1)]$ . Because  $\pmb{x}$  and  $\pmb{x}'$  are symmetric, we can swap them and deduce Eq. (20). Eq. (21) can be deduced in the same manner, which concludes the proof.