SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin-Qiang Cai $^{1}$ , Yu-Jie Zhang $^{1}$ , Chao-Kai Chiang $^{1}$ , Masashi Sugiyama $^{2,1}$
Imitation learning from human feedback studies how to train well-performed imitation agents with an annotator's relative comparison of two demonstrations (one demonstration is better/worse than the other), which is usually easier to collect than the perfect expert data required by traditional imitation learning. However, in many real-world applications, it is still expensive or even impossible to provide a clear pairwise comparison between two demonstrations with similar quality. This motivates us to study the problem of imitation learning with vague feedback, where the data annotator can only distinguish the paired demonstrations correctly when their quality differs significantly, i.e., one from the expert and another from the non-expert. By modeling the underlying demonstration pool as a mixture of expert and non-expert data, we show that the expert policy distribution can be recovered when the proportion  $\alpha$  of expert data is known. We also propose a mixture proportion estimation method for the unknown  $\alpha$  case. Then, we integrate the recovered expert policy distribution with generative adversarial imitation learning to form an end-to-end algorithm<sup>1</sup>. Experiments show that our methods outperform standard and preference-based imitation learning methods on various tasks.
Imitation learning (IL) is a popular approach for training agents to perform tasks by learning from expert demonstrations [1-3]. It has been applied successfully to a variety of tasks, including robot control [1], autonomous driving [4], and game playing [5]. However, traditional IL methods struggle when presented with both expert and non-expert demonstrations, as the agents may learn incorrect behaviors from the non-expert demonstrations [6]. This problem, which researchers refer to as "Imitation Learning from Imperfect Demonstration" (ILfID), arises when the demonstrations used to train the model may contain some non-expert data [6-9].
One popular solution to ILfID is resorting to an oracle to provide specific information, such as explicit labels (confidence from an expert) of each demonstration, as in Figure 1a. However, such a specific oracle is quite expensive. A more recent framework, Reinforcement Learning from Human Feedback (RLHF) [10, 11], incorporates human feedback into the learning process, including a key part of the pipeline used for training ChatGPT [12]. There exist two widely studied paradigms of feedback: full ranking and global binary comparison, as in Figures 1b and 1c. Methods that use full-ranked demonstrations assumed that the feedback between every pair of trajectories is available, and further employed preference-based methods to solve the problem [13, 14]. Other studies have investigated situations where demonstrations can be divided into two global binary datasets, allowing the learner to filter out non-expert data from these global comparisons [7, 8]. However, data processed by any feedback of Figures 1a, 1b, and 1c require the guarantee that clear ranking information or at least one set consisting of pure expert demonstrations exists, so that off-the-shall IL methods can be applied immediately. The availability of expert demonstrations raises the question of what if, in practice, the feedback is not clear enough to provide purely expert demonstrations but mixed with non-expert demonstrations? The question poses a challenge that previous methods fail to address since none of
1The code is available on https://github.com/caixq1996/COMPILER.
(a) Explicit label
(b) Full ranking
(c) Global (clear) comparison
(d) Vague comparison (our setting)
Figure 1: (a)-(d): The comparisons among the imitation learning with the explicit label, full ranking, global comparison, and our setting.  $\square$  denotes the expert data while  $\square$  denotes the non-expert one. (e) An example of labeling route navigation. In the example, Path 1 and Path 2 share the same distance. A similar situation lies in Path 3 and Path 4. The data collectors cannot provide explicit, ranking, or pairwise information as in (a), (b), and (c), while they can only provide vague comparisons that  $\Gamma^{+}$  can be more expert-like than  $\Gamma^{-}$ .
(e) A maze example
the previous IL algorithms handles the issue of the mixture of expert and non-expert demonstrations without other information. Their conventional strategies are to deal with the vagueness in the data processing stage (relying on human feedback) not during the learning stage (by the learning agent).
Therefore, we consider a kind of weaker feedback than those previously formulated in the literature, which unintentionally mixes expert and non-expert demonstrations due to vague feedback. Specifically, the human annotator demonstrates in a pairwise way and only distinguishes the expert demonstration from the non-expert one. However, the annotator cannot tell the origin when both are from an expert or a non-expert. As depicted in Figure 1d, this annotation process results in two datasets:  $\Gamma^{+}$ , containing demonstrations the annotator believes to be most likely from experts, and  $\Gamma^{-}$ , containing non-expert-like demonstrations. If one only places the distinguishable demonstrations into the pools and discards the indistinguishable ones, it results in Figure 1c. Since we aim to investigate the potential of IL under the presence of non-expert demonstrations, we distribute the indistinguishable demonstrations randomly, one to  $\Gamma^{+}$  and the other to  $\Gamma^{-}$ . This step embodies the vagueness we wish to study in this paper. Note that either dataset contains non-expert demonstrations, and thus, a direct application of the existing IL method might be inappropriate due to the danger of learning from undesirable demonstrations.
Vague feedback commonly exists in many scenarios, especially regarding RLHF [12]. In a crowdsourcing example of routes navigation, as shown in Figure 1e, most of the time crowd-workers may lack domain expertise. Meanwhile, it is quite difficult to distinguish every pair of demonstrations (when facing two trajectories from the same source, such as  $(\square, \square)$  or  $(\square, \square))$ . Therefore, we cannot obtain high-quality crowd-sourcing labels when the data are annotated [15] as in Figures 1a, 1b, and 1c. On the other hand, it is natural for workers to provide vague comparisons as in Figure 1d.
In this work, we formulate the problem of Vaguely Pairwise Imitation Learning (VPIL), in which the human annotator can only distinguish the paired demonstrations correctly when their quality differs significantly. In Section 4, we analyze two situations within this learning problem: VPIL with known expert ratio  $\alpha$  and unknown  $\alpha$ . For VPIL with known  $\alpha$ , we provide rigorous mathematical analysis and show that the expert policy distribution can be empirically estimated with the datasets  $\Gamma^{+}$  and  $\Gamma^{-}$ ; for the more challenging situation of VPIL with unknown  $\alpha$ , we propose a reduction of the problem of estimating  $\alpha$  to a mixture proportion estimation problem [16] and develop an iterative update procedure to estimate  $\alpha$ . In Section 5, we integrate our algorithm with an off-the-shelf IL method to solve VPIL problems. In Section 6, we evaluate our methods with state-of-the-art ILfID methods on a variety of tasks of MuJoCo [17] with different  $\alpha$  ratios and find that our methods obtained the best performance.
In imitation learning scenarios, a model is trained to mimic the actions of expert demonstrations. One of the main challenges of imitation learning is that the gathered demonstrations can be imprecise, making it difficult for the learner to accurately replicate the underlying policy of the demonstrations. To overcome this issue, various works have utilized an annotator, a source of supplementary supervi
Figure 2: The description of the data collection process.
sion that can aid the learner in better understanding the expert's intent. For example, [18] used explicit action signals of the annotator; [14], [19], and [13] used ranking/preference information from the annotator to learn the policy; [6] utilized confidence information from the annotator to re-weight the unlabeled demonstrations and further learned the policy. However, as illustrated in Section 1, in some cases, the annotator may not be able to provide explicit information for the imperfect demonstrations, especially when the demonstrations come from different sources.
Alternative strategies for IL by imposing prior knowledge instead of an annotator have also been put forward, such as state density estimation for importance weighting [20], state-action extrapolating [21], and adding noise to recover ranking [22]. But this research line relies on certain assumptions about the state/state-action spaces, which may not always hold in many real-world applications. In this work, we focus on using an annotator with vague information, which is also low-cost.
Our methods drew inspiration from the risk rewriting technique in weakly supervised learning literature [23], which aims to establish an unbiased risk estimator for evaluating the model's quality only with weak supervision. Although the technique has been successfully applied to various specific weakly supervised learning problems [24-27], these methods typically require knowledge of a parameter called the mixture proportion, i.e., the expert ratio in our problem [28], the estimation of which can be challenging in the imitation learning problem (see Section 6.3 for more details). To overcome this issue, we introduce an iterative data selection procedure to better estimate the expert ratio, leading to superior empirical performance. It is worth noting that [6] also used the risk rewriting technique to identify the expert demonstrations, but their algorithm does not require estimation of the expert ratio as the confidence score is given. However, such information is unavailable in VPIL, making the problem more challenging.
3 Preliminaries and Problem Setting
In this section, we first introduce the IL process. Then we formulate the VPIL problem.
Markov Decision Process. In policy learning problems, a Markov Decision Process (MDP) can be represented by a tuple  $\langle S, \mathcal{A}, \mathcal{P}, \gamma, r, T \rangle$ , where  $\mathcal{S}$  is the state space;  $\mathcal{A}$  is the action space;  $\mathcal{P}: S \times \mathcal{A} \times S \to [0,1]$  is the transition probability distribution;  $\gamma$  is a discount factor in the range  $(0,1]$ ;  $r: S \to \mathbb{R}$  is the reward function; and  $T$  is the horizon. The goal is to learn a policy  $\pi$  that maximizes the expected returns  $\mathbb{E}[\sum_{t=0}^{\infty} \gamma^{t} r(s_{t}, a_{t})]$ , where  $\mathbb{E}$  denotes the expectation.
In IL, the learner does not have access to the reward function  $r$ . Instead, the learner is given  $m$  expert demonstrations  $\tau_{\mathsf{E},1}, \tau_{\mathsf{E},2}, \ldots, \tau_{\mathsf{E},m}$ , where  $\tau_{\mathsf{E},i}, i \in \{1, \ldots, m\}$  is the  $i$ -th trajectory (a series of state-action pairs) drawn independently following the demonstrator's policy  $\pi_{\mathsf{E}}$ . The goal of the learner is to learn a policy  $\pi_{\theta}$  to mimic  $\pi_{\mathsf{E}}$ .
Occupancy Measure. Since in IL we do not have reward functions, we need to measure the policy performance in the state-action space, i.e., occupancy measure  $\rho_{\pi}$ . The occupancy measure  $\rho_{\pi}: S \times \mathcal{A} \mapsto \mathbb{R}$  was introduced to characterize the distribution of state-action pairs generated by a given policy  $\pi$ , which was defined with the discount factor:  $\rho_{\pi}(s, a) = \pi(a|s) \sum_{t=1}^{\infty} \gamma^{t} \operatorname*{Pr}[s_{t} = s | \pi]$ , where  $\operatorname*{Pr}[s_{t} = s | \pi]$  is the probability of reaching state  $s$  at time  $t$  following policy  $\pi$  [29].
Moreover, we know that there is a one-to-one correspondence between the occupancy measure and the policy. Specifically, we have the following theorem.
Theorem 1 (Theorem 2 of [29]). Suppose  $\rho$  is the occupancy measure for  $\pi_{\rho}(a|s) := \frac{\rho(s,a)}{\sum_{a'\in\mathcal{A}}\rho(s,a')}$ . Then,  $\pi_{\rho}$  is the only policy whose occupancy measure is  $\rho$ .
We can also define a normalized version of the occupancy measure by  $p_{\pi}(s,a) \triangleq \frac{\rho_{\pi}(s,a)}{\sum_{s' \in S, a' \in A} \rho_{\pi}(s',a')} = (1 - \gamma) \rho_{\pi}(s,a)$ . If we have  $m$  expert demonstrations  $\tau_{\mathsf{E},1}, \tau_{\mathsf{E},2}, \dots, \tau_{\mathsf{E},m}$ , the occupancy measure  $p_{\pi_E}$  can be empirically estimated by the trajectories as  $\widehat{p}_{\pi_E}(s,a) = \frac{1 - \gamma}{m} \sum_{i=1}^{m} \sum_{t=0}^{\infty} \gamma^t \mathbb{1}\left[(s_t^{(i)}, a_t^{(i)}) = (s,a)\right]$ , where  $\mathbb{1}[\cdot]$  is an indicator function that returns 1 if  $\cdot$  is true and returns 0 otherwise. Finding out the underlying  $p_{\pi_E}$  is the key for solving IL problems [1, 30].
3.2 Vaguely Pairwise Imitation Learning
In this work, we focus on the ILfID problem with pairwise information, where the learner aims to learn a good policy with a pairwise dataset  $(\Gamma^{+},\Gamma^{-})$ , generated from a mixture demonstration pool performed by an expert policy  $\pi_{\mathsf{E}}$  and a set of non-expert policies  $\{\pi_{\mathsf{NE}}^{(k)}\}_{k = 1}^{K}$ . In this work, we consider the mixed occupancy measure of the non-expert policy set as  $p_{\pi_{\mathsf{NE}}}$ . The proportion of the expert data within the pool is denoted as  $\alpha \in (0,1]$ . The data collection process is shown in Figure 2. To clearly reveal the effect of mixed demonstrations and explain the proposed framework, we choose to assume that the data collector is not an attacker and will make mistakes, so there will be no noise during the collection process. In Section 6.4, we discuss how our method can easily be extended to handle the presence of human error.
Also, in this work, we do not consider that the data collector could be an attacker or make mistakes, so there will be no noise during the collection process.
Collection of Pairwise Datasets. The data collector would first sample a pair of trajectories  $(\tau_{i},\tau_{j})$  independently from the mixture pool. If  $(\tau_{i},\tau_{j})$  are from different sources, i.e., one is from the expert, and another is not, then the collector will take the expert one  $\tau_{i}$  into  $\Gamma^{+}$  and the non-expert one  $\tau_{j}$  into  $\Gamma^{-}$ . Otherwise, the collector randomly puts them into  $\Gamma^{+}$  and  $\Gamma^{-}$ . Under such a data generation process, the expertise probabilities of this pair are as follows:
$$
\left\{ \begin{array}{l} \Pr \left[ \tau_ {i} \sim p _ {\pi_ {\mathsf {E}}}, \tau_ {j} \sim p _ {\pi_ {\mathsf {E}}} \right] = \alpha^ {2}, \\ \Pr \left[ \tau_ {i} \sim p _ {\pi_ {\mathsf {N E}}}, \tau_ {j} \sim p _ {\pi_ {\mathsf {N E}}} \right] = (1 - \alpha) ^ {2}, \\ \Pr \left[ \tau_ {i} \sim p _ {\pi_ {\mathsf {E}}}, \tau_ {j} \sim p _ {\pi_ {\mathsf {N E}}} \right] = 2 \alpha (1 - \alpha). \end{array} \right. \tag {1}
$$
In such a case,  $\Gamma^{+}$  is always "not worse" than  $\Gamma^{-}$  as it contains more expert data.
4 Learning frameworks for Solving VPII Problems
In this section, we analyze the core challenge in VPIL problems with the pairwise datasets  $\Gamma^{+}$  and  $\Gamma^{-}$ , i.e., recovering the occupancy measure of the expert policy  $p_{\pi_E}$ . To this end, we propose two learning frameworks for VPIL with known  $\alpha$  and unknown  $\alpha$ , respectively.
4.1 VPIL with Known  $\alpha$
Most of the IL methods based on the occupancy measure need to assume that the demonstrations are sampled only from the expert policy, so that they can directly estimate  $p_{\pi_E}$  and match the learner's distribution with  $p_{\pi_E}$ . However, the occupancy measure of the expert policy  $p_{\pi_E}$  is inaccessible in the VPIL problem, since both  $\Gamma^+$  and  $\Gamma^-$  contain non-expert data while the label of each data is unavailable. Below, we attempt to approximate  $p_{\pi_E}$  with  $\{\Gamma^+, \Gamma^-\}$ .
Let  $n_{+} = |\Gamma^{+}|$ ;  $\widehat{p}_i^+(s,a) = (1 - \gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{1}[(s_{t,i},a_{t,i}) = (s,a)]$  for  $i = 1,\dots,n_{+}$  be the empirical occupancy measure of a trajectory  $\tau_i \in \Gamma^+$ , where  $(s_{t,i},a_{t,i}) \in \tau_i$  is a state action pair at time  $t$ . Let  $\widehat{p}_{\pi_+}(s,a) = \sum_{i=1}^{n_+}\widehat{p}_i^+(s,a)/n_+$ . Define  $\widehat{p}_{\pi_-}(s,a)$  similarly. We have the following theorem.
Theorem 2. Assume the pairwise datasets  $(\Gamma^{+},\Gamma^{-})$  are generated following the procedure in Section 3.2. Let  $p_{\pi_+}(s,a) = \mathbb{E}_{\Gamma^+}\left[\widehat{p}_{\pi_+}(s,a)\right]$  and  $p_{\pi_-}(s,a) = \mathbb{E}_{\Gamma^-}\left[\widehat{p}_{\pi_-}(s,a)\right]$  be the expected occupancy measures of  $\Gamma^{+}$  and  $\Gamma^{-}$ , where the randomness is taken over the draws of  $\Gamma^{+}$  and  $\Gamma^{-}$ . Then, we have
$$
\left\{ \begin{array}{l} p _ {\pi_ {+}} (s, a) = \quad \left(2 \alpha - \alpha^ {2}\right) p _ {\pi_ {E}} (s, a) + (1 - \alpha) ^ {2} p _ {\pi_ {N E}} (s, a), \end{array} \right. \tag {2}
$$
$$
\left\{ \begin{array}{l l} p _ {\pi_ {E}} (s, a) = & \frac {1 + \alpha}{2 \alpha} p _ {\pi_ {+}} (s, a) - \frac {1 - \alpha}{2 \alpha} p _ {\pi_ {-}} (s, a), \\ p _ {\pi_ {N E}} (s, a) = & - \frac {\alpha}{2 (1 - \alpha)} p _ {\pi_ {+}} (s, a) + \frac {2 - \alpha}{2 - 2 \alpha} p _ {\pi_ {-}} (s, a). \end{array} \right. \tag {3}
$$
A proof can be found in Appendix A. The Theorem provides a feasible way to recover the unknown occupancy measure of the expert policy from contaminated data pools  $\Gamma^{+}$  and  $\Gamma^{-}$ . Thus, we can use an off-the-shelf IL method to learn the policy.
Once we have recovered  $p_{\pi_E}$  by Eq. (10), we can use an off-the-shelf IL method to learn the policy.
4.2 VPIL with Unknown  $\alpha$
When facing a more challenging problem in which the expert ratio  $\alpha$  is unknown, a straightforward way is to estimate the ratio first and then reconstruct the expert policy by the approach developed for known  $\alpha$  cases. Here, we will first introduce how to estimate the expert ratio by reducing it into a mixture proportion estimation (MPE) problem [28] and then identify that a direct application of the estimated ratio is not accurate enough to reconstruct the expert policy. To this end, we further propose an iterative sample selection procedure to exploit the estimated expert ratio, which finally leads to a better approximation of the expert policy.
Estimation of the Expert Ratio  $\alpha$ . In this paragraph, we show that the estimation of the expert ratio  $\alpha$  can be reduced to two MPE problems [28]. Let  $\mathcal{P}$  and  $\mathcal{N}$  be two probability distributions, and
$$
\mathcal {U} = \beta \mathcal {P} + (1 - \beta) \mathcal {N} \tag {4}
$$
be a mixture of them with a certain proportion  $\beta \in (0,1)$ . The MPE problem studies how to estimate mixture proportion  $\beta$  with the empirical observations sampled from  $\mathcal{U}$  and  $\mathcal{P}$  (not from  $\mathcal{N}$ ). Over the decades, various algorithms were proposed with sound theoretical and empirical studies [31, 32].
To see how the estimation of the expert ratio  $\alpha$  is related to the MPE problem, we rewrite (2) as
$$
p _ {\pi_ {+}} (s, a) = \beta_ {1} p _ {\pi_ {-}} (s, a) + (1 - \beta_ {1}) p _ {\pi_ {\mathsf {E}}} \quad \text {a n d} \quad p _ {\pi_ {-}} (s, a) = \beta_ {2} p _ {\pi_ {+}} (s, a) + (1 - \beta_ {2}) p _ {\pi_ {\mathsf {N E}}}, \tag {5}
$$
where  $\beta_{1} = \frac{1 - \alpha}{1 + \alpha}$  and  $\beta_{2} = \frac{\alpha}{2 - \alpha}$ . By taking  $p_{\pi_{+}}$  as  $\mathcal{U}$  and  $p_{\pi_{-}}$  as  $\mathcal{P}$ , the first line of (5) shares the same formulation as the MPE problem (4). Since  $p_{\pi_{+}}$  and  $p_{\pi_{-}}$  are empirically accessible via  $\Gamma^{+}$  and  $\Gamma^{+}$ , we can estimate  $\beta_{1}$  by taking  $\Gamma^{+}$  and  $\Gamma^{-}$  as the input of any MPE solver and obtain the expert ratio by  $\alpha = \frac{1 - \beta_{1}}{1 + \beta_{1}}$ . The same argument also holds for the second line of (5), where we can take  $p_{\pi_{-}}$  as  $\mathcal{U}$  and  $p_{\pi_{+}}$  as  $\mathcal{P}$  to estimate  $\beta$ . Then, the expert ratio can also be obtained by  $\alpha = \frac{2\beta_{2}}{1 + \beta_{2}}$ .
One might worry about the identifiability of the expert ratio  $\alpha$  by estimating it with MPE techniques [28]. We can show that the true parameter is identifiable if the distribution  $p_{\pi_+}$  and distribution  $p_{\pi_-}$  are mutually irreducible [16] as follows:
Proposition 1. Suppose the distributions  $p_{\pi_E}$  and  $p_{\pi_{NE}}$  are mutually irreducible such that there exists no decomposition of the form  $p_{\pi_E} = (1 - \eta)Q + \eta p_{\pi_{NE}}$  and  $p_{\pi_{NE}} = (1 - \eta')Q' + \eta' p_{\pi_E}$  for any probability distributions  $Q, Q'$  and scalars  $\eta, \eta' \in (0,1]$ . Then, the true mixture proportions  $\beta_1$  and  $\beta_2$  are unique and can be identified by
$$
\left\{ \begin{array}{l} \beta_ {1} = \sup  \left\{\eta \mid p _ {\pi_ {+}} = \eta p _ {\pi_ {-}} + (1 - \eta) K, K \in \mathcal {C} \right\}, \\ \beta_ {2} = \sup  \left\{\eta^ {\prime} \mid p _ {\pi_ {-}} = \eta^ {\prime} p _ {\pi_ {+}} + (1 - \eta^ {\prime}) K ^ {\prime}, K ^ {\prime} \in \mathcal {C} \right\}, \end{array} \right. \tag {6}
$$
where  $\mathcal{C}$  is the set containing all possible probability distributions. Thus,  $\alpha$  is identifiable by
$$
\alpha = (1 - \beta_ {1}) / (1 + \beta_ {1}) \quad o r \quad \alpha = 2 \beta_ {2} / (1 + \beta_ {2}). \tag {7}
$$
Proposition 1 demonstrates that the true mixture proportion  $\beta_{1}$  (resp.  $\beta_{2}$ ) can be identified by finding the maximum proportion of  $p_{\pi_{-}}$  contained in  $p_{\pi_{+}}$  (resp.  $p_{\pi_{+}}$  contained in  $p_{\pi_{-}}$ ). This idea can be empirically implemented via existing MPE methods [28, 33]. We note that the expert ratio is identifiable by either estimating  $\beta_{1}$  or  $\beta_{2}$  when we have infinite samples. In the finite sample case, the two estimators coming from different distribution components could lead to different estimation biases. Since the MPE solutions tend to have a positive estimation bias on the true value  $\beta_{1}$  and  $\beta_{2}$  as shown by [28, Theorem 12] and [31, Corollary 1], the estimation  $\alpha = (1 - \beta_{1}) / (1 + \beta_{1})$  tends to yield an underestimated  $\alpha$  while that with  $\beta_{2}$  will lead to an overestimation. Besides, when the number of expert data is quite small, the underlying true parameter  $\beta_{2}$  would also have a small value,
Algorithm 1 ExpertRatioEstimation
Input: Pairwise Demonstrations  $\Gamma^{+},\Gamma^{-}$  ; Numbers of Iterations  $I$    
Output: Estimated Expert's Ratio  $\hat{\alpha}$    
1: function ESTIMATION(  $\Gamma^{+},\Gamma^{-}$  1   
2: Initialize  $\mathcal{D}^+\gets \Gamma^+$  and  $\mathcal{D}^{-}\leftarrow \Gamma^{-}$    
3: for each iteration until I do   
4: Train a binary classifier  $f_{\psi}:S\times \mathcal{A}\to [0,1]$  by taking  $\mathcal{D}^+$  as positive data and  $\mathcal{D}^{-}$  as negative.   
5: Assign a score for each data  $S^{+}\gets \{f_{\psi}(s,a)\mid (s,a)\in \mathcal{D}^{+}\}$  and  $S^{-}\gets \{f_{\psi}(s,a)\mid$ $(s,a)\in \mathcal{D}^{-}\}$    
6: Estimate  $\beta$  with a mixture proportion estimator by taking  $S^{+}$  as  $\mathcal{U}$  and  $S^{-}$  as  $\mathcal{P}$  in (4).   
7: Estimate  $\widehat{\alpha}$  by (7).  $\triangleright$  We choose  $(1 - \beta) / (1 + \beta)$  for better estimation.   
8: Data selection for  $\Gamma^{+}\colon \mathcal{D}^{+}\gets \{2\widehat{\alpha} -\widehat{\alpha}^{2}$  fraction of  $\Gamma^{+}$  with top scores  $S^{+}\}$    
9: Data selection for  $\Gamma^{-}\colon \mathcal{D}^{-}\gets \{1 - \widehat{\alpha}^{2}$  fraction of  $\Gamma^{-}$  with bottom scores  $S^{-}\}$    
10: end for   
11: return  $\widehat{\alpha}$    
12: end function