SlowGuess/NeurIPS_2023_OCR huggingface dataset:
James Cheshire StephanClemencon
first.last@telecom-paris.fr
Vincent Laurent
ENS Paris Saclay
first.last@ens-paris-saclay.fr
In this paper, we develop an active learning framework for the bipartite ranking problem. Motivated by numerous applications, ranging from supervised anomaly detection to credit-scoring through the design of medical diagnosis support systems, and usually formulated as the problem of optimizing (a scalar summary of) the ROC curve, bipartite ranking has been the subject of much attention in the passive context. Various dedicated algorithms have been recently proposed and studied by the machine-learning community. In contrast, active bipartite ranking rule is poorly documented in the literature. Due to its global nature, a strategy for labeling sequentially data points that are difficult to rank w.r.t. to the others is required. This learning task is much more complex than binary classification, for which many active algorithms have been designed. It is the goal of this article to provide a rigorous formulation of such a selective sampling approach. We propose a dedicated algorithm, referred to as active-rank, which aims to minimise the distance between the ROC curve of the ranking function built and the optimal one, w.r.t. the sup norm. We show that, for a fixed confidence level  $\varepsilon$  and probability  $\delta$ , active-rank is  $\mathrm{PAC}(\varepsilon, \delta)$ . In addition, we provide a problem dependent upper bound on the expected sampling time of active-rank and also demonstrate a problem dependent lower bound on the expected sampling time of any  $\mathrm{PAC}(\varepsilon, \delta)$  algorithm. Beyond the theoretical analysis carried out, numerical results are presented, providing strong empirical evidence of the performance of the algorithm proposed, which compares favorably with more naive approaches.
In bipartite ranking, the statistical framework is exactly the same as that in standard binary classification, the flagship problem in statistical learning theory. One observes  $n \geq 1$  independent copies  $\mathcal{D}_n = \{(X_1, Y_1), \ldots, (X_n, Y_n)\}$  of a generic random pair  $(X, Y)$  with (unknown) distribution  $P$ , where  $Y$  is a binary random label, valued in  $\{-1, +1\}$  say, and  $X$  is a high dimensional random vector, taking its values in  $\mathcal{X} \subset \mathbb{R}^d$  with  $d \geq 1$ , that models some information hopefully useful to predict  $Y$ . In contrast to binary classification, the goal pursued is of global (and not local) nature. It is not to assign a label, positive or negative, to any new input observation  $X$  but to rank any new set of (temporarily unlabeled) observations  $X_1', \ldots, X_{n'}'$  by means of a (measurable) scoring function  $s: \mathcal{X} \to \mathbb{R}$ , so that those with positive label appear on top of the list (i.e. are those with the highest scores) with high probability. More formally, the accuracy of any scoring rule can be evaluated through the ROC curve criterion or its popular scalar summary, the AUC (standing for the Area Under the ROC Curve), and, as expected, optimal scoring functions w.r.t. these performance measures can be shown to be increasing transforms of the posterior probability  $\eta(x) = \mathbb{P}\{Y = +1 \mid X = x\}$ ,
$x \in \mathcal{X}$ . Though easy to formulate, this problem encompasses many applications, ranging from credit risk screening to the design of decision support tools for medical diagnosis through (supervised) anomaly detection. Hence, motivated by a wide variety of applications, bipartite ranking has received much attention these last few years. Many approaches to this global learning problem (i.e. the problem of learning a preorder on the input space  $\mathcal{X}$  based on a binary feedback) have been proposed and investigated. In Clémençon and Vayatis (2009), optimization of the (empirical) ROC curve in sup norm is considered via nonlinear approximation techniques, while this functional optimization problem is viewed as a superposition of cost-sensitive binary classification problems in Clémençon and Vayatis (2008) so as to propose an alternative method. In Clémençon et al. (2008) (see also Agarwal et al. (2005)), empirical maximization of the empirical AUC criterion is considered, bipartite ranking being viewed as a pairwise classification problem, a plug-in approach to bipartite ranking is developed in Clémençon and Robbiano (2011) and scalar performance criteria other than the AUC have been recently reviewed in Menon and Williamsson (2016). Whereas the vast majority of dedicated articles consider the batch situation solely, where the learning procedure fully relies on a set  $\mathcal{D}_n$  of training examples given in advance, the goal of this paper is to develop an active learning framework for bipartite ranking, in other words to investigate this problem in an iterative context, where the learning procedure can formulate queries in a sequential manner, so as to observe the labels at new data points in order to refine progressively the scoring/ranking model. Precisely, the challenge consists in determining an incremental experimental design to label the data points in  $\mathcal{X}$  that would permit to improve the ROC curve progressively, with statistical guarantees.
Our contributions We describe an algorithm, active-rank, which sequentially queries points of the feature space  $\mathcal{X}$ . Given a confidence level  $\varepsilon > 0$  and probability  $\delta > 0$ , the goal of active-rank is, in a few queries as possible, to output a ranking of  $\mathcal{X}$ , such that the induced ROC curve of said ranking is within  $\varepsilon$  of the optimal ROC curve, in terms of the sup norm, with probability greater than  $1 - \delta$ . We restrict our selves to dimension 1, i.e.  $\mathcal{X} = [0,1]$  and make a single key assumption that the posterior  $\eta$  is piecewise constant on a grid of size  $K$ . Theorem 3.2 then shows that active-rank satisfies the above statistical guarantee and furthermore, provides an upper bound on its expected total number of queries. In Theorem 3.3 we provide a lower bound on the expected number of queries for any possible policy, which satisfies a confidence level  $\varepsilon$  with probability greater than  $1 - \delta$ . The aforementioned bounds are problem dependent, in the sense that they depend on features of the posterior  $\eta$ . Finally we conduct a practical analysis of active-rank on synthetic data, comparing it to several naive approaches.
The article is structured as follows. In Section 2 we formally define our setting as well recalling some key notions related to bipartite ranking and ROC analysis. In Section 2 we also cover some of the existing literature in active learning, of relevance to bipartite ranking. We then describe the active-rank algorithm in Section 3.1. Following this our theoretical results are presented in Section 3.2. Lastly, the experiments are displayed in Section 4. We then conclude and discuss some perspectives for future research in Section 5. Technical details and proofs are deferred to the Supplementary Material.
2 Background and preliminaries
2.1 Notation
Here we introduce several dedicated notions that will be extensively used in the subsequent analysis. For any integer  $n \geq 1$ , we set  $[n] := \{1, \dots, n\}$ , denote by  $\mathfrak{S}_n$  the symmetric group of permutations on  $\{1, \dots, n\}$ , by  $\mathbf{I}_n$  the identity map of  $\mathfrak{S}_n$ . By  $\lambda$  is meant the Lebesgue measure on  $[0,1]$ . Given two probability distributions  $P$  and  $Q$  on a measurable space  $(\Omega, \mathcal{F})$ , we write  $P \ll Q$  when  $P$  is absolutely continuous w.r.t.  $Q$ . For any  $a, b$  in  $[0,1]$ ,  $\mathcal{B}\mathrm{er}(a)$  refers to the Bernoulli distribution with mean  $a$  and  $\mathrm{kl}(a,b)$  to the Kullback Leibler divergence between the Bernoulli distributions  $\mathcal{B}\mathrm{er}(a)$  and  $\mathcal{B}\mathrm{er}(b)$ . For any  $a, b \in [0,1]$ , the Chernoff Information between the distributions  $\mathcal{B}\mathrm{er}(a)$  and  $\mathcal{B}\mathrm{er}(b)$  is defined as,  $\mathrm{kl}^*(a,b) = \mathrm{kl}(x^*,a) = \mathrm{kl}(x^*,b)$ , where  $x^*$  is the unique  $x \in [0,1]$  such that  $\mathrm{kl}(x,a) = \mathrm{kl}(x,b)$ . The indicator function of any event  $\mathcal{E}$  is denoted by  $\mathbb{I}\{\mathcal{E}\}$ , the Dirac mass at any point  $x$  by  $\delta_x$ , and the pseudo-inverse of any cdf  $\kappa(u)$  on  $\mathbb{R}$  by  $\kappa^{-1}(t) = \inf\{v \in \mathbb{R} : \kappa(v) \geq t\}$ .
2.2 Setting
The bipartite ranking problem A rigorous formulation of bipartite ranking involves functional performance measures. Let  $S$  be the set of all scoring functions, any  $s \in S$  defines a preorder  $\preceq_s$  on  $\mathcal{X}$ : for all  $(x, x') \in \mathcal{X}^2$ ,  $x \preceq_s x' \Leftrightarrow s(x) \leq s(x')$ . From a quantitative perspective, the accuracy of any scoring rule can be evaluated through the ROC curve criterion, namely the PP-plot  $t \in \mathbb{R} \mapsto (1 - H_s(t), 1 - G_s(t))$ , where  $H_s(t) = \mathbb{P}\{s(X) \leq t \mid Y = -1\}$  and  $G_s(t) = \mathbb{P}\{s(X) \leq t \mid Y = +1\}$ , for all  $t \in \mathbb{R}$ . The curve can also be viewed as the graph of the càd-làg function  $\alpha \in (0, 1) \mapsto \mathrm{ROC}(s, \alpha) = 1 - G_s \circ H_s^{-1}(1 - \alpha)$ . The notion of ROC curve defines a partial order on the set of all scoring functions (respectively, the set of all preorders on  $\mathcal{X}$ ):  $s_1$  is more accurate than  $s_2$  when  $\mathrm{ROC}(s_2, \alpha) \leq \mathrm{ROC}(s_1, \alpha)$  for all  $\alpha \in (0, 1)$ . As can be proved by a straightforward Neyman-Pearson argument, the set  $S^*$  of optimal scoring functions is composed of increasing transforms of the posterior probability  $\eta(x) = \mathbb{P}\{Y = +1 \mid X = x\}$ ,  $x \in \mathcal{X}$ . We have  $S^* = \{s \in S : \forall (x, x') \in \mathcal{X}^2, \eta(x) < \eta(x') \Rightarrow s^*(x) < s^*(x')\}$  and
$$
\forall (s, s ^ {*}) \in \mathcal {S} \times \mathcal {S} ^ {*}, \forall \alpha \in (0, 1), \operatorname {R O C} (s, \alpha) \leq \operatorname {R O C} ^ {*} (\alpha) := \operatorname {R O C} (s ^ {*}, \alpha).
$$
The ranking performance of a candidate  $s \in S$  can be thus measured by the distance in sup-norm between its ROC curve and ROC*, namely  $d_{\infty}(s, s^{*}) \coloneqq \sup_{\alpha \in (0,1)} \{ROC^{*}(\alpha) - ROC(s, \alpha)\}$ . An alternative convention to represent the ROC of a scoring function  $s$ , which we will use for the remainder of this paper, is to consider the broken line  $\widetilde{\mathrm{ROC}}(s,.)$ , which arises from connecting the PP-plot by line segments at each possible jump of the cdf  $H_{s}$ . From here on out when referring to the ROC of a scoring function  $s$ , we refer to the broken line  $\widetilde{\mathrm{ROC}}(s,.)$ .
The active learning setting Whereas in the batch mode, the construction of a nearly optimal scoring function (i.e. a function  $s \in S$  such that  $d_{\infty}(s, s^{*})$  is 'small' with high probability) is based on a collection of independent training examples given in advance, the objective of an active learner is to formulate queries in order to recover sequentially the optimal preorder on the feature space  $\mathcal{X}$  defined by the supposedly unknown function  $\eta$ . That is, the active learner plays a game with multiple time steps, where, at time each step  $n$ , they must choose a point  $a_{n} \in \mathcal{X}$  to query, so as to observe the random label  $Y_{n} \sim \mathcal{B}\mathrm{er}(\eta(a_{n}))$  and refine the scoring model incrementally. After a sufficient number of rounds has elapsed, chosen at the learner's discretion, a final scoring function  $\hat{s}$ , is output.
Piecewise constant scoring functions. Here we consider the simplest scoring functions, measurable functions that are constant on pieces of the input space  $\mathcal{X}$  forming a partition. As shown in Clemencon and Vayatis (2009) (see subsection 2.3 therein), when smooth enough,  $\mathrm{ROC}^*$  can be accurately approximated by the (stepwise) ROC curve of a piecewise constant scoring function. Because the goal of this paper is to highlight the nature of active bipartite ranking rather than treating the problem in full generality, various simplifying assumptions are made in the subsequent analysis. For simplicity we suppose that  $\mathcal{X} = [0,1)$  and introduce the grid points  $\{G_1,\dots,G_K\} = \{i / K: i = 1,\dots,K - 1\}$ , where  $K\geq 1$ . A preorder on  $\mathcal{X}$  can be then naturally defined by means of a permutation  $\sigma \in \mathfrak{S}_K$ . Consider indeed the scoring function
$$
s _ {\sigma} (x) := \sum_ {i = 1} ^ {K} i \cdot \mathbb {I} \{x \in \left[ G _ {\sigma (i)}, G _ {\sigma (i + 1)}\right) \}. \tag {1}
$$
We denote by  $S_K$  the set of all functions of type (1). To avoid dealing with model bias here, we assume that the optimal preorder, that induced by  $\eta(x)$  namely, can be defined by a scoring function in  $S_K$ .
Assumption 2.1. There exist a permutation  $\sigma \in \mathfrak{S}_K$  and distinct constants  $\mu_1, \ldots, \mu_K$  in  $(0,1)$  such that
$$
\eta (x) = \sum_ {i = 1} ^ {K} \mu_ {i} \cdot \mathbb {I} \{x \in \left[ G _ {\sigma (i)}, G _ {\sigma (i + 1)}\right) \} \text {f o r a l l} x \in [ 0, 1).
$$
We write  $p = \frac{1}{K}\sum_{i\in [K]}\mu_i$ . We point out that, as  $\eta$  may remain constant over multiple sections of the grid, the permutation  $\sigma$  satisfying assumption 2.1, is not necessarily unique. In the subsequent analysis, the parameter  $K$  is supposed to be known, in contrast with the  $\mu_i$ 's, which have to be learned by means of an active strategy. As we assume no structure between the grid points, one can easily consider Assumption 2.1 in higher dimensions, simply consider the  $d$  dimensional grid of size  $K^d$ . As such, all results presented in this paper extend trivially to higher dimensions.
Policies and fixed confidence regime. We denote the outputted scoring function of the learner  $\hat{s} \in S_K$ . The way the learner interacts with the environment - i.e. their choice of points to query, how many samples to draw in total and their choice of  $\hat{s} \in S_K$ , we term the policy of the learner. We write  $\mathcal{C}$  for the set of all possible policies of the learner. For a policy  $\pi \in \mathcal{C}$  and problem  $\nu \in \mathcal{B}$  we denote random variable  $\tau_{\nu}^{\pi}$  as the stopping time of policy  $\pi$ . We write  $\hat{s}_{\nu}^{\pi}$  for the scoring function outputted by policy  $\pi$  on problem  $\nu$ . Where obvious we may drop the dependency on  $\pi, \nu$  in the notation, referring to the scoring function outputted by the learner as simply  $\hat{s}$ . We write  $\mathbb{P}_{\nu, \pi}$  as the distribution on all samples gathered by a policy  $\pi$  on problem  $\nu$ . We similarly define  $\mathbb{E}_{\nu, \pi}$ .
For the duration of this paper we will work in the fixed confidence regime. For a confidence level  $\varepsilon$ , define,  $S_K^\varepsilon \coloneqq \{s \in S_K : d_\infty(s, \eta) \leq \varepsilon\}$ . A policy  $\pi$  is said to be  $\mathrm{PAC}(\delta, \varepsilon)$  (probably approximately correct), on the class of problems  $\mathcal{B}$ , if,  $\forall \nu \in \mathcal{B}, \mathbb{P}_{\nu, \pi}[\hat{s} \in S_K^\varepsilon] \geq 1 - \delta$ . The goal of the learner is to then obtain a  $\mathrm{PAC}(\delta, \varepsilon)$  policy  $\pi$ , such that the expected stopping time in the worst case,  $\sup_{\nu \in \mathcal{B}} \mathbb{E}_{\nu, \pi}[\tau_\nu^\pi]$ , is minimised.
Defining problem complexity The expected minimum number of samples a policy must draw on a certain problem,  $\nu \in \mathcal{B}$ , to be  $\mathrm{PAC}(\delta, \varepsilon)$  is a quantity which depends upon the features of  $\nu$ , specifically, the shape of the posterior  $\eta$ . When defining our measure of problem complexity we must capture this dependence as succinctly as possible. To build intuition for our definition we first explore a naive strategy and introduce some informative Lemmas. A naive approach to the active bipartite ranking problem, is to treat each pair of points on the grid,  $i, j \in [K]$  as a separate classification problem. To correctly distinguish the situations,  $\mathcal{H}_0^{i,j} \coloneqq \mu_i > \mu_j$ ,  $\mathcal{H}_1^{i,j} \coloneqq \mu_i < \mu_j$ , with probability greater than  $1 - \delta$ , it is well known, see e.g. Kaufmann et al. (2014), that for small  $\delta$ , the minimum number of samples required is of the order  $\frac{\log(1/\delta)}{\mathrm{kl}^*(\mu_j, \mu_i)}$ , where we remind the reader  $\mathrm{kl}^*$  is the Chernoff Information, closely related to the kl divergence, see Section 2.1. Thus, if the learner wished to output a scoring function in  $S^*$ , the sample complexity would be of the order,  $\sum_{i \in [K]} \frac{\log(1/\delta)}{\min_{j \in [K]} (\mathrm{kl}^*(\mu_j, \mu_i))}$ . Of course, distinguishing between  $\mathcal{H}_0^{i,j}$  and  $\mathcal{H}_1^{i,j}$  is impractical when  $\mu_i$  and  $\mu_j$  are very close, or even equal. However, in our regime, the learner is not required to correctly rank every pair of points  $i, j \in [K]$ , only to output a scoring function existing in  $S_K^\varepsilon$ . Intuition indicates that the learner may be irrelevant to the ranking within certain groups of points on the gird, as long as there posterior values are sufficiently close. For instance, consider a partition of  $[0,1]$ ,  $\mathcal{P} = \{C_1, C_2, C_3\}$ , and increasing sequence  $(\beta_1, \beta_2, \beta_3) \in [0,1]^3$  with
$$
\eta (x) = \sum_ {i = 1} ^ {3} \mathbb {I} (x \in C _ {i}) \beta_ {i}, \quad \tilde {s} (x) = \mathbb {I} (x \in C _ {1}) + 2 \mathbb {I} (x \in \{C _ {2} \cup C _ {3} \}), \tag {2}
$$
Where the scoring function function  $\tilde{s}$  essentially, treats all points of  $C_2, C_3$  of the same rank. See Figure 1 for the ROC curves of  $\eta$  and  $\tilde{s}$ . Via simple calculation, we have the following,  $d_{\infty}(\eta, \tilde{s}) = \frac{\lambda(C_3)}{p} \frac{(\beta_3 - \beta_2) / 2}{1 - (\beta_3 + \beta_2) / 2}$ , which suggests that, whether or not there exists a scoring function in  $S_K^{\varepsilon}$ , which treats the groups  $C_2, C_3$  of the same rank, depends upon two things, the size of the groups and also their position on the ROC curve. Specifically, if  $\beta_3 - \beta_2 \geq \frac{2\varepsilon p(1 - (\beta_3 + \beta_2) / 2)}{\lambda(C_3)}$ , then  $\tilde{s} \notin S_K^{\varepsilon}$ . The following lemma formalises this intuition, the proof follows via a direct generalisation of the above example.
Lemma 2.2. Let  $\Delta >0$ ,  $i\in [K]$  and define  $S_K^{(i,\Delta)}$  as the set of scoring functions such that, for all  $s\in S_K^{(i,\Delta)}$ , one has that  $\forall j:|\mu_j - \mu_i|\geq \Delta$ ,  $\mathrm{sign}(s(i) - s(j)) = \mathrm{sign}(\mu_i - \mu_j)$ . There exist a  $\tilde{s}\in S_K^{(i,\Delta)}$ ,  $\nu \in \mathcal{B}$ , such that on problem  $\nu$ ,  $d_{\infty}(\eta ,\tilde{s})\geq \frac{\Delta|\{j:|\mu_i - \mu_j|\leq\Delta\}|}{p(1 - \mu_i)}$ .
Lemma 2.2 suggests that the for  $i \in [K]$  the learner must be at least able to distinguish  $\mathcal{H}_0^{i,j}$  vs  $\mathcal{H}_1^{i,j}$ , for all  $j: |\mu_i - \mu_k| \leq \tilde{\Delta}_i$ , where,  $\tilde{\Delta}_i := \max \left\{x > 0: \sum_{i \neq j} x \mathbb{I}(|\mu_i - \mu_j| \leq x) < K \varepsilon p(1 - \mu_i)\right\}$ .
The following Lemma shows that  $\tilde{\Delta}_i$  is not only an upper bound on the necessary order of the confidence level around  $\mu_i$  but also a lower bound, the proof of which can be found in the proof of Theorem 3.2.
Lemma 2.3. For a problem  $\nu \in \mathcal{B}$ , let  $s \in S_K$  be a scoring function such that the following holds,  $\forall i \in [K]$ ,
$$
\forall j: | \mu_ {j} - \mu_ {i} | \geq \tilde {\Delta} _ {i} / 4, \operatorname {s i g n} (s (i) - s (j)) = \operatorname {s i g n} (\mu_ {i} - \mu_ {j})
$$
Figure 1: The ROC curves of  $\eta$  and  $\tilde{s}$ , as defined in Equations (2), with  $(\beta_{1},\beta_{2},\beta_{3}) = (0.1,0.2,0.6)$  and  $C_1,C_2,C_3$  equally sized.
Figure 2: The complexity of points  $i \in [K]$ , for  $\eta$  as defined in Scenario 1 of the experiments, see Section 4.
then  $s\in S_K^\varepsilon$
Under the conditions where  $K\varepsilon p > 1$  and for a grid point  $i \in [K]$ ,  $\mu_{i}$  is close to one, a certain phenomenon occurs. Specifically, when  $\tilde{\Delta}_i \geq 1 - \mu_i$ , for the learner to know the value of  $\mu_{i}$  up to an error of  $\tilde{\Delta}_i$  is no longer sufficient, as in such a case, to the best of the learners knowledge, the value of  $\mu_{i}$  may be arbitrarily close to 1, and thus have an arbitrarily large effect on the regret. With this in mind, we define  $\Delta_{i}$  as follows,
$$
\Delta_ {i} := \max  \left\{x > 0: \sum_ {i \neq j} x \mathbb {I} \big (| \mu_ {i} - \mu_ {j} | \leq x \big) <   K \varepsilon p (1 - \mu_ {i}) \right\} \wedge (1 - \mu_ {i})
$$
Note that in the case where  $K\varepsilon p < 1$ , we have  $\Delta_i = \tilde{\Delta}_i$ . In the case where  $\mu_i \geq \Delta_i$ , we thus define the complexity at a point  $i \in [K]$  as  $H_i^{(1)} = \frac{1}{\mathrm{kl}(\mu_i, \mu_i + \Delta_i) \wedge \mathrm{kl}(\mu_i, \mu_i - \Delta_i)}$ . If  $\mu_i \leq \Delta_i$ ,  $H_i^{(1)} = \frac{1}{\mathrm{kl}(\mu_i, \mu_i + \Delta_i)}$ . For a problem  $\nu \in \mathcal{B}$ , the total problem complexity is then given as  $\sum_{i \in [K]} H_i^{(1)}$ . As an illustrative example, see Figure 2 for the complexity of  $i \in [K]$  with  $\eta$  defined as in Scenario 1 of the experiments, see section 4.
There is a natural comparison to multi armed bandits, where the problem complexity is typically given as the summation across the individual complexity of each arm. However, in most multi armed settings the complexity of a single arm is dependent upon its distance to a single other arm, e.g. the optimal arm, whereas in our setting the complexity of a single grid point  $i \in [K]$  has a more complex dependency on the shape of the posterior around  $G_{i}$ .
2.3 Performance of the passive approach
At this point we can consider how a uniform sampling strategy would perform, that is, the learner simply draws a sample from each section of the grid in turn. This would be essentially a passive approach, in line with the classical batch setting. For a uniform/passive sampling strategy to be  $\mathrm{PAC}(\varepsilon ,\delta)$  one would have to draw samples until the width of the confidence interval at all points  $i$  is less than  $\Delta_{i}$ . Therefore, a uniform sampling strategy, with an appropriate stopping rule, would have the following tight upper bound on its expected sampling time, up to log terms,  $cK\max_{i\in [K]}\frac{1}{kl(\mu_i,\mu_i + c'\Delta_i)}$ , for some absolute constants  $c,c' > 0$ . The improvement we hope to gain in the active setting is to replace the max with a weighted summation across the grid. Thus, in settings where the  $\Delta_{i}$  are relatively constant across large sections of the grid, then the theoretical performance of a passive approach can be close to optimal. In contrast, in cases where a very small section of the interval is hard to rank and the rest is easy - i.e.  $K\max_{i\in [K]}\frac{1}{kl(\mu_i,\mu_i + \Delta_i)}$  is much greater than  $\sum_{i\in [K]}\frac{1}{kl(\mu_i,\mu_i + \Delta_i)}$ , a passive approach will fail. Such settings involve large  $K$  where the gaps  $\Delta_{i}$  on the majority of cells are large, with a relatively small number of cells with small gaps  $\Delta_{i}$ . Incidentally, we point out that this corresponds to many situations of interest in practice (in
information retrieval, for a specific request, the vast majority of the documents are equally irrelevant, while the ranking of a very small fraction of relevant documents is challenging; the same phenomenon is also observed in credit-risk screening). In these cases the benefit to the practitioner, will be that they quickly focus on the interesting sections of the feature space.
2.4 Related literature in active learning