SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present the sparse variant of BCA (Algorithm 1) in Algorithm 5 and the sparse variant of Greedy (Algorithm 3) in Algorithm 7. The sparse versions of their specialized counterparts for coverage are presented in Algorithm 6 and Algorithm 8.
E Extended results, experiments and details of empirical comparison
E.1 Datasets characteristics
In Table 5 we include an overview of the main characteristics of the datasets used in this paper.
Table 5: Multi-label datasets from XMLC repository [6]. APpL and ALpP represent average points per label and average labels per point, respectively.
Dataset	#Labels	#Training	#Testing	#Features	APpL	ALpP
EURLEX-4K	3,993	15,539	3,809	5,000	25.7	5.3
WIKI-31K	30,938	14,146	6,616	101,938	8.5	18.6
AMAZONCAT-13K	13,330	1,186,239	306,782	203,882	448.6	5.0
WIKIPEDIALARGE-500K	501,070	1,813,391	783,743	2,381,304	24.8	4.8
AMAZON-670K	670,091	490,449	153,025	135,909	4.0	5.5
E.2 Extended results of the main experiment
In this section, we present the extended results of our main experiment from Section 7. Here in Table 6 and Table 7, we present additional results for  $k = 1$ , as well as results for additional methods:
- MACRO- $\mathbf{P}_{\mathrm{GREEN}}$ , MACRO-F1 $_{\mathrm{GREEN}}$ , COV $_{\mathrm{GREEN}}$ - the greedy algorithms (Algorithm 3) for optimizing macro-precision, -F1, and coverage (Algorithm 4),
- MACRO- $\mathbf{R}_{\mathrm{PRIOR}}$ - the optimal strategy for macro-recall: selection of  $k$  labels with the highest  $p_j^{-1}\eta_j$ , with  $p$  estimated on the training set.
Because both greedy and block coordinate-ascent algorithms depend on the order in which examples are presented, we check if this has a significant influence on the results. We ran the procedures 5 times with different seeds and presented both mean and standard deviation, which is mostly less than 0.25 for the macro measures.
E.3 Results on true labels
In the main experiment, the reported performance depends on three things: the inherent difficulty of the data, the success of the inference algorithm and the quality of the provided marginal probabilities. To be able to judge these terms, we also ran the inference algorithms on probabilities generated from the test set labels:  $\eta_j(\pmb{x}_i) = 1$  iff  $Y_{ij} = 1$  to rest only our inference strategy. We present the result of this experiment in Table 8. Notice that in this experiment, the specialized inference strategies are almost always the best on measures they aim to optimize. Also, the obtained results for macro measures are significantly below  $100\%$ . This is because, in this datasets, not all labels are represented with even one positive training example in the test set. Still, there remains a significant gap between these results and the results of the main experiments, with probability estimates coming from the LIGHTXML model.
E.4 Impact of stopping criterion and using only top- $k'$  predictions on predictive and computational performance.
In this section, we investigate the impact of stopping criterion and using precalculated top- $k'$  predictions on the predictive performance as well as computational performance.
To materialize all marginal probability estimates for datasets like WIKIPEDIA LARGE-500K and Amazon-670K, one requires an enormous amount of memory and time. Because of that, we are not able to calculate the exact results for these datasets, and instead, we used pre-select top- $k'$  labels with the highest  $\hat{\eta}_j$  as described in Section 6. In the main experiment in Section 7, we used  $k' = 100$  for smaller datasets (EURLEX-4K, WIKI-31K, AMAZONCAT-13K) and  $k' = 1000$  for larger datasets (WIKIPEDIA LARGE-500K and AMAZON-670K). Here, we additionally compare the results for  $k' = 100$  and  $k' = 1000$  for WIKIPEDIA LARGE-500K and AMAZON-670K investigating the impact of predictive and computational performance.
Table 8: Results of different inference strategies on  $@k$  measures calculated with  $k \in \{1,3,5,10\}$ , using true labels as predictions. Notation: P—precision, R—recall, F1—F1-measure. The green color indicates cells in which the strategy matches the metric. The best results are in bold and the second best are in italic.
Another factor that impacts the running time is stopping criterion  $\epsilon$ , which indicates minimal expected utility gain to continue BCA algorithm. Here we test and report results for 3 values of  $\epsilon \in \{10^{-3}, 10^{-5}, 10^{-7}\}$  on AmazonCAT-13K, WIKIPEDIALarge-500K, and Amazon-670K datasets that are the largest in terms of number of samples and labels.
The results of BCA methods with different values of  $\epsilon$  and  $k^{\prime}$  are presented in Table 9. In this table, we additionally report the number of iterations (number of passes over datasets performed by the BCA algorithm) and time in seconds. The numbers are the mean results over 5 runs with different seeds. For all values of  $\epsilon$  and  $k^{\prime}$ , the results are very similar. In the case of WIKIPEDIALARGE-500K dataset,  $k^{\prime} = 1000$  achieves slightly better results than  $k^{\prime} = 100$  in a few cases. In the case of all datasets, the smallest  $\epsilon = 10^{-7}$  is usually the best. However, even the largest tested  $\epsilon = 10^{-3}$  that is greater than the inverse of a number of labels and samples is only slightly worse at the same time, requiring a much smaller number of iterations. For all values of  $\epsilon$  and  $k^{\prime}$ , the BCA algorithm terminates in just a few iterations, finishing always in less than 20 minutes. Please note that we implemented our algorithms in Python with some parts optimized using Numba [24] - LLVM-based just-in-time (JIT) compiler for Python. Because of that, we believe that the running time can be further reduced by using a more performant programming language.
E.5 Results with mixed utilities
As we already discussed in Section 7, the optimization of macro-measures comes with the cost of a significant drop in performance on instance-wise measures, which in some cases may not be acceptable. To achieve the desired trade-off between tail and head label performance, one can optimize
Figure 2: Comparison of the baseline algorithms with the BCA inference with mixed objectives with  $k = 3$ . The green line shows the results for different interpolations between two measures.
Figure 3: Comparison of the baseline algorithms with the BCA inference with mixed objectives with  $k = 5$ . The green line shows the results for different interpolations between two measures.
a mixed utility that is a linear combination of instance-wise measures and selected macro-measures. Here, we present the results for three such mixed utilities (combinations of instance-precision with macro-precision, macro-f1-measure, and coverage):
$$
\begin{array}{l} \Psi_ {1} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) := (1 - \alpha) \Psi_ {\text {I n s t a n c e - P}} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) + \alpha \Psi_ {\text {M a c r o - P}} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) \\ = \sum_ {j = 1} ^ {m} (1 - \alpha) \psi_ {\text {I n s t a n c e - P}} (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) + \alpha \psi_ {\text {M a c r o - P}} (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) \\ = \sum_ {j = 1} ^ {m} (1 - \alpha) \frac {t _ {j}}{k} + \alpha \frac {t _ {j}}{m q _ {j}}, \\ \end{array}
$$
$$
\begin{array}{l} \varPsi_ {2} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) := (1 - \alpha) \varPsi_ {\mathrm {I n s t a n c e - P}} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) + \alpha \varPsi_ {\mathrm {M a c r o - F 1}} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) \\ = \sum_ {j = 1} ^ {m} (1 - \alpha) \psi_ {\text {I n s t a n c e - P}} (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) + \alpha \psi_ {\text {M a c r o - F l}} (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) \\ = \sum_ {j = 1} ^ {m} (1 - \alpha) \frac {t _ {j}}{k} + \alpha \frac {2 t _ {j}}{m \left(q _ {j} + p _ {j}\right)}, \\ \end{array}
$$
$$
\begin{array}{l} \varPsi_ {3} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) := (1 - \alpha) \varPsi_ {\text {I n s t a n c e - P}} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) + \alpha \varPsi_ {\text {C o v}} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) \\ = \sum_ {j = 1} ^ {m} (1 - \alpha) \psi_ {\text {I n s t a n c e - P}} (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) + \alpha \psi_ {\mathrm {C o v}} (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) \\ = \sum_ {j = 1} ^ {m} (1 - \alpha) \frac {t _ {j}}{k} + \alpha \frac {\mathbb {1} [ t _ {j} > 0 ]}{m}, \tag {75} \\ \end{array}
$$
In Figure 2 and Figure 3, we present the plots with results on two combined measures for different values of  $\alpha \in \{0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99, 0.995, 0.999\}$ . Once again, the presented results are the mean values over 5 runs with different seeds. The plots show that the instance-vs-macro curve has a nice concave shape that dominates simple baselines. In particular, we can initially improve macro-measures significantly with only a minor drop in instance-measures, and only if we want to optimize even more strongly for macro-measures, we get larger drops in instance-wise measures. A particularly notable feature of the plug-in approach is that the curves in the figure are cheap to produce since there is no requirement for expensive re-training of the entire architecture, so one can easily select an optimal interpolation constant according to some criteria, such as a maximum decrease of instance-wise performance.
E.6 Hardware
The LIGHTXML model was trained on a workstation with a single Nvidia Tesla V100 GPU with 32 GB of memory and 64 GB of RAM. All the inference strategies were then run on the workstation with 64 GB of RAM. However, to run them, one requires only 16 GB of memory.
F Efficient inference with Probabilistic Label Trees
In this section, we describe probabilistic labels trees (PLTs) [16], which allow for efficient retrieval of only the top- $k$  labels with the highest conditional probabilities. Then, we introduce a weighted version and show how to modify it to achieve efficient and exact optimal prediction for any  $g_{j}(\eta_{j}(\pmb{x}))$  that is monotonous in  $\eta_j(\pmb{x})$ . Thus, if the LPE  $\pmb{\eta}$  is modeled by a PLT, this provides an alternative to the sparse inference introduced in Appendix D that also scales to XMLC problems.
F.1 Probabilistic labels trees
We denote a tree by  $\mathcal{T}$ , the set of all its nodes by  $\nu_{\mathcal{T}}$ , the root node by  $r_{\mathcal{T}}$ , and the set of its leaves by  $\mathcal{L}_{\mathcal{T}}$ . The leaf  $l_j \in \mathcal{L}_{\mathcal{T}}$  corresponds to the label  $j \in [m]$ . The parent node of  $\nu$  is denoted by  $\mathrm{pa}(\nu)$  and the set of child nodes by  $\mathrm{ch}(\nu)$ . The set of leaves of a (sub)tree rooted in node  $\nu$  is denoted by  $\mathcal{L}_{\nu}$  and path from node  $\nu$  to the root by  $\mathrm{path}(\nu)$ .
A PLT uses a tree  $\mathcal{T}$  to factorize conditional probabilities of labels,  $\eta_j(\pmb{x}) = \mathbb{P}[y_j = 1 \mid \pmb{x}]$ ,  $j \in [m]$ , by using the chain rule. Let us define an event that  $\mathcal{L}_{\mathbf{v}}$  contains at least one relevant label in  $\pmb{y}$ , denoting  $z_{\mathbf{v}} := \mathbb{1}[\exists j : \mathsf{l}_j \in \mathcal{L}_{\mathbf{v}} \wedge y_j = 1]$ . Now for every node  $\mathbf{v} \in \mathcal{V}_{\mathcal{T}}$ , the conditional probability of containing at least one relevant label is given by:
$$
\eta_ {\mathrm {v}} (\boldsymbol {x}) := \mathrm {P} \left[ z _ {\mathrm {v}} = 1 \mid \boldsymbol {x} \right] = \prod_ {\mathrm {v} ^ {\prime} \in \operatorname {p a t h} (\mathrm {v})} \eta (\boldsymbol {x}, \mathrm {v} ^ {\prime}), \tag {76}
$$
where  $\eta(\boldsymbol{x},\mathbf{v}) \coloneqq \mathbb{P}\left[z_{\mathbf{v}} = 1 \mid z_{\mathrm{pa}(\mathbf{v})} = 1, \boldsymbol{x}\right]$  for non-root nodes, and  $\eta(\boldsymbol{x},\mathbf{v}) \coloneqq \mathbb{P}[z_{\mathbf{v}} = 1 \mid \boldsymbol{x}]$  for the root. Notice that (76) can also be stated recursively as
$$
\eta_ {\mathrm {v}} (\boldsymbol {x}) = \eta (\boldsymbol {x}, \mathrm {v}) \cdot \eta_ {\mathrm {p a} (\mathrm {v})} (\boldsymbol {x}), \tag {77}
$$
and that for leaf nodes we get the conditional probabilities of labels
$$
\eta_ {l _ {j}} (\boldsymbol {x}) = \eta_ {j} (\boldsymbol {x}), \quad \text {f o r} l _ {j} \in \mathcal {L} _ {\mathcal {T}}. \tag {78}
$$
To obtain a PLT, it suffices, for a given  $\mathcal{T}$ , to train probabilistic classifiers estimating  $\eta(\boldsymbol{x},\nu)$  for all  $\nu \in \mathcal{V}_{\mathcal{T}}$ . Analogously to the main paper, we denote estimates of  $\eta$  by  $\hat{\eta}$ .
F.2 Weighted PLTs
[46] introduced an  $A^*$ -search based algorithm for efficiently finding the  $k$  leaves with the highest gain  $g(\mathsf{I}_j, \pmb{x}) = w_j \eta_j(\pmb{x})$ , where  $w_j \in [0, \infty)$  is the weight given to label  $j$ . The outline of the search method presented below is an adapted description from [46].
For the gain function  $g(\mathsf{l}_j, \boldsymbol{x}) = w_j \hat{\eta}_j(\boldsymbol{x})$ , the procedure uses a cost function  $c(\mathsf{l}_j, \boldsymbol{x})$  for each path from the root to a leaf. Notice that the following holds:
$$
g \left(\mathrm {l} _ {j}, \boldsymbol {x}\right) = w _ {j} \hat {\eta} _ {j} (\boldsymbol {x}) = \exp \left(- \left(- \log w _ {j} - \sum \log \hat {\eta} (\boldsymbol {x}, \mathrm {v})\right)\right). \tag {79}
$$
Because exp is monotonous, we can define the following cost function for a selected label  $j$ , which the algorithm will aim to minimize:
$$
c \left(\mathrm {l} _ {j}, \boldsymbol {x}\right) := - \log w _ {j} - \sum_ {\mathrm {v} \in \operatorname {p a t h} \left(\mathrm {l} _ {j}\right)} \log \hat {\eta} (\boldsymbol {x}, \mathrm {v}). \tag {80}
$$
We can then guide the  $A^*$ -search with the function  $\overline{c}(\mathbf{v}, \pmb{x}) = p(\mathbf{v}, \pmb{x}) + h(\mathbf{v}, \pmb{x})$ , estimating the value of the optimal path in the subtree of node  $\mathbf{v}$ , where
$$
p (\mathrm {v}, \boldsymbol {x}) = - \sum_ {\mathrm {v} ^ {\prime} \in \text {p a t h} (\mathrm {v})} \log \hat {\eta} (\boldsymbol {x}, \mathrm {v} ^ {\prime}) \tag {81}
$$
is the cost of reaching tree node v from the root, and
$$
h (\mathbf {v}, \boldsymbol {x}) = - \log \max  _ {j \in \mathcal {L} _ {\mathrm {v}}} w _ {j} \tag {82}
$$
is a heuristic function estimating the cost of reaching the best leaf from node  $\mathbf{v}$ . The  $A^{*}$ -search in our procedure evaluates nodes in ascending order of their estimated cost values  $\bar{c}(\mathrm{l}_j, \boldsymbol{x})$ .
This approach has been proven by [46] to guarantee that  $A^*$ -search finds the optimal solution—top- $k$  labels with the highest  $c(\mathfrak{l}_j, \pmb{x})$  and thereby top- $k$  labels with the highest  $w_j \eta_j(\pmb{x})$ —in the optimally efficient way, i.e., there is no other algorithm used with this heuristic that expands fewer nodes [38].
Algorithm 9 Select top-  $k$  labels with highest gain using PLTs  $(\mathcal{T},\hat{\eta},\boldsymbol {x},k,g_{j}(\cdot))$
1:  $\hat{\pmb{y}}\gets 0^{m}$  ▷ Initialize prediction  $\hat{\pmb{y}}$  vector to all zeros   
2:  $\mathcal{Q}\gets \emptyset$  ▷ Initialize priority queue  $\mathcal{Q}$  , ordered descending by  $\overline{g} (\mathbf{v},\mathbf{x})$    
3:  $p(\mathsf{r}_{\mathcal{T}},\mathbf{x})\leftarrow \hat{\eta} (\mathbf{x},\mathsf{r}_{\mathcal{T}})$  ▷ Calculate estimated conditional probability  $p(\mathsf{r}_{\mathcal{T}},\mathbf{x})$  for the tree root   
4:  $\overline{g} (\mathsf{r}_{\mathcal{T}},\mathbf{x})\leftarrow \max_{j\in \mathcal{L}_{\mathsf{r}_{\mathcal{T}}}}(g_j(p(\mathsf{r}_{\mathcal{T}},\mathbf{x})))$  ▷ Calculate estimated gain  $\overline{g} (\mathsf{r}_{\mathcal{T}},\mathbf{x})$  for the tree root   
5:  $\mathcal{Q}.add((\mathsf{r}_{\mathcal{T}},p(\mathsf{r}_{\mathcal{T}},\mathbf{x}),\overline{g} (\mathsf{r}_{\mathcal{T}},\mathbf{x})))$  ▷ Add the tree root with estimates  $p(\mathsf{r}_{\mathcal{T}},\mathbf{x})$  and  $\overline{g} (\mathsf{r}_{\mathcal{T}},\mathbf{x})$  to the queue   
6: while  $\| \hat{\pmb{y}}\| _1 <   k$  do ▷ While the number of predicted labels is less than  $k$    
7:  $(\mathrm{v},p(\mathrm{v},\pmb {x}),\_)\gets \mathcal{Q}.pop()$  ▷ Pop the element with the lowest cost from the queue   
8: if v is a leaf then  $\hat{y}_v\gets 1$  ▷ If the node is a leaf, set the corresponding label in the prediction vector   
9: else for  $\nu^{\prime}\in \operatorname {ch}(\nu)$  do ▷ If the node is an internal node, for all child nodes   
10:  $p(\mathrm{v}',\mathrm{x})\gets p(\mathrm{v},\mathrm{x})\hat{\eta} (\mathrm{x},\mathrm{v}')$  ▷ Calculate  $p(\mathrm{v}',\mathrm{x})$    
11:  $\overline{g} (\mathrm{v}',\mathrm{x})\leftarrow \max_{j\in \mathcal{L}_{\mathrm{v}}}(g_j(p(\mathrm{v}',\mathrm{x})))$  ▷ Calculate estimate  $\overline{g} (\mathrm{v}',\mathrm{x})$    
12: Q.add((v',p(v',x),g(v',x))) ▷ Add v', and its estimates  $p(\mathrm{r}_{\mathcal{T}},\mathbf{x})$  and  $\overline{g} (\mathsf{r}_{\mathcal{T}},\mathbf{x})$  to the queue   
13: return  $\hat{\pmb{y}}$  ▷ Return the prediction vector
F.3 PLTs for monotonous gain functions
Notice that in Weighted PLTs instead of minimizing cost, one can directly optimize gain by using a tree-search procedure that evaluates nodes in descending order of their estimated gain  $\overline{g}(\mathbf{v},\boldsymbol{x}) = p(\mathbf{v},\boldsymbol{x})\cdot h(\mathbf{v},\boldsymbol{x})$ , where  $p(\mathbf{v},\boldsymbol{x}) = \sum_{\mathbf{v}'\in \mathrm{path}(\mathbf{v})}\hat{\eta}(\boldsymbol{x},\mathbf{v}')$  and  $h(\mathbf{v},\boldsymbol{x}) = \max_{j\in \mathcal{L}_{\mathbf{v}}}w_{j}$ . To generalize this formulation and apply it to the proposed algorithms, we need to be able to find the top- $k$  with any gain function  $g_{j}(\eta_{j}(\boldsymbol{x}))$  that is monotonous with  $\eta_{j}(\boldsymbol{x})$ .
We can guide the tree search using the estimated gain function  $\bar{g} (\mathbf{v},\pmb {x}) = h(\mathbf{v},p(\mathbf{v},\pmb {x}))$  , where
$$
p (\mathrm {v}, \boldsymbol {x}) = \prod_ {\mathrm {v} ^ {\prime} \in \operatorname {p a t h} (\mathrm {v})} \hat {\eta} (\boldsymbol {x}, \mathrm {v} ^ {\prime}), \tag {83}
$$
the same as in the case of Weighted PLT, it simply corresponds to the conditional probability of finding at least one positive label in the subtree of node  $\nu$ , and the heuristic part is:
$$
h (\mathbf {v}, \boldsymbol {x}) = \max  _ {j \in \mathcal {L} _ {\mathbf {v}}} \left(g _ {j} (p (\mathbf {v}, \boldsymbol {x}))\right). \tag {84}
$$
As in the case of Weighted PLT, we would like to guarantee that this search procedure finds the optimal solution—the top- $k$  labels with the highest  $g(\mathfrak{l}_j, \boldsymbol{x})$ . To do that, we need to ensure that  $\overline{g}(\mathbf{v}, \boldsymbol{x})$  is admissible, i.e., in case of maximization, it never underestimates the gain from reaching a leaf node [38]. We also would like  $\overline{g}(\mathbf{v}, \boldsymbol{x})$  to be consistent, making the proposed tree search procedure optimally efficient, i.e., there is no other algorithm used with the same  $\overline{g}(\mathbf{v}, \boldsymbol{x})$  that expands fewer nodes [38]. Algorithm 9 outlines this procedure for finding the top- $k$  labels with highest values of  $g_j(\eta_j(\boldsymbol{x}))$  that is monotonous with  $\eta_j(\boldsymbol{x})$ , and that can replace the select top- $\mathrm{k}(g)$  operation in the proposed BCA and Greedy algorithms. Unfortunately, not all gain functions  $g_j(\eta_j(\boldsymbol{x}))$  can be calculated efficiently. To calculate the semi-empirical quantity  $\tilde{p}$  exactly, one needs to know all the  $\eta_j(\boldsymbol{x})$ . To obtain them, the whole tree need to be evaluated, which prevents this method from being efficient. Because of that, we use this technique to get exact predictions for the measures that are defined solely on  $\tilde{t}$  and  $\tilde{q}$ , like macro-precision and coverage.
F.4 Experimental comparison of inference with PLTs
In this experiment, we evaluate the inference using probabilistic label trees that can efficiently perform the exact select top- $k(g)$  operation. We compare inference times of these variants of the methods that require only one pass over the dataset: TOP-K, PS-K, POW-K, LOG-K, MACRO-  $\mathbf{P}_{\mathrm{GREEN}}$ , MACRO- $\mathbf{R}_{\mathrm{PRIOR}}$ , and  $\mathrm{COV}_{\mathrm{GREEN}}$ , for  $k = \{3,5\}$ . As a baseline, we use the same PLT to obtain the  $k' = 100$  and  $k' = 1000$  highest  $\boldsymbol{\eta}(\boldsymbol{x}_i)$ , and then run the sparse algorithms on top of these predictions. We use a modified implementation of PS-PLT [46, 17], trained with default settings.
We present the results in Table 10. As in the case of the main experiment, we again observe that the specialized inference strategies are indeed the best on the measure they aim to optimize. Compared to the cost of obtaining marginals for  $k' = 100$  and  $k' = 1000$  labels, which can be later used with one of the proposed inference algorithms (the cost of running this step is not included in the table), the greedy methods combined with PLT-based search consistently achieve speed-up when compared to
Table 10: Results of different inference strategies on  $@k$  measures with  $k = \{3,5\}$  using PLT model. Notation: P—precision, R—recall, F1—F1-measure, Cov—Coverage, T—time in seconds,  $k' = 100/1000$ —speed-up relative to the time required to obtain top-100/1000  $\eta(x)$  for all the instances in  $X$ . The green color indicates cells in which the strategy matches the metric. The best results are in bold and the second best are in italic.