SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xingjian Bai  
Department of Computer Science  
University of Oxford, UK  
xingjian.bai@sjc.ox.ac.uk
Christian Coester  
Department of Computer Science  
University of Oxford, UK  
christian.coester@cs.ox.ac.uk
We explore the fundamental problem of sorting through the lens of learning-augmented algorithms, where algorithms can leverage possibly erroneous predictions to improve their efficiency. We consider two different settings: In the first setting, each item is provided a prediction of its position in the sorted list. In the second setting, we assume there is a "quick-and-dirty" way of comparing items, in addition to slow-and-exact comparisons. For both settings, we design new and simple algorithms using only  $O(\sum_{i} \log \eta_{i})$  exact comparisons, where  $\eta_{i}$  is a suitably defined prediction error for the  $i$ th element. In particular, as the quality of predictions deteriorates, the number of comparisons degrades smoothly from  $O(n)$  to  $O(n \log n)$ . We prove that this comparison complexity is theoretically optimal with respect to the examined error measures. An experimental evaluation against existing adaptive and non-adaptive sorting algorithms demonstrates the potential of applying learning-augmented algorithms in sorting tasks.
Sorting is one of the most basic algorithmic problems, commonly featured as one of the initial topics in computer science education, and with a vast array of applications spanning various domains. In recent years, the emerging field of algorithms with predictions (Lykouris and Vassilvitskii, 2021, Mitzenmacher and Vassilvitskii, 2022), also known as learning-augmented algorithms, has opened up new possibilities for algorithmic improvement, where algorithms aim to leverage predictions (possibly generated through machine learning, or otherwise) to improve their performance. However, the classical sorting problem with predictions, along with the discussion of different types of predictors for sorting, appears to have been largely overlooked by this recent movement. This paper explores the problem of sorting through the lens of algorithms with predictions in two settings, aiming to overcome the classical  $\Omega(n \log n)$  barrier with the aid of various types of predictors.
The first setting involves each item having a prediction of its position in the sorted list. This type of predictor is commonly found in real-world scenarios. For instance, empirical estimations of element distribution can generate positional predictions. Another example is that a fixed set of items has their ranking evolve over time, with minor changes at each timestep. Here, an outdated ranking can serve as a natural prediction for the current ranking. As re-evaluating the true relation between items can be costly, the provided positional predictions offer useful information. The positional prediction setting is closely related to adaptive sorting of inputs with existing presortedness (Estivill-Castro and Wood, 1992), but we consider different measures of error on the predictor, resulting in algorithms with a more fine-grained complexity.
In the second setting, a "dirty" comparison function is provided to assist sorting. In biological experiments, for instance, some "indicating factors" might be used to approximately compare two molecules or drugs. Despite potential errors due to the oversight of minor factors, these comparisons can still offer preliminary insights into the properties of the subjects. More broadly, in experimental science, researchers often carry out costly experiments to compare subject behaviours. By utilizing a
proficient sorting algorithm that capitalizes on dirty comparisons, the need for costly experiments can be reduced and substituted by less expensive, albeit noisier, experiments.
We propose sorting algorithms to leverage either type of predictor. In the positional prediction setting, we design two deterministic algorithms with different complexity bounds, while in the dirty comparisons setting, we develop a randomized algorithm. In all settings, we provide bounds of the form  $O(\sum_{i=1}^{n} \log(\eta_i + 2))$  on the number of exact comparisons, for different notions of element-wise prediction errors  $\eta_i \in [0, n]$ . In particular, all three proposed algorithms only require  $O(n)$  exact comparisons if predictions are accurate (consistency), never use more than  $O(n \log n)$  comparison regardless of prediction quality (robustness), and their performance degrades slowly as a function of prediction error (smoothness). Moreover, we show that all algorithms have optimal comparison complexity with respect to the error measures examined.
Finally, through experiments on both synthetic and real-world data, we evaluate the proposed algorithms against existing (adaptive and non-adaptive) algorithms. Results demonstrate their superiority over the baselines in multiple scenarios.
Let  $A = \langle a_1, \ldots, a_n \rangle$  be an array of  $n$  items, equipped with a strict linear order  $<$ . Let  $p \colon [n] \to [n]$  be the permutation that maps each index  $i$  to the position of  $a_i$  in the sorted list; that is,  $a_{p^{-1}(1)} < a_{p^{-1}(2)} < \dots < a_{p^{-1}(n)}$ . We consider two settings of sorting with predictions.
Sorting with Positional Predictions. In sorting with positional predictions, the algorithm receives for each item  $a_i$  a prediction  $\hat{p}(i)$  of its position  $p(i)$  in the sorted list. We allow  $\hat{p}$  to be any function  $[n] \to [n]$ , which need not be a permutation (i.e., it is possible that  $\hat{p}(i) = \hat{p}(j)$  for some  $i \neq j$ ).
Positional predictions can be generated by models that roughly sort the items, e.g. focusing on major factors while neglecting minor ones. Or they can stem from past item rankings, while the properties of the items evolve over time. In such cases, the objective is to obtain the latest ranking of items.
The error of a positional prediction can be naturally quantified by the displacement of each element's prediction; that is, the absolute difference of the predicted ranking and the true ranking. We define the displacement error of item  $a_{i}$  as
$$
\eta_ {i} ^ {\Delta} := | \hat {p} (i) - p (i) |.
$$
The following notion of one-sided error provides an alternative perspective to evaluate the complexity of algorithms with positional predictions. We denote the left-error and right-error of item  $a_{i}$  as
$$
\eta_ {i} ^ {l} := | \{j \in [ n ]: \hat {p} (j) \leq \hat {p} (i) \wedge p (j) > p (i) \} |
$$
$$
\eta_ {i} ^ {r} := \left| \left\{j \in [ n ] \colon \hat {p} (j) \geq \hat {p} (i) \wedge p (j) <   p (i) \right\} \right|.
$$
In certain contexts, it may be impossible to obtain a predictor with small displacement error, but possible to obtain one with a small one-sided error. By developing algorithms for this setting, we expand the space of problems where sorting algorithms with predictions can be applied.
Sorting with Dirty and Clean Comparisons. The other setting we consider involves a predictor that estimates which of two elements is larger without conducting a proper comparison, providing a faster but possibly inaccurate result. This type of predictor is applicable in scenarios where exact comparisons are costly but a rough estimate of the comparison outcome can be obtained more easily.
Formally, in sorting with dirty comparisons, the algorithm has access to a complete, asymmetric relation  $\preceq$  on the items, while still also having access to the exact comparisons  $<$ . That is, for any two distinct items  $a_i$  and  $a_j$ , either  $a_i \preceq a_j$  or  $a_j \preceq a_i$ . We think of  $\preceq$  as an unreliable, but much faster to evaluate prediction of  $<$ . We also refer to the (fast) comparisons according to  $\preceq$  as dirty whereas the (slow) comparisons according to  $<$  are clean. We emphasize that the relation  $\preceq$  need not be transitive, so it is not necessarily a linear order. Instead,  $\preceq$  induces a tournament graph on the items of  $A$ , containing a directed edge  $(a_i, a_j)$  if and only if  $a_i \preceq a_j$ , and in many applications, we expect this graph to have cycles.
We denote by  $\eta_{i}$  the number of incorrect dirty comparisons involving  $a_{i}$ , that is,
$$
\eta_ {i} := \left| \left\{j \in [ n ]: \left(a _ {i} <   a _ {j}\right) \neq \left(a _ {i} \widehat {<  } a _ {j}\right) \right\} \right|.
$$
1.2 Main Results
Our main result for the dirty comparison setting is given by the following theorem:
Theorem 1.1. Augmented with dirty comparisons, there is a randomized algorithm that sorts an array within  $O(n \log n)$  running time,  $O(n \log n)$  queries to dirty comparisons, and  $O\left(\sum_{i=1}^{n} \log (\eta_i + 2)\right)$  clean comparisons in expectation.
By the classical lower bound, the total number of dirty  $+$  clean comparisons must be at least  $\Omega (n\log n)$  regardless of prediction quality, but for sufficiently good predictions the theorem allows us to replace most clean comparisons with dirty ones. The case where dirty comparisons are probabilistic is discussed in Section B.1.
The following theorem generalizes the previous one to the case that there are  $k$  different dirty comparison operators:
Theorem 1.2. Augmented with  $k \leq 2^{O(n / \log n)}$  dirty comparison predictors, where the error of the  $p$ th predictor is denoted by  $\eta^p$ , there is a randomized algorithm that sorts an array with at most  $O(\min_p \sum_{i=1}^n \log (\eta_i^p + 2))$  clean comparisons.
In other words, the number of clean comparisons is as good as if we knew in advance which of the  $k$  predictors is best. The bound on  $k$  is almost tight, since already  $k \geq 2^{n\log n}$  would mean there could be one predictor for each of the  $n!$  possible sorting outcomes, which would render them useless. The proof of Theorem 1.2 is given in Appendix B.2.
The next two theorems capture our algorithms for the positional prediction setting:
Theorem 1.3. Augmented with a positional predictor, there is a deterministic algorithm that sorts an array within  $O\left(\sum_{i=1}^{n} \log \left( \eta_i^\Delta + 2 \right)\right)$  running time and comparisons.
Theorem 1.4. Augmented with a positional predictor, there is a deterministic algorithm that sorts an array within  $O\left(\sum_{i=1}^{n} \log \left( \min \left\{ \eta_i^l, \eta_i^r \right\} + 2 \right) \right)$  comparisons.
We remark that there exist instances where the bound of Theorem 1.3 is stronger than that of Theorem 1.4 and vice versa.<sup>1</sup>
The following theorem, proved in Appendix E, shows tightness of the aforementioned upper bounds.
Theorem 1.5. Augmented with dirty comparisons, no sorting algorithm uses  $o\left(\sum_{i=1}^{n} \log \left( \eta_i + 2 \right)\right)$  clean comparisons. Augmented with a positional predictor, no sorting algorithm uses  $o\left(\sum_{i=1}^{n} \log \left( \eta_i^{\Delta} + 2 \right)\right)$  or  $o\left(\sum_{i=1}^{n} \log \left( \min \left\{ \eta_i^l, \eta_i^r \right\} + 2 \right)\right)$  comparisons.
Bounds in Terms of Global Error. One may wonder how the above bounds translate to a global error measure such as the number of item pairs where the larger one is incorrectly predicted to be no larger than the smaller one. Writing  $D$  for this error measure, in the dirty comparison setting we simply have  $D = \frac{1}{2}\sum_{i}\eta_{i}$ , and in the positional prediction setting  $D\geq \frac{1}{2}\sum_{i}\eta_{i}^{\Delta}$  by (Rohatgi, 2020, Lemma 11). Thus, concavity of logarithm and Jensen's inequality yield an upper bound of  $O\left(n\log \left(\frac{D}{n} +2\right)\right)$  for both settings. This bound is tight² as a function of  $D$  and corresponds to the optimal complexity of adaptive sorting as a function of the number of inversions (Mannila, 1985).
However, our guarantees in terms of element-wise error are strictly stronger whenever Jensen's inequality is not tight, i.e., when the  $\eta_{i}$  are non-uniform. Furthermore, it is reasonable to expect predictors to exhibit varying levels of error for different items, especially when the error originates from element-wise noise.
Algorithms with Predictions. Our study aligns with the broader field of learning-augmented algorithms, also known as algorithms with predictions. The majority of research has focused on
<sup>1</sup>If predictions are correct except that the positions of the  $\sqrt{n}$  smallest and  $\sqrt{n}$  largest items are swapped, then  $\sum_{i}\log (\eta_{i}^{\Delta} + 2) = \Theta (n)$ , but  $\sum_{i}\log (\min \{\eta_{i}^{l},\eta_{i}^{r}\} +2)) = \Theta (n\log n)$ . Conversely, if  $\hat{p} (i) = p(i) + \sqrt{n} \mod n$ , then  $\sum_{i}\log (\eta_{i}^{\Delta} + 2) = \Theta (n\log n)$ , but  $\sum_{i}\log (\min \{\eta_{i}^{l},\eta_{i}^{r}\} +2)) = \Theta (n)$ .
Indeed, our proof of Theorem 1.5 constructs a family of instances where each  $\eta_{i}$  is bounded by the same quantity, so Jensen's inequality is tight for these instances.
classical online problems such caching (Lykouris and Vassilvitskii, 2021, Rohatgi, 2020, Wei, 2020, Bansal et al., 2022), rent-or-buy problems (Purohit et al., 2018, Gollapudi and Panigrahi, 2019, Angelopoulos et al., 2020, Wang et al., 2020, Antoniadis et al., 2021), scheduling (Purohit et al., 2018, Lattanzi et al., 2020, Mitzenmacher, 2020, Azar et al., 2021, 2022, Lindermayr and Megow, 2022) and many others. In comparison, research on learning-augmented algorithms to improve running time for offline problems is relatively sparser, but examples include matching (Dinitz et al., 2021, Sakaue and Oki, 2022), clustering (Ergun et al., 2022), and graph algorithms (Chen et al., 2022, Davies et al., 2023). Motivated by (Kraska et al., 2018), (Lykouris and Vassilvitskii, 2021) describes a simple method to speed up binary search with predictions, which has been inspirational for our work. Learning-augmented algorithms have also been applied to data structures such as binary search trees (Lin et al., 2022, Cao et al., 2023), and empirical works demonstrate the benefits of ML-augmentation for index structures (Kraska et al., 2018) and database systems (Kraska et al., 2019). There has also been increasing interest in settings where algorithms have access to multiple predictors (Gollapudi and Panigrahi, 2019, Wang et al., 2020, Bhaskara et al., 2020, Almanza et al., 2021, Emek et al., 2021, Dinitz et al., 2022, Anand et al., 2022, Antoniadis et al., 2023).
Related to sorting, Lu et al. (2021) studied learning-augmented generalized sorting, a variant of sorting where some comparisons are allowed while others are forbidden. The predictions they consider are similar to our dirty comparisons. They proposed two algorithms with comparison complexities  $O(n \log n + w)$  and  $O(nw)$ , where  $w$  is the total number of incorrect dirty comparisons. In the classical (non-generalized) setting with all comparisons allowed, only the second bound theoretically improves upon  $O(n \log n)$ , but the dependence on the error is exponentially worse than for our algorithms; even with a single incorrect dirty comparison per item, the  $O(nw)$  bound becomes  $O(n^2)$ , whereas ours is  $O(n)$ . A recent work of Erlebach et al. (2023) studies sorting under explainable uncertainty with predictions, where initially only an interval around the value of each item is known, the exact values can be queried, a prediction of these values is given, and the goal is to minimize the number of queries needed to sort the list.
Deep Learning-Based Sorting. The thriving development of deep learning has inspired research into new sorting paradigms. A DeepMind study (Mankowitz et al., 2023) recast sorting as a single-player game, training agents to perform effectively, thus uncovering faster sorting routines for short sequences. Kristo et al. (2020) proposed a sorting algorithm that uses a learning component to improve the empirical performance of sorting numerical values. Their algorithm tries to approximate the empirical CDF of the input by applying ML techniques to a small subset of the input. This setting diverges from ours: If inputs are non-numeric (and no monotonous mapping to numbers is known), then one has to rely on a comparison function, and the approach of Kristo et al. (2020) would not be well-defined, whereas our algorithms can sort arbitrary data types.
Conversely, the input in (Kristo et al., 2020) is solely the item list, without any additional predictions. Note that predictions (or other assumptions) are necessary to beat the entropic  $\Omega(n\log n)$  lower bound. $^3$
Noisy Sorting. Noisy sorting contemplates scenarios where comparison results may be incorrect. This model is useful to simulate potential faults in large systems. Two noisy sorting settings have primarily been considered: In independent noisy setting, each query's result is independently flipped with probability  $p \in (0, \frac{1}{2})$ . Recently, Gu and Xu (2023) provided optimal bounds on the number of queries to sort  $n$  elements with high probability. Recurrent noisy setting (Braverman and Mossel, 2008) further assumes any repeated comparisons will yield consistent results. Geissmann et al. (2019) present an optimal algorithm that guarantees  $O(n \log n)$  time,  $O(\log n)$  maximum dislocation, and  $O(n)$  total dislocation with high probability. While the recurrent noisy setting is closely related to dirty comparisons, studies in that field focus primarily on approximate sorting; to the best of our knowledge, no exact sorting algorithms that use both dirty and clean comparisons have been studied.
Adaptive Sorting. Adaptive sorting algorithms leverage various types of existing order in the input, achieving better complexity for partially sorted data. Examples include TimSort (Peters, 2002), which is the standard sorting algorithm in a variety of programming languages, Cook-Kim division
<sup>3</sup>The theoretical guarantee of their algorithm is  $O(n^{2})$ , although they observe much better empirical performance. Indeed, this makes sense for numerical inputs drawn from a sufficiently nice distribution, since then one can extrapolate from a small part of the input to the rest.
(Cook and Kim, 1980), and Powersort (Munro and Wild, 2018), which recently replaced TimSort in Python's standard library. We refer to the survey of Estivill-Castro and Wood (1992) for a broader overview. The concept of pre-sortedness is closely related to positional predictions; however, without the motivation from predictors, the complexity bound on adaptive sorting algorithms was often considered under error measures on the entire array, instead of on each element. In contrast, our error measure is element-wise, allowing algorithms with stronger complexity bounds.
2 Sorting with Dirty Comparisons
Given a dirty predictor  $\widehat{<}$ , our goal is to sort  $A$  with the least possible number of clean comparisons. Note that, if  $\widehat{<}$  is accurate,  $A$  can be sorted using only  $O(n)$  clean comparisons and  $O(n\log n)$  dirty comparisons. This could be achieved, for example, by performing Merge Sort with dirty comparisons and then validating the result through clean comparisons between adjacent elements. This observation motivates us to consider that not all  $O(n^2)$  dirty comparisons are necessary, and we should devise an algorithm minimizing the number of both clean and dirty comparisons.
We propose a randomized algorithm that sorts  $A$  with expected  $O(\sum \log (2 + \eta_i))$  clean comparisons, expected  $O(n\log n)$  dirty comparisons, and expected  $O(n\log n)$  running time. The key idea consists of three parts: 1) Sequentially insert each element of  $A$  into a binary search tree, following random order; 2) Guide each insertion primarily with dirty comparisons, while verifying the correctness of it using a minimal number of clean comparisons. 3) Correct the mistake induced by the dirty insertion, ensuring that the clean comparisons needed for correction is  $O(\log \eta_{i})$  in expectation.
We describe the algorithm in Section 2.1 and prove its performance guarantees in Section 2.2. In Appendix B we discuss extensions of the dirty comparison setting.
2.1 Algorithm
We now describe the sorting algorithm with dirty comparisons in detail (Algorithm 1). We initialize  $B$  as an empty binary search tree (BST). For any vertex  $v$ , we denote by  $\text{left}(v)$  and  $\text{right}(v)$  its left and right children, and by  $\text{root}(B)$  the root of  $B$ . Slightly abusing notation, we write  $v$  also for the item stored at vertex  $v$ . If any of these vertices is missing, the respective variable has value NIL.
Algorithm 1: Sorting with dirty and clean comparisons
Input:  $A = \langle a_1,\dots ,a_n\rangle$  , dirty comparator  $\widehat{z}$  , clean comparator  $<  _{\cdot}$    
1  $B\gets$  empty binary tree   
2 for  $i\in [n]$  in uniformly random order do   
3  $(L_{1},C_{1},R_{1})\leftarrow (-\infty ,\mathrm{root}(B),\infty)$    
4  $t\gets 1$    
5 while  $C_t\neq \mathrm{NIL}$  do ▷ Dirty search   
6 if  $a_i\lesssim C_t$  then  $(L_{t + 1},C_{t + 1},R_{t + 1})\leftarrow (L_t,\mathrm{left}(C_t),C_t)$    
7 else  $(L_{t + 1},C_{t + 1},R_{t + 1})\leftarrow (C_t,\mathrm{right}(C_t),R_t)$    
8  $t\gets t + 1$    
9  $C\gets C_{t^*}$  , where  $t^{*}\leq t$  is maximal s.t.  $L_{t^{*}} <   a_{i} <   R_{t^{*}}$  ▷ Verification   
10 while  $C\neq \mathrm{NIL}$  do ▷ Clean search   
11 if  $a_i <   C$  then  $C\gets \mathrm{left}(C)$    
12 else  $C\gets \mathrm{right}(C)$    
13 Insert  $a_i$  at  $C$    
14 return inorder traversal of  $B$
Within each iteration of the for-loop starting in line 2, we select one item  $a_i$  of  $A$  uniformly randomly from the items that have not been processed yet. Then, we insert this item  $a_i$  into  $B$ , while maintaining the invariant that  $B$  remains a BST with respect to clean comparisons  $<$ .
Inserting item  $a_{i}$  into  $B$  requires three phases, as illustrated in Figure 2.1. The first phase involves performing a search for the insertion position using dirty comparisons  $\widehat{<}$ , keeping track of the search path. Here, we denote by  $C_t$  the  $t$ th vertex on this path, and by  $L_{t}$  and  $R_{t}$  the lower and upper
Figure 2.1: The insertion process in dirty comparison sorting.
dirty search
failed verification
successful verification
bounds on items that can be inserted in the subtree rooted at  $C_t$  without violating the BST property with respect to  $<$ . Correctness of the choice of  $L_t$  and  $R_t$  follows from the fact that  $B$  was a BST with respect to  $<$  before the current insertion. This dirty procedure stops when the search path reaches a NIL-leaf, regarded as the predicted position for  $a_i$ 's insertion. However, since we used dirty comparisons to trace the path,  $a_i$  might violate one of the boundary conditions  $L_t < a_i$  or  $a_i < R_t$  at some recursion step  $t$ . We call a recursion step  $t$  valid for  $a_i$  if  $L_t < a_i < R_t$ . Then, we enter the verification phase in line 9. We traverse the dirty search path in reverse order to locate the last valid step  $t^*$ . A naive method to do this (which is sufficient for our asymptotic guarantees) is to repeatedly decrease  $t$  by 1 until  $t$  is valid; we discuss an alternate, more efficient method in Remark A.6, which yields a better constant factor.
The final phase involves performing a clean search starting from  $C_{t^*}$  to determine the correct insertion position for  $a_i$ . After inserting  $a_i$  into that position,  $B$  remains a BST with respect to  $<$ . Once all items of  $A$  are inserted into  $B$ , we can obtain the sorted order through the inorder traversal of  $B$ .
2.2 Complexity Analysis
The idea is to show that the dirty search path has expected depth  $O(\log n)$ , whereas the verification path and the clean search path only have expected depth  $O(\log \eta_i)$ . Therefore, the algorithm only needs  $O(\log n)$  dirty and  $O(\log \eta_i)$  clean comparisons for inserting each  $a_i$ ; Theorem 1.1 is established consequently. The full proof is given in Appendix A.
3 Sorting with Positional Predictions