Title: Sorting with Predictions
Venue: NeurIPS 2023
Accepted: yes
Keywords: sorting

learning-augmented algorithms

algorithms with predictions

adaptive sorting
Abstract: We explore the fundamental problem of sorting through the lens of learning-augmented algorithms, where algorithms can leverage possibly erroneous predictions to improve their efficiency. We consider two different settings: In the first setting, each item is provided a prediction of its position in the sorted list. In the second setting, we assume there is a ``quick-and-dirty'' way of comparing items, in addition to slow-and-exact comparisons. For both settings, we design new and simple algorithms using only $O(\sum_i \log \eta_i)$ exact comparisons, where $\eta_i$ is a suitably defined prediction error for the $i$th element. In particular, as the quality of predictions deteriorates, the number of comparisons degrades smoothly from $O(n)$ to $O(n\log n)$. We prove that this comparison complexity is theoretically optimal with respect to the examined error measures. An experimental evaluation against existing adaptive and non-adaptive sorting algorithms demonstrates the potential of applying learning-augmented algorithms in sorting tasks.

Sorting with Predictions

Abstract

We explore the fundamental problem of sorting through the lens of learningaugmented algorithms, where algorithms can leverage possibly erroneous predictions to improve their efficiency. We consider two different settings: In the first setting, each item is provided a prediction of its position in the sorted list. In the second setting, we assume there is a 'quick-and-dirty' way of comparing items, in addition to slow-and-exact comparisons. For both settings, we design new and simple algorithms using only O ( ∑ i log η i ) exact comparisons, where η i is a suitably defined prediction error for the i th element. In particular, as the quality of predictions deteriorates, the number of comparisons degrades smoothly from O ( n ) to O ( n log n ) . We prove that this comparison complexity is theoretically optimal with respect to the examined error measures. An experimental evaluation against existing adaptive and non-adaptive sorting algorithms demonstrates the potential of applying learning-augmented algorithms in sorting tasks.

1 Introduction

Sorting is one of the most basic algorithmic problems, commonly featured as one of the initial topics in computer science education, and with a vast array of applications spanning various domains. In recent years, the emerging field of algorithms with predictions (Lykouris and Vassilvitskii, 2021, Mitzenmacher and Vassilvitskii, 2022), also known as learning-augmented algorithms, has opened up new possibilities for algorithmic improvement, where algorithms aim to leverage predictions (possibly generated through machine learning, or otherwise) to improve their performance. However, the classical sorting problem with predictions, along with the discussion of different types of predictors for sorting, appears to have been largely overlooked by this recent movement. This paper explores the problem of sorting through the lens of algorithms with predictions in two settings, aiming to overcome the classical Ω( n log n ) barrier with the aid of various types of predictors.

The first setting involves each item having a prediction of its position in the sorted list. This type of predictor is commonly found in real-world scenarios. For instance, empirical estimations of element distribution can generate positional predictions. Another example is that a fixed set of items has their ranking evolve over time, with minor changes at each timestep. Here, an outdated ranking can serve as a natural prediction for the current ranking. As re-evaluating the true relation between items can be costly, the provided positional predictions offer useful information. The positional prediction setting is closely related to adaptive sorting of inputs with existing presortedness (Estivill-Castro and Wood, 1992), but we consider different measures of error on the predictor, resulting in algorithms with a more fine-grained complexity.

In the second setting, a 'dirty' comparison function is provided to assist sorting. In biological experiments, for instance, some 'indicating factors' might be used to approximately compare two molecules or drugs. Despite potential errors due to the oversight of minor factors, these comparisons can still offer preliminary insights into the properties of the subjects. More broadly, in experimental science, researchers often carry out costly experiments to compare subject behaviours. By utilizing a

proficient sorting algorithm that capitalizes on dirty comparisons, the need for costly experiments can be reduced and substituted by less expensive, albeit noisier, experiments.

We propose sorting algorithms to leverage either type of predictor. In the positional prediction setting, we design two deterministic algorithms with different complexity bounds, while in the dirty comparisons setting, we develop a randomized algorithm. In all settings, we provide bounds of the form O ( ∑ n i =1 log( η i +2)) on the number of exact comparisons, for different notions of element-wise prediction errors η i ∈ [0 , n ] . In particular, all three proposed algorithms only require O ( n ) exact comparisons if predictions are accurate ( consistency ), never use more than O ( n log n ) comparison regardless of prediction quality ( robustness ), and their performance degrades slowly as a function of prediction error ( smoothness ). Moreover, we show that all algorithms have optimal comparison complexity with respect to the error measures examined.

Finally, through experiments on both synthetic and real-world data, we evaluate the proposed algorithms against existing (adaptive and non-adaptive) algorithms. Results demonstrate their superiority over the baselines in multiple scenarios.

1.1 Preliminaries

Let A = ⟨ a 1 , . . . , a n ⟩ be an array of n items, equipped with a strict linear order < . Let p : [ n ] → [ n ] be the permutation that maps each index i to the position of a i in the sorted list; that is, a p -1 (1) < a p -1 (2) < · · · < a p -1 ( n ) . We consider two settings of sorting with predictions.

̸

Sorting with Positional Predictions. In sorting with positional predictions , the algorithm receives for each item a i a prediction ˆ p ( i ) of its position p ( i ) in the sorted list. We allow ˆ p to be any function [ n ] → [ n ] , which need not be a permutation (i.e., it is possible that ˆ p ( i ) = ˆ p ( j ) for some i = j ).

Positional predictions can be generated by models that roughly sort the items, e.g. focusing on major factors while neglecting minor ones. Or they can stem from past item rankings, while the properties of the items evolve over time. In such cases, the objective is to obtain the latest ranking of items.

The error of a positional prediction can be naturally quantified by the displacement of each element's prediction; that is, the absolute difference of the predicted ranking and the true ranking. We define the displacement error of item a i as

The following notion of one-sided error provides an alternative perspective to evaluate the complexity of algorithms with positional predictions. We denote the left-error and right-error of item a i as

In certain contexts, it may be impossible to obtain a predictor with small displacement error, but possible to obtain one with a small one-sided error. By developing algorithms for this setting, we expand the space of problems where sorting algorithms with predictions can be applied.

Sorting with Dirty and Clean Comparisons. The other setting we consider involves a predictor that estimates which of two elements is larger without conducting a proper comparison, providing a faster but possibly inaccurate result. This type of predictor is applicable in scenarios where exact comparisons are costly but a rough estimate of the comparison outcome can be obtained more easily.

Formally, in sorting with dirty comparisons , the algorithm has access to a complete, asymmetric relation ̂ < on the items, while still also having access to the exact comparisons < . That is, for any two distinct items a i and a j , either a i ̂ < a j or a j ̂ < a i . We think of ̂ < as an unreliable, but much faster to evaluate prediction of < . We also refer to the (fast) comparisons according to ̂ < as dirty whereas the (slow) comparisons according to < are clean . We emphasize that the relation ̂ < need not be transitive , so it is not necessarily a linear order. Instead, ̂ < induces a tournament graph on the items of A , containing a directed edge ( a i , a j ) if and only if a i ̂ < a j , and in many applications, we expect this graph to have cycles.

We denote by η i the number of incorrect dirty comparisons involving a i , that is,

̸

1.2 Main Results

Our main result for the dirty comparison setting is given by the following theorem:

Theorem 1.1. Augmented with dirty comparisons, there is a randomized algorithm that sorts an array within O ( n log n ) running time, O ( n log n ) queries to dirty comparisons, and O ( ∑ n i =1 log ( η i +2)) clean comparisons in expectation.

By the classical lower bound, the total number of dirty + clean comparisons must be at least Ω( n log n ) regardless of prediction quality, but for sufficiently good predictions the theorem allows us to replace most clean comparisons with dirty ones. The case where dirty comparisons are probabilistic is discussed in Section B.1.

The following theorem generalizes the previous one to the case that there are k different dirty comparison operators:

Theorem 1.2. Augmented with k ≤ 2 O ( n/ log n ) dirty comparison predictors, where the error of the p th predictor is denoted by η p , there is a randomized algorithm that sorts an array with at most O (min p ∑ n i =1 log( η p i +2)) clean comparisons.

In other words, the number of clean comparisons is as good as if we knew in advance which of the k predictors is best. The bound on k is almost tight, since already k ≥ 2 n log n would mean there could be one predictor for each of the n ! possible sorting outcomes, which would render them useless. The proof of Theorem 1.2 is given in Appendix B.2.

The next two theorems capture our algorithms for the positional prediction setting:

Theorem 1.3. Augmented with a positional predictor, there is a deterministic algorithm that sorts an array within O (∑ n i =1 log ( η ∆ i +2 )) running time and comparisons.

Theorem 1.4. Augmented with a positional predictor, there is a deterministic algorithm that sorts an array within O (∑ n i =1 log ( min { η l i , η r i } +2 )) comparisons.

We remark that there exist instances where the bound of Theorem 1.3 is stronger than that of Theorem 1.4 and vice versa. 1

The following theorem, proved in Appendix E, shows tightness of the aforementioned upper bounds.

Theorem 1.5. Augmented with dirty comparisons, no sorting algorithm uses o ( ∑ n i =1 log ( η i +2)) clean comparisons. Augmented with a positional predictor, no sorting algorithm uses o (∑ n i =1 log ( η ∆ i +2 )) or o (∑ n i =1 log(min { η l i , η r i } +2) ) comparisons.

Bounds in Terms of Global Error. One may wonder how the above bounds translate to a global error measure such as the number of item pairs where the larger one is incorrectly predicted to be no larger than the smaller one. Writing D for this error measure, in the dirty comparison setting we simply have D = 1 2 ∑ i η i , and in the positional prediction setting D ≥ 1 2 ∑ i η ∆ i by (Rohatgi, 2020, Lemma 11). Thus, concavity of logarithm and Jensen's inequality yield an upper bound of O ( n log ( D n +2 )) for both settings. This bound is tight 2 as a function of D and corresponds to the optimal complexity of adaptive sorting as a function of the number of inversions (Mannila, 1985).

However, our guarantees in terms of element-wise error are strictly stronger whenever Jensen's inequality is not tight, i.e., when the η i are non-uniform. Furthermore, it is reasonable to expect predictors to exhibit varying levels of error for different items, especially when the error originates from element-wise noise.

1.3 Related Works

Algorithms with Predictions. Our study aligns with the broader field of learning-augmented algorithms, also known as algorithms with predictions. The majority of research has focused on

√

√

classical online problems such caching (Lykouris and Vassilvitskii, 2021, Rohatgi, 2020, Wei, 2020, Bansal et al., 2022), rent-or-buy problems (Purohit et al., 2018, Gollapudi and Panigrahi, 2019, Angelopoulos et al., 2020, Wang et al., 2020, Antoniadis et al., 2021), scheduling (Purohit et al., 2018, Lattanzi et al., 2020, Mitzenmacher, 2020, Azar et al., 2021, 2022, Lindermayr and Megow, 2022) and many others. In comparison, research on learning-augmented algorithms to improve runnning time for offline problems is relatively sparser, but examples include matching (Dinitz et al., 2021, Sakaue and Oki, 2022), clustering (Ergun et al., 2022), and graph algorithms (Chen et al., 2022, Davies et al., 2023). Motivated by (Kraska et al., 2018), (Lykouris and Vassilvitskii, 2021) describes a simple method to speed up binary search with predictions, which has been inspirational for our work. Learning-augmented algorithms have also been applied to data structures such as binary search trees (Lin et al., 2022, Cao et al., 2023), and empirical works demonstrate the benefits of ML-augmentation for index structures (Kraska et al., 2018) and database systems (Kraska et al., 2019). There has also been increasing interest in settings where algorithms have access to multiple predictors (Gollapudi and Panigrahi, 2019, Wang et al., 2020, Bhaskara et al., 2020, Almanza et al., 2021, Emek et al., 2021, Dinitz et al., 2022, Anand et al., 2022, Antoniadis et al., 2023).

Related to sorting, Lu et al. (2021) studied learning-augmented generalized sorting, a variant of sorting where some comparisons are allowed while others are forbidden. The predictions they consider are similar to our dirty comparisons. They proposed two algorithms with comparison complexities O ( n log n + w ) and O ( nw ) , where w is the total number of incorrect dirty comparisons. In the classical (non-generalized) setting with all comparisons allowed, only the second bound theoretically improves upon O ( n log n ) , but the dependence on the error is exponentially worse than for our algorithms; even with a single incorrect dirty comparison per item, the O ( nw ) bound becomes O ( n 2 ) , whereas ours is O ( n ) . A recent work of Erlebach et al. (2023) studies sorting under explorable uncertainty with predictions, where initially only an interval around the value of each item is known, the exact values can be queried, a prediction of these values is given, and the goal is to minimize the number of queries needed to sort the list.

Deep Learning-Based Sorting. The thriving development of deep learning has inspired research into new sorting paradigms. A DeepMind study (Mankowitz et al., 2023) recast sorting as a singleplayer game, training agents to perform effectively, thus uncovering faster sorting routines for short sequences. Kristo et al. (2020) proposed a sorting algorithm that uses a learning component to improve the empirical performance of sorting numerical values. Their algorithm tries to approximate the empirical CDF of the input by applying ML techniques to a small subset of the input. This setting diverges from ours: If inputs are non-numeric (and no monotonous mapping to numbers is known), then one has to rely on a comparison function, and the approach of Kristo et al. (2020) would not be well-defined, whereas our algorithms can sort arbitrary data types.

Conversely, the input in (Kristo et al., 2020) is solely the item list, without any additional predictions. Note that predictions (or other assumptions) are necessary to beat the entropic Ω( n log n ) lower bound. 3

Noisy Sorting. Noisy sorting contemplates scenarios where comparison results may be incorrect. This model is useful to simulate potential faults in large systems. Two noisy sorting settings have primarily been considered: In independent noisy setting, each query's result is independently flipped with probability p ∈ (0 , 1 2 ) . Recently, Gu and Xu (2023) provided optimal bounds on the number of queries to sort n elements with high probability. Recurrent noisy setting (Braverman and Mossel, 2008) further assumes any repeated comparisons will yield consistent results. Geissmann et al. (2019) present an optimal algorithm that guarantees O ( n log n ) time, O (log n ) maximum dislocation, and O ( n ) total dislocation with high probability. While the recurrent noisy setting is closely related to dirty comparisons, studies in that field focus primarily on approximate sorting; to the best of our knowledge, no exact sorting algorithms that use both dirty and clean comparisons have been studied.

Adaptive Sorting. Adaptive sorting algorithms leverage various types of existing order in the input, achieving better complexity for partially sorted data. Examples include TimSort (Peters, 2002), which is the standard sorting algorithm in a variety of programming languages, Cook-Kim division

(Cook and Kim, 1980), and Powersort (Munro and Wild, 2018), which recently replaced TimSort in Python's standard library. We refer to the survey of Estivill-Castro and Wood (1992) for a broader overview. The concept of pre-sortedness is closely related to positional predictions; however, without the motivation from predictors, the complexity bound on adaptive sorting algorithms was often considered under error measures on the entire array, instead of on each element. In contrast, our error measure is element-wise, allowing algorithms with stronger complexity bounds.

2 Sorting with Dirty Comparisons

Given a dirty predictor ̂ < , our goal is to sort A with the least possible number of clean comparisons. Note that, if ̂ < is accurate, A can be sorted using only O ( n ) clean comparisons and O ( n log n ) dirty comparisons. This could be achieved, for example, by performing Merge Sort with dirty comparisons and then validating the result through clean comparisons between adjacent elements. This observation motivates us to consider that not all O ( n 2 ) dirty comparisons are necessary, and we should devise an algorithm minimizing the number of both clean and dirty comparisons.

We propose a randomized algorithm that sorts A with expected O ( ∑ log(2 + η i )) clean comparisons, expected O ( n log n ) dirty comparisons, and expected O ( n log n ) running time. The key idea consists of three parts: 1) Sequentially insert each element of A into a binary search tree, following random order; 2) Guide each insertion primarily with dirty comparisons, while verifying the correctness of it using a minimal number of clean comparisons. 3) Correct the mistake induced by the dirty insertion, ensuring that the clean comparisons needed for correction is O (log η i ) in expectation.

We describe the algorithm in Section 2.1 and prove its performance guarantees in Section 2.2. In Appendix B we discuss extensions of the dirty comparison setting.

2.1 Algorithm

We now describe the sorting algorithm with dirty comparisons in detail (Algorithm 1). We initialize B as an empty binary search tree (BST). For any vertex v , we denote by left( v ) and right( v ) its left and right children, and by root( B ) the root of B . Slightly abusing notation, we write v also for the item stored at vertex v . If any of these vertices is missing, the respective variable has value NIL.

̸

̸

Within each iteration of the for-loop starting in line 2, we select one item a i of A uniformly randomly from the items that have not been processed yet. Then, we insert this item a i into B , while maintaining the invariant that B remains a BST with respect to clean comparisons < .

Inserting item a i into B requires three phases, as illustrated in Figure 2.1. The first phase involves performing a search for the insertion position using dirty comparisons ̂ < , keeping track of the search path. Here, we denote by C t the t th vertex on this path, and by L t and R t the lower and upper

root

dirtysearch

stept*

verification

cleansearch

failedverification

successfulverification

aj

stepT

bounds on items that can be inserted in the subtree rooted at C t without violating the BST property with respect to < . Correctness of the choice of L t and R t follows from the fact that B was a BST with respect to < before the current insertion. This dirty procedure stops when the search path reaches a NIL-leaf, regarded as the predicted position for a i 's insertion. However, since we used dirty comparisons to trace the path, a i might violate one of the boundary conditions L t < a i or a i < R t at some recursion step t . We call a recursion step t valid for a i if L t < a i < R t . Then, we enter the verification phase in line 9. We traverse the dirty search path in reverse order to locate the last valid step t ∗ . A naive method to do this (which is sufficient for our asymptotic guarantees) is to repeatedly decrease t by 1 until t is valid; we discuss an alternate, more efficient method in Remark A.6, which yields a better constant factor.

The final phase involves performing a clean search starting from C t ∗ to determine the correct insertion position for a i . After inserting a i into that position, B remains a BST with respect to < . Once all items of A are inserted into B , we can obtain the sorted order through the inorder traversal of B .

2.2 Complexity Analysis

The idea is to show that the dirty search path has expected depth O (log n ) , whereas the verification path and the clean search path only have expected depth O (log η i ) . Therefore, the algorithm only needs O (log n ) dirty and O (log η i ) clean comparisons for inserting each a i ; Theorem 1.1 is established consequently. The full proof is given in Appendix A.

3 Sorting with Positional Predictions

In this section, we propose two algorithms that are capable of leveraging positional predictions. The first one has complexity bounds in terms of the displacement error measure, and the second one has complexity bounds in terms of η l and η r , the one-sided error measures. Each algorithm is effective in some tasks, where the given prediction is accurate with respect to its corresponding error measure.

3.1 Displacement Sort

We present a sorting algorithm with positional prediction, whose comparison and time complexity rely solely on the displacement error of the predictor. The algorithm is adapted from Local Insertion Sort (Mannila, 1985), an adaptive sorting algorithm proven to be optimal for various measures of presortedness.

We use a classic data structure called finger tree (Guibas et al., 1977), which is a balanced binary search tree (BBST) equipped with a pointer ('finger') pointing towards the last inserted vertex. When a new value v is to be inserted, rather than searching for insertion position from the root, the insertion position is found by moving the finger from the last inserted vertex u to the suitable new position. By the balance property of BBST, the insertion can be performed in O (log d ( u, v )) amortized time, where d ( u, v ) is the number of vertices in the tree whose value lies in the closed interval from u to v .

Algorithm 2 details the proposed method. We first bucket sort (in time O ( n ) ) the items in A based on their predicted positions, such that we may assume for all i < j that ˆ p ( i ) ≤ ˆ p ( j ) . Following the

rearranged order, items in A are sequentially inserted into an initially empty finger tree T . After all insertions, we obtain the exactly sorted array by an inorder traversal of T .

The full proof of run time and comparison complexity of Algorithm 2 is in Appendix C.

3.2 Double-Hoover Sort

Now we turn our focus to settings where one-sided errors are small. We first describe a simple algorithm with comparison complexity as a function of either η l or η r . Following this, we introduce our algorithm Double-Hoover Sort, which has comparison complexity as claimed in Theorem 1.4. Both algorithms begin by bucket sorting A with respect to the positional prediction in O ( n ) time. Thus, we may subsequently assume that A is rearranged such that ∀ i < j, ˆ p ( i ) ≤ ˆ p ( j ) .

A First Approach. A left-sided sorting complexity of O ( ∑ n i =1 log(2 + η l i )) can be easily achieved using the standard technique of learning-augmented binary search, as described in (Lykouris and Vassilvitskii, 2021, Mitzenmacher and Vassilvitskii, 2022). Specifically, a sorted array L is maintained, and a 1 , . . . , a n are sequentially inserted into L . During each insertion, we perform a learningaugmented binary search starting from the rightmost position of L , taking O (log( η l i +2)) comparisons to find the correct insertion position. In total, O ( ∑ i log( η l i +2)) comparisons are taken among all insertions. By replacing the array L with an appropriate data structure (e.g., a BBST with a finger that always returns to the rightmost element), one can achieve the same bound also for time complexity. A reversed 'right-sided' version of this algorithm achieves complexity O ( ∑ i log( η r i + 2)) . By simultaneously running the left-sided and right-sided algorithms, one can achieve the complexity bound of O ( min {∑ n i =1 log ( 2 + η l i ) , ∑ n i =1 log(2 + η r i )}) . However, moving the min operator inside the summation requires more elaborate approach.

Double-Hoover Sort. The basic idea is that, to utilize a similar insertion scheme to that employed in the one-sided algorithm, we maintain two sorted structures L and R at the same time, and insert each item into one of them, depending on which operation is faster, hereby achieving a complexity bound of O (log(min( η l i , η r i ) + 2)) . Then, a final sorted list is attained by merging L and R in linear time. However, a significant issue yet to be addressed is how to decide the insertion order of different items.

Consider two items a u and a v with u < v (so ˆ p ( u ) ≤ ˆ p ( v ) by Bucket Sort). In our algorithm, if both are to be inserted into L , it is crucial that a u is inserted prior to a v . Otherwise, the insertion complexity of a u could exceed the bound of log( η l u +2) . Conversely, if both a u and a v are to be inserted into R , then a v should be inserted prior to a u . Since we cannot predict whether an item will be inserted into L or R , formulating an appropriate insertion order that respects the constraints on both sides is impossible.

We tackle this issue of insertion order with a strength-based, log n -rounds insertion scheme . Intuitively, we think of L and R as two 'hoovers', with their 'suction power' increasing simultaneously over time. Each item (as 'dust') is extracted from the array and inserted into one hoover once the suction power reaches the required strength: a hoover with suction power δ is able to absorb items that can be inserted into it with O (log δ ) comparisons. Details are illustrated in Algorithm 3.

The sorted structures L and R can be implemented by arrays, though alternative data structures could provide better time complexity.

We conduct insertions in ⌈ log n ⌉ rounds, setting δ to be 1 , 2 , 4 , . . . , 2 ⌈ log n ⌉ . In each round, we iterate over a 1 , . . . , a n to decide if they should be inserted to L in the current round. Then, we iterate over a n , . . . , a 1 reversely, to decide if they should be inserted to R in the current round. Note that if an

Algorithm 3: Double-Hoover Sort

A: a1 (

a2

ai

an

items in L

OOOO

O

OOO

inserteditem

strength: 8

ai>ls

comparison with boundary

L

binarysearch

OOOOQ

insert a; to L

item is inserted into either L or R , it is omitted in later rounds. The insertion process of an item in one round is depicted in Figure 3.1.

To decide whether a i should be inserted into L with strength δ , let l i δ denote the δ th largest item in L with index smaller than i , representing the 'boundary value' in this round. If there are less than δ eligible items in L , set l i δ to be -∞ . Let l i represent min δ ′ <δ l i δ ′ , the minimum boundary value in previous rounds. If l i δ is smaller than a i , we employ binary search to insert a i into L , starting with the interval { x ∈ L : l i δ ≤ x ≤ l i } . Conversely, to decide whether a i should be inserted into R , we adopt a symmetrical approach as depicted in lines 11 to 15 of Algorithm 3.

After all insertion rounds, we merge L and R in linear time to obtain the sorted result.

Correctness. The correctness of the Double-Hoover Sort arises from the invariance that both L and R remain sorted after all insertions. It is sufficient to show that the initial insertion intervals at line 8 and line 14 always cover the value of a i . At line 8, l i δ < a i holds trivially by conditioning. Since a i is not inserted into L in any previous round, a i < l i δ ′ for all δ ′ < δ . Since the minimum operation preserves inequality, a i < l i also holds. A similar argument can be made for the interval at line 14.

To discern the comparison complexity of the proposed algorithm, we propose the following lemma.

Lemma 3.1. Upon the insertion of an item a i into L , all items in L \ L <i are larger than l i . Similarly, when an item a i is inserted into R , all items in R \ R >i are smaller than r i . As a result, the initial interval at line 8 and line 14 are subsets of L <i and R <i , and hence have sizes no larger than δ .

Theorem 3.2. For each item a i , its insertion process takes O (log(min { η l i , η r i } +2)) comparisons.

The full proof of Lemma 3.1 and Theorem 3.2 are given in Appendix D.

Then, Theorem 1.4 can be obtained from Theorem 3.2, by summing up the number of comparisons in the insertion process of each a i .

4 Experiments

In this section, we conduct experiments both on synthetic data, crafted to simulate predictions in real-world settings, and also on real-world data of countries' population ranking. The source code used for experiments is available at https://github.com/xingjian-bai/ learning-augmented-sorting .

We assess the performance of our proposed sorting algorithms against five well-established baselines. Quick Sort and Merge Sort are classic sorting algorithms with O ( n log n ) complexity; Tim Sort (Peters, 2002) is a popular hybrid sorting algorithm designed to perform efficiently on real-world datasets and widely adopted in standard libraries. Further, we choose two adaptive sorting algorithms, Odd-Even Straight Merge Sort (Estivill-Castro and Wood, 1992) and Cook-Kim division (Cook and Kim, 1980), which are proven to be optimal with respect to several measures of disorderness. They serve as adaptive variants of Merge Sort and Quick Sort. To apply adaptive sorting algorithms in positional prediction settings, we first execute bucket sort on the items by their predicted ranking, breaking ties arbitrarily. This 'sorted-by-prediction' array is then inputted into the baselines. In Appendix F.2, we also show results of an experimental comparison against Insertion Sort.

Positional Predictions. First, we elaborate on our synthetic data generation process. In many sorting tasks, items belong to different 'grades', which represent a coarse version of the ranking. For example, students are classified into grades A, B, C, and D based on their exam scores; with their grades in hand, we want to find out their accurate ranking. We denote this scenario as the class setting . Specifically, we divide an array of n items into c classes, sampling the thresholds t 0 = 0 ≤ t 1 < t 2 < . . . , t c = n uniformly at random. Then, for items a i with t k -1 < i ≤ t k , we say that they belong to the k th class, and their predicted position is uniformly generated from ( t k -1 , t k ] .

To model the tasks where we have an 'outdated' ranking, we design the decay setting . The accurate ranking is obtained as the prediction at time 0 . Then, during each time step, one item is randomly selected to be perturbed: its predicted position is shifted by 1, towards either left or right, with uniform probability. We then ask the sorting algorithms to retrieve the original ranking of items based on the prediction at each time step.

We also utilize data from a real-world setting. We draw the annual population ranking of countries and smaller regions from 1960 to 2010 from World Bank (2023). Then, we feed in the ranking in year x = 1960 , . . . , 2010 respectively as the prediction, and ask the sorting algorithm to predict the ranking in year 2010 .

50

25

10

#comparisons

n

/

40

OESM

Double-Hoover Sort

comparisons

n

20

comparisons

n

30

15

20

10

#

#

10

5

Double-Hoover Sort

o w

Double-Hoover Sort

OESM

OESM

0.0

0.2

#classes/n

0.4

0.6

0.8

1.0

0

0

100

200

300

#timesteps/n

400

500

600

700

800

900

1000

5

ot

15

20

gapinyears

25

30

35

40

45

50

55

60

In all the plots, the X-axis indicates the quality of predictions, and the Y-axis indicates the number of comparisons used. The red dotted line is n log 2 n . The bold curves represent the proposed algorithms. All experiments are repeated 30 times, with the standard deviation indicated by shade. A scapegoat tree implementation of Double-Hoover Sort is used for synthetic settings, while an array implementation is used for population ranking given the small sample size.

As depicted in Figure 4.1, our algorithms consistently outperform the baselines in all settings with various task sizes. Specifically, in the class setting with n = 1 , 000 , 000 , Displacement Sort and Double-Hoover Sort outperform all baselines when the number of classes is larger than 0 . 05 n . In the decay setting, both our algorithms perform better than the others as time progresses. In the real-world dataset, country population ranking, Displacement Sort needs the fewest comparisons when the given prediction is within 5 years, and Double-Hoover Sort dominates the rest when the prediction is obtained 6 to 60 years ago. These experiments illustrate that the proposed algorithms can leverage positional predictions more effectively than traditional adaptive and non-adaptive sorting algorithms in a variety of settings.

Dirty Comparisons. In some sorting scenarios, some 'indicating factors' can be used to cheaply compare two items. For instance, in biology, we can compare the binding affinities of two molecules for a specific target protein and provide information about their potential efficacy as drugs. However, comparisons based on indicating factors may have error induced by element-wise noise. Hence, we consider a two dirty-comparison settings in which a ratio r of items is damaged. We say a dirty comparison is perturbed if its outcome is uniformly random. In the Good-Dominating setting , a dirty comparison between two items is perturbed if both are damaged; in the Bad-Dominating setting , a dirty comparison between two items is perturbed if either item is damaged.

In dirty comparisons settings, we use the 3-approximation feedback arc set algorithm proposed by Ailon et al. (2008) to preprocess the dirty comparisons. This algorithm uses O ( n log n ) dirty comparisons, the same order of magnitude as our Dirty-Clean Sort, to construct a positional prediction that roughly aligns with the given dirty comparisons. Then, we feed in the induced positional prediction to the baselines.

40

40

comparisons

一

n

35

30

OESM

Dirty-Clean Sort

Double-Hoover Sort

comparisons

n

35

30

25

25

20

20

15

15

#

10

#

10

5

Double-Hoover Sort

OESM

—Dirty-Clean Sort

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

damageratior

damageratior

As showcased in Figure 4.2, when n = 100 , 000 , in the Good-Dominanting setting, our proposed Dirty-Clean Sort outperforms baselines when r ≤ 0 . 7 ; in the Bad-Dominating setting, it outperforms other algorithms when r ≤ 0 . 25 . If the damage ratio is large, the prediction becomes chaotic, but still performs essentially no worse than Quick Sort for reasons discussed in Remark A.6 in the Appendix.

5 Limitations and Future Work

In the dirty-clean setting, our algorithm still requires a time complexity of O ( n log n ) due to the processing of O ( n log n ) dirty comparisons. Consequently, the algorithm is more appropriate for situations where exact comparisons are expensive than for those where comparisons are fast. In the positional prediction setting, Displacement Sort achieves a bound of O ( ∑ i log( η ∆ i + 2)) for both comparison and time complexity, whereas the Double-Hoover sort achieves its guarantee O ( ∑ i log(min { η l i , η r i } +2)) only for comparison complexity. An intriguing question is whether the latter bound can be achieved for time complexity as well. Another potential limitation is that predictions might not be learnable in some sorting settings; future work could focus on exploring the conditions under which predictions are learnable.

Broader Impact. Our paper proposes efficient sorting algorithms when a predictor is available. We do not expect any negative societal impact.

Acknowledgments. We thank the anonymous reviewers at NeurIPS and Luke Melas-Kyriazi for their valuable comments.

References

Ailon, M. Charikar, and A. Newman. Aggregating inconsistent information: Ranking and clustering. J. ACM , 55(5):23:1-23:27, 2008. URL https://doi.org/10.1145/1411509. 1411513 .

Almanza, F. Chierichetti, S. Lattanzi, A. Panconesi, and G. Re. Online facility location with multiple advice. In NeurIPS , 2021. URL https://proceedings.neurips.cc/paper/2021/ hash/250473494b245120a7eaf8b2e6b1f17c-Abstract.html .

Anand, R. Ge, A. Kumar, and D. Panigrahi. Online algorithms with multiple predictions. In ICML , 2022. URL https://proceedings.mlr.press/v162/anand22a.html .

Angelopoulos, C. Dürr, S. Jin, S. Kamali, and M. P. Renault. Online computation with untrusted advice. In ITCS , 2020. URL https://doi.org/10.4230/LIPIcs.ITCS.2020.52 .

Antoniadis, C. Coester, M. Eliás, A. Polak, and B. Simon. Learning-augmented dynamic power management with multiple states via new ski rental bounds. In NeurIPS , 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 8b8388180314a337c9aa3c5aa8e2f37a-Abstract.html .

Antoniadis, C. Coester, M. Eliás, A. Polak, and B. Simon. Mixing predictions for online metric algorithms. In ICML , 2023. URL https://proceedings.mlr.press/v202/antoniadis23b. html .

Azar, S. Leonardi, and N. Touitou. Flow time scheduling with uncertain processing time. In STOC , 2021. URL https://doi.org/10.1145/3406325.3451023 .

Azar, S. Leonardi, and N. Touitou. Distortion-oblivious algorithms for minimizing flow time. In SODA , 2022. URL https://doi.org/10.1137/1.9781611977073.13 .

Bansal, C. Coester, R. Kumar, M. Purohit, and E. Vee. Learning-augmented weighted paging. In SODA , 2022. URL https://doi.org/10.1137/1.9781611977073.4 .

Bhaskara, A. Cutkosky, R. Kumar, and M. Purohit. Online linear optimization with many hints. In NeurIPS , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 6c250b592dc94d4de38a79db4d2b18f2-Abstract.html .

Braverman and E. Mossel. Noisy sorting without resampling. In SODA , 2008. URL http: //dl.acm.org/citation.cfm?id=1347082.1347112 .

Cao, J. Chen, L. Chen, C. Lambert, R. Peng, and D. Sleator. Learning-augmented b-trees, 2023. URL https://doi.org/10.48550/arXiv.2303.15379 .

Y. Chen, S. Silwal, A. Vakilian, and F. Zhang. Faster fundamental graph algorithms via learned predictions. In ICML , 2022. URL https://proceedings.mlr.press/v162/chen22v.html .

R. Cook and D. J. Kim. Best sorting algorithm for nearly sorted lists. Commun. ACM , 23(11): 620-624, 1980. URL https://doi.org/10.1145/359024.359026 .

Davies, B. Moseley, S. Vassilvitskii, and Y. Wang. Predictive flows for faster ford-fulkerson. In ICML , 2023. URL https://proceedings.mlr.press/v202/davies23b.html .

Dinitz, S. Im, T. Lavastida, B. Moseley, and S. Vassilvitskii. Faster matchings via learned duals. In NeurIPS , 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 5616060fb8ae85d93f334e7267307664-Abstract.html .

Dinitz, S. Im, T. Lavastida, B. Moseley, and S. Vassilvitskii. Algorithms with prediction portfolios. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 7f9220f90cc85b0da693643add6618e6-Abstract-Conference.html .

Emek, S. Kutten, and Y. Shi. Online paging with a vanishing regret. In ITCS , 2021. URL https://doi.org/10.4230/LIPIcs.ITCS.2021.67 .

Ergun, Z. Feng, S. Silwal, D. P. Woodruff, and S. Zhou. Learning-augmented k-means clustering. In ICLR , 2022. URL https://openreview.net/forum?id=X8cLTHexYyY .

Erlebach, M. S. de Lima, N. Megow, and J. Schlöter. Sorting and hypergraph orientation under uncertainty with predictions. In IJCAI , 2023. URL https://doi.org/10.24963/ijcai.2023/ 619 .

Estivill-Castro and D. Wood. A survey of adaptive sorting algorithms. ACM Comput. Surv. , 24(4): 441-476, 1992. URL https://doi.org/10.1145/146370.146381 .

Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997. URL https://www.sciencedirect.com/science/article/pii/S002200009791504X .

Geissmann, S. Leucci, C. Liu, and P. Penna. Optimal sorting with persistent comparison errors. In ESA , 2019. URL https://doi.org/10.4230/LIPIcs.ESA.2019.49 .

Gollapudi and D. Panigrahi. Online algorithms for rent-or-buy with expert advice. In ICML , 2019. URL http://proceedings.mlr.press/v97/gollapudi19a.html .

Gu and Y. Xu. Optimal bounds for noisy sorting. In STOC , 2023. URL https://doi.org/10. 1145/3564246.3585131 .

J. Guibas, E. M. McCreight, M. F. Plass, and J. R. Roberts. A new representation for linear lists. In STOC , 1977. URL https://doi.org/10.1145/800105.803395 .

Kraska, A. Beutel, E. H. Chi, J. Dean, and N. Polyzotis. The case for learned index structures. In SIGMOD , 2018. URL https://doi.org/10.1145/3183713.3196909 .

Kraska, M. Alizadeh, A. Beutel, E. H. Chi, A. Kristo, G. Leclerc, S. Madden, H. Mao, and V. Nathan. SageDB: A learned database system. In CIDR , 2019. URL http://cidrdb.org/ cidr2019/papers/p117-kraska-cidr19.pdf .

Kristo, K. Vaidya, U. Çetintemel, S. Misra, and T. Kraska. The case for a learned sorting algorithm. In SIGMOD , 2020. URL https://doi.org/10.1145/3318464.3389752 .

Lattanzi, T. Lavastida, B. Moseley, and S. Vassilvitskii. Online scheduling via learned weights. In SODA , 2020. URL https://doi.org/10.1137/1.9781611975994.114 .

Lin, T. Luo, and D. P. Woodruff. Learning augmented binary search trees. In ICML , 2022. URL https://proceedings.mlr.press/v162/lin22f.html .

Lindermayr and N. Megow. Permutation predictions for non-clairvoyant scheduling. In SPAA , 2022. URL https://doi.org/10.1145/3490148.3538579 .

Lu, X. Ren, E. Sun, and Y. Zhang. Generalized sorting with predictions. In SOSA , 2021. URL https://doi.org/10.1137/1.9781611976496.13 .

Lykouris and S. Vassilvitskii. Competitive caching with machine learned advice. J. ACM , 68(4): 24:1-24:25, 2021. URL https://doi.org/10.1145/3447579 .

J. Mankowitz, A. Michi, A. Zhernov, M. Gelmi, M. Selvi, C. Paduraru, E. Leurent, S. Iqbal, J.-B. Lespiau, A. Ahern, T. Köppe, K. Millikin, S. Gaffney, S. Elster, J. Broshear, C. Gamble, K. Milan, R. Tung, M. Hwang, T. Cemgil, M. Barekatain, Y. Li, A. Mandhane, T. Hubert, J. Schrittwieser, D. Hassabis, P. Kohli, M. Riedmiller, O. Vinyals, and D. Silver. Faster sorting algorithms discovered using deep reinforcement learning. Nature , 618(7964):257-263, 2023. URL https://doi.org/10.1038/s41586-023-06004-9 .

Mannila. Measures of presortedness and optimal sorting algorithms. IEEE Trans. Computers , 34 (4):318-325, 1985. URL https://doi.org/10.1109/TC.1985.5009382 .

Mitzenmacher. Scheduling with predictions and the price of misprediction. In ITCS , 2020. URL https://doi.org/10.4230/LIPIcs.ITCS.2020.14 .

Mitzenmacher and S. Vassilvitskii. Algorithms with predictions. Commun. ACM , 65(7):33-35, 2022. URL https://doi.org/10.1145/3528087 .

I. Munro and S. Wild. Nearly-optimal mergesorts: Fast, practical sorting methods that optimally adapt to existing runs. In ESA , 2018. URL https://doi.org/10.4230/LIPIcs.ESA.2018.63 .

Peters. [Python-Dev] Sorting. Python Developers Mailinglist, 2002. URL https://mail. python.org/pipermail/python-dev/2002-July/026837.html .

Purohit, Z. Svitkina, and R. Kumar. Improving online algorithms via ML predictions. In NeurIPS , 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 73a427badebe0e32caa2e1fc7530b7f3-Abstract.html .

Rohatgi. Near-optimal bounds for online caching with machine learned advice. In SODA , 2020. URL https://doi.org/10.1137/1.9781611975994.112 .

Sakaue and T. Oki. Discrete-convex-analysis-based framework for warm-starting algorithms with predictions. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/ hash/844e61124d9e1f58632bf0c8968ad728-Abstract-Conference.html .

Wang, J. Li, and S. Wang. Online algorithms for multi-shop ski rental with machine learned advice. In NeurIPS , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 5cc4bb753030a3d804351b2dfec0d8b5-Abstract.html .

Wei. Better and simpler learning-augmented online caching. In APPROX/RANDOM , 2020. URL https://doi.org/10.4230/LIPIcs.APPROX/RANDOM.2020.60 .

World Bank. Population, total, 2023. URL https://data.worldbank.org/indicator/SP.POP. TOTL . United Nations Population Division. World Population Prospects: 2022 Revision.

A Proof of Comparison Complexity of Dirty-Clean Sort

The goal of this section is to prove Theorem 1.1. We start with several lemmas on the expected behavior of dirty search and clean search, focusing on a single iteration of the for-loop corresponding to item a i .

For an iteration of the dirty and clean search while-loops, respectively, we call the vertex stored as C t resp. C at the start of the iteration the pivot ; the subtree rooted at the pivot is referred to as the active subtree . The size of a subtree is the number of non-NIL vertices it contains. For the dirty search, let s t denote the size of the active subtree at iteration t , and T denote the number of recursion steps needed. Denote by m dirty s and m clean s the number of iterations of the dirty and clean search where the size of the active subtree lies in ( s 2 , s ] . In particular, m dirty s = |{ t : s t ∈ ( s 2 , s ] }| .

Proof. We employ a percentile argument. Consider the first step of either while-loop where the active subtree has at most s vertices, and let V be the set of these vertices. Note that the pivot in this step is the first element in V inserted into the tree. Conditioned on the vertices of the active subtree being V , elements of V are equally likely to be the pivot, since their insertion order is uniformly random. This is true even when conditioned on s t ∗ = s ′ , since reordering the elements within the set V does not change the value of s t ∗ , which is determined at higher vertices of the tree. Thus, we have at least a 50% chance that the pivot lies between the 25th percentile and the 75th percentile, in which case both children subtrees contain at most 3 4 | V | vertices each. Hence, the size of active subtree shrinks by a factor of 3 4 after at most 2 steps in expectation, and it shrinks to size smaller than s 2 after at most O (1) steps in expectation.

Based on this lemma, we are able to characterize the expected length of dirty search, clean search, and dirty search after time t ∗ .

Lemma A.2. A dirty search takes O (log n ) steps in expectation, i.e. E [ T ] = O (log n ) .

Proof. In each dirty search, the initial largest subtree has size at most n . We can apply Lemma A.1 repeated on s = n, ⌊ n 2 ⌋ , . . . , 1 . By linearity of expectation, we derive that a dirty search takes ⌈ log n ⌉ · O (1) = O (log n ) steps in expectation.

Lemma A.3. A clean search takes O ( E [log( s t ∗ +1)]) steps in expectation; in dirty search after reaching t ∗ , there are O ( E [log( s t ∗ +1)]) steps in expectation, i.e. E [ T -t ∗ ] = O ( E [log( s t ∗ +1)]) .

Proof. Assume s t ∗ = s ′ . Then, in dirty search after t ∗ , the initial largest subtree has size s ′ , and the rest of the active subtrees all have size at most s ′ -1 . By applying the second part of Lemma A.1 on s = s ′ -1 , ⌊ s ′ -1 2 ⌋ , . . . , 1 and summing, we obtain E [ T -t ∗ | s t ∗ = s ′ ] = O (log( s ′ +1)) . Thus,

The bound on clean search follows in the same way.

In order to relate E [log ( s t ∗ +1)] to the prediction error η i , the next lemma first characterizes the probability of a given time step t being t ∗ as a function of the subtree size s t .

Proof. Recall that t ∗ is the last valid time step. A shift from a valid time step to an invalid time step occurs only if the dirty comparison between the pivot and a i is wrong. Among all s t potential pivots, at most η i can have mistaken dirty comparisons with a i , and they are equally likely to be the pivot (depending on which of them was inserted first). Hence, given s t ∈ ( 2 k -1 , 2 k ] , the probability of a pivot with mistaken comparison is at most η i /s t ≤ η i / 2 k -1 .

Based on Lemma A.4, we present the central claim bridging prediction error with comparison complexity.

Lemma A.5. E [log ( s t ∗ )] = O (log ( η i +1)) .

Proof. We have

where the first inequality uses Lemma A.4 and the second is due to Lemma A.1. Thus,

Theorem 1.1 is subsequently deduced.

Proof of Theorem 1.1. Dirty comparisons are only conducted during dirty search, when the recursion step is incremented by one. As per Lemma A.2, the total number of dirty comparisons is bounded by the sum of steps across all insertions, which is n · O (log n ) .

In each verification phase, as we traverse the dirty search in reverse order to locate t ∗ , at most T -t ∗ +2 clean comparisons suffice. In each clean search phase, the expected number of clean comparisons is the expected number of steps in clean search. Therefore, based on Lemma A.3 and Lemma A.5, the total number of clean comparisons performed in both phases is O ( ∑ log ( η i +2)) .

Additionally, the running time is dominated by the number of dirty and clean comparisons.

Remark A.6. The algorithm can be implemented such that the number of clean comparisons is at most that of quicksort plus O ( n log log n ) , regardless of prediction error. Thus, even with terrible predictions our algorithm matches the performance of quicksort up to a factor that tends to 1 as n →∞ .

To achieve this, we can implement the verification step by decreasing t in geometrically increasing step sizes until a valid t has been found, and then perform a binary search for t ∗ between the last two attempted values of t . This reduces the number of clean comparisons in line 9 from O ( T -t ∗ ) to O (log( T -t ∗ )) , which is at most O (log log n ) in expectation by Lemma A.2, and at most O ( n log log n ) for all verification steps together. The remaining clean comparisons are performed during the clean searches. In the worst case (when all dirty comparisons are incorrect) all clean searches start from the root, and together they perform exactly the same set of comparisons as quicksort (by a coupling argument between the random choices of the two algorithms: E.g., the root of the search tree corresponds to the initial uniformly random pivot of quicksort).

B Extentions of Dirty Comparison Setting

We consider two extensions of the dirty comparison setting. The first one assumes that dirty comparisons are probabilistic; the second one discusses the setting where multiple predictors are available. We briefly discuss the extension of our algorithms and results in these new settings.

B.1 Probabilistic Dirty Comparisons

Our algorithm extends to the case where dirty comparisons are probabilistic. Assume that for each pair of objects i, j , the dirty comparison between them yields an incorrect result with probability η ij . Algorithm 1 can be directly applied in this setting, achieving the same guarantees by defining η i = ∑ j η ij . The proof remains unchanged.

If repeatedly querying the same dirty comparison multiple times yields independent results, the number of clean comparisons can be further reduced: Let ϵ ij := min { η ij , 1 / 2) } . When querying a dirty comparison 2 k times, the probability that the correct answer fails to secure a majority vote is at most (4 ϵ ij (1 -ϵ ij )) k : For η ij ≥ 0 . 5 , this bound is trivial. Otherwise, there are 2 2 k strings of length 2 k over the alphabet { correct , incorrect } , and each string that is at least half incorrect has probability at most ( ϵ ij (1 -ϵ ij )) k .

So by repeating each dirty comparison query 2 k times, we obtain an algorithm that performs O ( kn log n ) dirty comparisons and O ( ∑ i log ( ∑ j (4 ϵ ij (1 -ϵ ij )) k )) clean comparisons.

B.2 Multiple Predictors

We now discuss the setting where multiple predictors are available and prove Theorem 1.2. Suppose we have k different dirty comparison predictors. Let η p i denote the number of incorrect comparisons by predictor p for item i .

We prove Theorem 1.2 by reduction to the problem of 'prediction with expert advice': In this problem, there are k experts, and each incurs a loss in the range [0 , 1] per time step. An algorithm must select an expert in each round before the losses are revealed and then incurs the loss of the chosen expert. According to (Freund and Schapire, 1997, Equation(9)), their algorithm HEDGE has an expected loss of O ( L +log( k )) , where L is the total loss of the best expert in hindsight.

In our case, the experts correspond to the predictors, and time steps correspond to the n iterations of the for-loop of Algorithm 1 where an item is inserted into the BST. We define the loss of expert p in the time step where a i ought to be inserted by ℓ p i = log(1 + ˜ η p i ) / log( n ) , where ˜ η p i ≤ η p i is the number of incorrect comparisons of predictor p between a i and the items already in the BST at this time. Note that the value of ˜ η p i can be determined at the end of the time step (once a i is correctly placed in the BST) without any additional clean comparisons; the division by log n ensures that ℓ p i ∈ [0 , 1] as required.

The algorithm for multiple predictors proceeds as Algorithm 1, querying for dirty comparisons at a given time step the predictor p corresponding to the expert chosen by HEDGE at that time step. Recall from the analysis in Section 2.2 that the expected number of clean comparisons in this time step is then O (log(1 + ˜ η p i )) , which is an O (log n ) factor larger than the loss suffered by HEDGE. Consequently, by the O ( L + log k ) bound on the cost of HEDGE, the total expected number of clean comparisons is O (min p ∑ i log(1 + η p i ) + log( k ) log( n )) . Since k ≤ 2 O ( n/ log n ) , the term log( k ) log( n ) = O ( n ) is negligible, and Theorem 1.2 follows.

C Proof of Comparison Complexity of Displacement Sort

Proof of Theorem 1.3. We focus on the insertion process of each item a i in Algorithm 2. Let d i denote the number of nodes between a i and a i -1 in an inorder traversal of the tree after inserting a i , including themselves. These nodes must have their correct ranking in the final sorted list between p ( i -1) and p ( i ) ; hence

Therefore, the running time and number of comparisons among all insertions are bounded by

The second inequality is by ˆ p ( i -1) ≤ ˆ p ( i ) and the triangle inequality; the third inequality is by monotonicity of logarithm. The penultimate inequality is justified by log(max { x, y, z } ) ≤ log x +log y +log z for all x, y, z ≥ 1 and

This concludes the proof of Theorem 1.3.

D Proof of Comparison Complexity of Double-Hoover Sort

Lemma D.1. Upon the insertion of an item a i into L , all items in L \ L <i are larger than l i . Similarly, when an item a i is inserted into R , all items in R \ R >i are smaller than r i . As a result, the initial interval at line 8 and line 14 are subsets of L <i and R <i , and hence have sizes no larger than δ .

Proof. Assume a i is inserted into L in round δ . Consider any a j ∈ L \ L <i at the time of a i insertion. Then, a j was inserted into L previously with a smaller insertion strength δ ′ < δ . By the insert condition, a j > l j δ ′ .

Since L <i is a subset of L <j , the δ ′ th largest item of the latter set must exist and be no smaller than the δ ′ th largest items of the former set. Then, we obtain

Hence, the interval { x ∈ L | l i δ ≤ x ≤ l i } only contains items in L <i . Since there are δ items in L <i that are no smaller than l i δ , the interval has at most δ items. An analogous proof shows the symmetric property in R .

Theorem D.2. For each item a i , its insertion process takes O (log(min { η l i , η r i } +2)) comparisons.

Proof. Each item a i goes through some rejected insertions in earlier rounds, and then gets inserted into L or R in a certain round. We refer to them as the exploration phase and insertion phase , respectively, and prove that the number of comparisons needed in both phases is bounded by O (log(min { η l i , η r i } +2)) .

First, we claim that each a i is inserted into either L or R prior to or during the round with insertion strength δ l i := 2 ⌈ log( η l i +2) ⌉ . If a i is inserted before round δ l i , the claim trivially holds. The claim also holds if at round δ l i , L <i contains fewer than δ l i items. In the absence of these conditions,

Hence, in round δ l i , a i must be larger than the boundary value, which is the δ l i th largest item in L <i . Consequently, it will be inserted into L in round δ l i .

A similar argument shows that a i will be inserted prior to or during round 2 ⌈ log( η r i +2) ⌉ . Combining these two bounds, we find that a i must be inserted prior to or during round δ i := 2 ⌈ log(min { η l i ,η r i } +2) ⌉ . Hence, the exploration phase only needs O (log(min { η l i , η r i } +2)) comparisons.

Next, we continue to examine the number of comparisons needed in the insertion phase. Suppose a i is inserted into L in some round δ ≤ δ l i . Then, the binary search starts with an interval of size

The first inequality is due to Lemma 3.1. Hence, the insertion phase of a i by binary search needs O (log(min { η l i , η r i } +2)) comparisons.

E Lower Bounds on Comparison Complexity

We prove the lower bounds stated in Theorem 1.5.

E.1 Optimality of Displacement Sort

In this section, we show that an exact sorting algorithm, augmented with a positional prediction, cannot sort the array with o ( ∑ i ∈ [ n ] log( η ∆ i )) comparisons.

Definition E.1. Given n , a positional prediction ˆ p , and a real number U , define the size of the U -candidate set as

where S n is the set of permutations of [ n ] (viewed as an array), η ∆ is calculated accordingly for each A against ˆ p .

Theorem E.2. Given any function f ( U ) = o (max ˆ p ∈ S n log( cand (ˆ p, U ))) , there does not exist any positional augmented sorting algorithm with comparison complexity O ( f ( U )) for instances with ∑ n i =1 log( η ∆ i +2) ≤ U .

Proof. Given the predictor ˆ p and an upper bound U on the error, a sorting algorithm needs to determine the correct permutation A from a candidate set of size cand (ˆ p, U ) . If the algorithm only uses x comparisons, then it can only distinguish 2 x different outcomes; if 2 x is smaller than the number of candidates, it cannot determine the correct answer in every situation, since the information given by comparisons is not sufficient to distinguish all candidates. Hence, at least ⌈ log cand (ˆ p, U ) ⌉ comparisons are needed.

Theorem E.3. For all n ≤ U ≤ O ( n log n ) , we have max ˆ p ∈ S n log cand (ˆ p, U )) = Ω( U ) .

Proof. Proof by construction. Assume without loss of generality that U is a multiple of n and let U ′ = U/n . Take ˆ p = ⟨ 1 , 2 , . . . , n ⟩ , the identity prediction. We aim to construct sufficient number of candidate permutations A ∈ S n , which all fall in the U -candidate set.

Consider every permutation A ∈ S n constructed in the follow way: Initially, set A = ⟨ 1 , 2 , . . . , n ⟩ . Then, divide A into n 2 U ′ adjacent subarrays, each containing 2 U ′ items (the last subarray is potentially smaller if there are not enough remaining elements). Finally, we permute the items in each subarray arbitrarily, and retrieve A after the permutation.

In each possible outcome A , log η ∆ i ≤ U ′ for all i since each item is only permuted locally. Hence, the error of ˆ p w.r.t. A is no larger than n · U ′ = U . Each A falls inside the U -candidate set. Counting the number of possible different outcomes in our construction, we obtain

where the right-hand-side represents the possible ways to permute 2 U ′ items in each subarray. By Stirling's formula,

Directly combining Theorem E.2 and Theorem E.3, and noting that at least Ω( n ) comparisons are always needed to verify correctness of a sorted list, we obtain the following:

Corollary E.4. For sorting with positional predictions, there exists no algorithm with comparison complexity o ( ∑ n i =1 log( η ∆ i +2)) .

E.2 Optimality of Dirty Comparisons Sort and Double-Hoover Sort

We can use the same definition of cand (ˆ p, U ) and construction as above. Every positional prediction can be view as a total linear relation on array A , therefore induces a unique dirty-comparison predictor. Since each element is locally perturbed, we can prove that for each permutation A obtained from the construction, ∑ n i =1 log( η i +2) and ∑ n i =1 log(min { η l i , η r i } +2) are also upper bounded by U . Hence, the optimality of Algorithm 2 and 3 can be proven in the exact same way, and Theorem 1.5 follows.

F Extra Experimental Results

F.1 Synthetic Datasets with Varying Sizes

This section presents additional experimental results to provide a more comprehensive evaluation of our proposed algorithms. As scalability is a crucial aspect, we conducted further experiments on synthetic datasets with varying sizes from n = 1,000 to n = 1,000,000. Results are shown in Figure F.1, F.2, and F.3.

In these figures, the shaded area denotes the standard deviation of each run among 30 repetitions. As the results show, our proposed algorithms perform consistently with relatively small deviations.

F.2 Comparisons against Insertion Sort

The performance of Insertion Sort against our proposed algorithms is shown in Figure F.4. As the plots show, insertion sort runs fast for small errors in the decay and country settings, though the performance declines steeply when predictions deteriorate (i.e., smoothness is worse, and robustness is non-existent). The experimental results suggest that in practical settings, it is a good idea to incorporate insertion sort as a fall-back option in case predictions are extremely accurate.

comparisons / n

(a) class setting,

n

= 1

,

000

(c) class setting,

n

= 100

,

000

(d) class setting,

n

= 1

,

000

,

000

20.0

o w

Double-Hoover Sort

Double-Hoover Sort

n

17.5

OESM

=

25

OESM

comparisons

comparisons/

15.0

20

12.5

15

10.0

7.5

10

#

#

5.0

2.5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

#classes /n

#classes / n

40

50

MergeSort

comparisons

n

35

30

OESM

Double-Hoover Sort

comparisons

n

40

OESM

Double-Hoover Sort

25

30

20

20

15

#

10

#

10

5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

8'0

1.0

#classes/n

#classes/n

(c) decay setting,

n

= 100

,

000

12

# comparisons / r

n

10

Double-Hoover Sort

OESM

12

15

18

22

25

28

31

#timesteps/n

20.0

17.5

15.0

12.5

10.0

7.5

#

5.0

2.5

Double-Hoover Sort

OESM

0°0

0

31

9

6

126

158

189

221

252

284

316

#timesteps/n

16

n

14

# comparisons /

12

10

8

Double-Hoover Sort

OESM

10

20

30

40

50

60

70

80

90

100

#timesteps/ n

25

# comparisons / n

20

15

5

Double-Hoover Sort

OESM

0

0

100

200

300

400

500

600

700

800

900

1000

#timesteps/n

(a) good-dominating,

n

= 1

,

000

(b) good-dominating,

n

= 10

,

000

(c) good-dominating,

n

= 100

,

000

(d) bad-dominating,

n

= 1

,

000

(e) bad-dominating,

n

= 10

,

000

(f) bad-dominating,

n

= 100

,

000

20.0

OE

n

17.5

OESM

Dirty-Clean Sort

Double-Hoover Sort

n

25

OESM

Dirty-Clean Sort

Double-Hoover Sort

comparisons

15.0

comparisons,

20

12.5

10.0

15

7.5

101

#

5.0

#

2.5

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

damage ratio r

damage ratio r

40

QuickSort

DisplacementSort

20 -

n

o w

OESM

Dirty-Clean Sort

Double-Hoover Sort

/n

comparisons/

30

comparisons

15

20

10

#

10

#

o w

Double-Hoover Sort

OESM

Dirty-Clean Sort

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

damage ratio r

damage ratio r

30

40

35

n

25

n

# comparisons / I

# comparisons /

OE

20

25

15

20 -

15

10

10

Double-Hoover Sort

5

Double-Hoover Sort

OESM

Dirty-Clean Sort

OESM

Dirty-Clean Sort

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

damage ratior

damage ratior

Insertion Sort

Double-Hoover Sort

17.5

# comparisons / n

2000

#comparisons / n

15.0

5.0

00S

Insertion Sort

Double-Hoover Sort

0.2

#classes/n

0.4

0.6

0.8

200

300

#timesteps/n

400

500

600

700

1000

10

comparisons / n

#

Insertion Sort

Double-Hoover Sort

10

15

2025303540

50

55

60

gap in years

Insertion Sort

Double-Hoover Sort

Dirty-Clean Sort

comparisons/n

2000

#comparisons / n

00

1500

1000

#

500

Insertion Sort

Double-Hoover Sort

Dirty-Clean Sort

0.2

0.4

0.6

0.8

0'0

0.2

0.4

0.6

0.8

1.0

damageratior damageratior

100

Insertion Sort

Double-Hoover Sort

250

Insertion Sort

Double-Hoover Sort

250

Insertion Sort

Double-Hoover Sort

Dirty-Clean Sort

Dirty-Clean Sort

#comparisons

n

08

comparisons/

n

200

#comparisons/

n

200

09

150

150

40

100

100

#

20

50

50

0.0

#classes/n

0.4

0.6

8'0

1.0

0'0

0.2

damageratior

0.4

0.6

0.8

1.0

0.0

0.2

damageratior

0.4

0.6

0.8

1.0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents sorting algorithms augmented by prediction under two settings: (i) sorting with positional predictions and (ii) sorting with dirty and clean comparisons. In (i), the algorithm can use a predicted position $\\hat{p}(i)$ of each element $i$, and in (ii), the algorithm is provided a faster but possibly inaccurate comparison oracle besides a slow but accurate comparison. In both settings, the authors present provably faster algorithms making use of the positional predictions/dirty comparisons. They also show that the complexities of the presented algorithms are asymptotically optimal. The efficacy of the algorithms is validated through numerical experiments on synthetic and real-world datasets.\n\nSTRENGTHS:\nThe two settings introduced in this paper are well-grounded in real-world scenarios, demonstrating their relevance and applicability. The algorithms proposed are elegant and straightforward, contributing to their practical utility. I also find the lower-bound results commendable. The paper is well-composed, particularly the introduction, which provides an accessible entry point for readers unfamiliar with algorithm-with-predictions research. Therefore, my overall assessment leans towards a positive evaluation of this paper.\n\nWEAKNESSES:\n- The sorting algorithm with dirty comparisons is randomized and takes $O(n \\log n)$ comparisons, as explained in Section 5. The contribution of this paper will be stronger if the algorithm is derandomized and/or the worst-case time-complexity bound is improved to $O(\\sum_i \\log (\\eta_i^\\triangle + 2))$.\n- The algorithms cannot be implemented in the \"in-place manner\" because they use extra binary trees.\n\nQUESTIONS:\n- The height of a binary search tree can be deterministically maintained within a logarithmic order of #nodes by using the \"rotation\" operation. Can the presented algorithm with dirty comparisons also be derandomized by rotating the tree?\n- What happens if there are ties in the correct order of elements? Are your sorting algorithms work? If so, are they stable? Can you extend your result to the setting where the dirty comparison for two elements reports that \"they are in the same order\"?\n- The insertion sort runs fast if the array is almost sorted. Thus, in the positional prediction setting, the bucket sort (by the prediction) followed by the insertion sort seems to run fast. I'm interested in an experimental comparison.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning-augmented sorting, where an algorithm obtains access to a potentially erroneous prediction on the sorting of an array. The paper considers two prediction models. In the positional predictor, every item has a prediction for its place in the sorted array, and the error of item i is the number of indices its predicted location is from its true location in the sorted array. The also consider a one sided error model for that prediction. In sorting with dirty comparisons, every pair of items has a < or > relation between them. Note that the dirty comparisons do not have to satisfy transitivity, thus potentially giving rise to cycles. A query between two elements is called a clean comparison.\n\nThe authors design sorting algorithms that use these predictions, and the performance of the algorithms depends on the error in the prediction. The performance is measured by the number of comparisons. Two of the main results (given with matching lower bounds) are:\n- Given access to dirty comparisons, there is a randomized alg sorting in O(n log n) runtime, O(n log n) dirty comparisons, and O(sum_i log(eta_i + 2)) clean comparisons in expectation.\n- Given a positional predictor, there is an alg sorting in O(sum_i log(|p(i)-\\hat{p}_i|+ 2)) runtime and comparisons.\n\nSTRENGTHS:\n- The paper is well written, and I believe the theoretical results to be correct. \n- The experimental results sufficiently back the theory (for my taste). \n- Overall, I like the paper well enough. However, I’m concerned about how much new insight into sorting with predictions this is contributing, given that similar problems have been studied and the authors do not thoroughly compare to these prior works. Can the authors comment more on this?\n\nWEAKNESSES:\n- While the paper gives a nice background on learning-augmented algorithms, I think several key works are missing that need to be compared against. For example, I need a thorough comparison against The Case for a Learned Sorting Algorithm by Kristo et al. Also, the papers The Case for Learned Index Structures by Kraska et al and SageDB: a Learned Database System by Kraska et al should be at least mentioned. I would also like a few more sentences explaining why the Lu et al and Kristo et al algorithms are not amenable to this setting/ what the notable differences are.\n- From a cursory glance, I believe the paper from Kristo et al has analysis that depends on the total error of the indices, whereas the paper here considers a more fine-grained analysis by depending on each indices error. I don’t know how/ if the algorithms are different. Can the authors please comment more on this?\n\nQUESTIONS:\n(Please see questions in Weaknesses and Strengths)\n\nMany grammatical typos throughout, I highlights a few below but I recommend a full read through before submitting a CR to Neurips or submitting to another venue. Here are some things I noticed below:\n- (lines 15-17) The first sentence in the intro is oddly phrased, consider rewording.\n- (lines 24-38) Add citations for the examples of how learning-augmented sorting could be useful.\n- (line 41) In *the* dirty comparison setting…\n- (line 42) comparison *bounds* on the error…\n- (line 126-127) While *the* recurrent noisy… \n- (Lines 176-179) I think slightly more detail here could be added. The point is that with the dirty comparisons you end up without a total ordering, so the point is the go to the closest subtree that contains something breaking the total ordering and then fix that element’s position in this subtree? So this clean search is what will remove the cycles.\n- (Line 293) We design *the* decay setting.\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the oracle-augmented sorting problem, in which, in addition to the array, an oracle is provided to return ‘reference answers’ for the sorting of the array. The oracle is presumably efficient – queries are assumed to take $O(1)$ time – but might be noisy, i.e., at times return incorrect answers. The key question is to design algorithms that 1). When the oracle is great, the number of comparisons is significantly smaller than the baseline algorithms; and 2). When the oracle is very bad, the number of comparisons is still comparable to the baseline algorithms. \n\nTo this end, the paper considered oracles under two settings: a). An oracle that gives ‘dirty comparisons’ – a list of ‘all-pair’ comparisons that may include some errors and b). An oracle that gives a positional list – a list of predicted positions for each element in the array. The paper then designs algorithms under both models. In particular, for the dirty comparison model, the proposed algorithm achieves $O(\\sum_{i=1}^{n} \\log \\eta_{i})$ number of true comparisons and $O(n \\log n)$ dirty comparisons, where $\\eta_i$ is the number of ‘wrong comparisons’ based on element $a_i$. This demonstrates a smooth tradeoff between the number of true comparisons needed and the prediction error. The paper also obtained similar results for positional list predictions under different notions of error.\n\nI read the dirty-clean comparison algorithm and proofs to some details, and they look correct as far as I can tell. I skimmed the claims for the positional list algorithms, but did not verify the proofs (these algorithms seem to have built on more recent ideas). For the dirty-clean comparison, I think a key observation here is that the binary comparison trees are fairly ‘robust’ against perturbations in the sense that 1). The error is consistent – if there is $R<L$ on level $t$, then the error also happens for all $t’>t$ and 2). Thanks to randomization, the probability of the mess-up of the upper and lower bounds becomes exponentially small when the level moves up. As such, the binary search tree should be able to travel sufficiently deep in the tree based only on the dirty comparisons unless a very large chunk of the comparisons is corrupted. These observations, although not technically novel, do seem to be neat and cute for this problem.\n\nOverall I think the paper makes a significant contribution to a fundamental problem, and the setting is interesting and well-motivated. The paper is also well-written and easy to follow; the experiments also demonstrate competitive performance for the new algorithms. As such, I have a positive evaluation of the paper.\n\nSTRENGTHS:\nAs I mentioned in the general evaluation, I think the paper is in a nice shape both in terms of the contributions and the presentations. To elaborate a bit more:\n- The paper follows a recent line of work on ‘learning-augmented algorithm’/’algorithm with advice’, and it tackles a very important and fundamental problem. It is surprising that previous work has not considered the problem. \n- Although the techniques are not complicated, the algorithms are based on several neat observations. The upper bounds are also tight by matching the lower bounds (unfortunately deferred to the appendix).\n- In addition to the theoretical results, the experimental performance is also quite competitive – one interesting observation is that the dirty-clean sort remains very competitive to baseline even when the corruption rate approaches 1. Furthermore, the experiments are run on large scales, which verifies the efficiency also from the practical time side (not necessarily captured by the theoretical time complexity).\n\nWEAKNESSES:\n- One potential criticism of the work is that the techniques are not novel and perhaps easy to come up with. This is true for the dirty-clean sorting algorithm; and although I *personally* do *not* think this is an issue, this perspective is frequently flagged for theoretical work. The positional list with left-right errors seems to require more involved techniques, but I do not really like that definition of error. \n- It seems the two settings of the oracles in this paper are very natural. However, there could still be other settings that your model does not capture. For instance, one can define a dirty comparison oracle with amortized error only (across ${n \\choose 2}$ comparisons), and what will be the optimal bound therein? That being said, I don’t think it’s very fair to ask a paper to address all possible models, so I’m not letting this question affect my evaluation.\n- Some (minor) presentation issues: In the binary tree search subroutines of algorithm 1, the use of tuples makes the text slightly confusing. I struggled for a minute to parse the value assignment of tuples. Maybe you can say we use tuples to represent (lower-bound, pivot, upper-bound) before pointing the readers to the algorithm. Also, what does it mean for ‘return inorder traversal of B’? (I know what it means, but is this the best way to say it?)\n\nQUESTIONS:\n- In your clean-dirty comparison model, I think you have this ‘deterministic error’ that if the comparison between a and b is wrong, you cannot get the correct answer by querying it again. This crucially separates your model from the classical noisy comparison line of work (your ‘dirty perturbation’ is not random). If what I said is correct, I would suggest adding a discussion about this.\n- In your experiments, the dirty-clean sorting seems extremely robust even when the damage ratio approaches $1$. Is this an artifact of your experiment setting/your implementation, or is there a reason that the algorithm performs in such a nice way? (Note that your proof shows only *asymptotically* competitive, but it seems even the constant is very competitive when the damage ratio is high.)\n(More presentation-related)\n- Does the term BST on page 4 stand for ‘binary search tree’? The abbreviation was never mentioned before.\n- Line 185, it’s a bit weird to talk about ‘timestep’ here – I think you mean the recursion step.\n\nLIMITATIONS:\nMost of the limitations are discussed in the ‘weakness’ section. From a broader societal perspective, since the work is of theoretical nature, there is no immediate negative impact, and I could not foresee any (take it with a grain of salt since I do not have experience in ethical/societal review).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes algorithms for sorting a set of items when unreliable extra information is available in the form of positional predictions or comparisons. The paper proposes a randomized algorithm for the former and deterministic algorithms for the latter problem.\n\nSTRENGTHS:\nThe paper proposes interesting algorithms for sorting when additional unreliable information is available.\nThe algorithms are simple and intuitive and the analysis (though several ideas are standard) is effective.\n\nWEAKNESSES:\n- A potential weakness is the relevance to the learning (NeuRIPS) community. In my opinion, the paper is more suited to a theoretical computer science venue as opposed to an ML venue such as NeuRIPS. Would like to understand the author's view on this\n\nQUESTIONS:\n- Will it help to detect at some point in the algorithm that the dirty comparisons are going to hurt more than help i.e., if the algorithm can learn/find out that it will spend more comparisons (dirty + clean) than just the O(nlogn) clean comparisons, it might abandon the dirty comparisons and thus help in reducing the total number of comparisons. Is this possible in the suggested algorithms? Is there a learning component that is natural here? \n- Standard metrics for comparing two permutations include the Kendall-Tau distance (between permutations), pairwise disagreement error (permutation vs tournament), etc. It might be useful to state the results in terms of these metrics.\n- How do these algorithms change when one has a budget of clean comparisons to work with? \n- An important pairwise comparison setup in the machine learning context is when the comparisons are not deterministic but probabilistic - for instance when a clean comparison is made between items i and j, the result might be based on the toss of a coin with probability P_{ij} = s_i/ (s_i + s_j) where s_k is the score of item k (The Bradley-Terry-Luce model). Here, one needs to learn the score vector to be able to rank (or sort) the items. How easy/hard is it to extend the results in this paper where one needs to be able to learn these scores from dirty or clean comparisons - if yes, this will make the work much more relevant to the ML community.\n\nLIMITATIONS:\nListed above in the weakness section - the results are less suited to a ML venue in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}