SlowGuess/NeurIPS_2023_OCR huggingface dataset:
If the insert causes the density of  $P_{i}$  to go above  $1/2$ , we merge it with its sibling LLA. Specifically, let  $P_{p}$  be the parent of  $P_{i}$ ; it must currently be a potential LLA. We take all actual LLAs that are descendants of  $P_{p}$ , and merge them into a single actual LLA. Let  $S_{p}$  be the set of elements currently stored in the slots assigned to  $P_{p}$  in the learned LLA; thus,  $S_{p}$  consists exactly of the contents of all actual LLAs that are descendants of  $P_{p}$ . We call  $\mathrm{INIT}(S_{p})$  on  $P_{p}$ , after which  $P_{p}$  is an actual LLA; all of its descendants are no longer actual LLAs. Note that after a merge, the assigned ranks and assigned slots of the actual LLAs still partition  $\{1, \dots, n\}$  and  $\{1, \dots, m\}$  respectively.
3.2 Arbitrary Predictions
In this section, we analyze the learnedLLA when the input sequence and associated predictions can be arbitrary. The adversary chooses elements one by one from the universe  $U$ . For each element, the adversary assigns a predicted rank (which can be based on past insertions, past predictions, and even the learnedLLA algorithm being used) and inserts the element. We show that the learnedLLA achieves amortized cost  $O(C(\eta))$ , where  $\eta$  is the maximum error and  $C(\cdot)$  is the admissible cost function for the amortized cost of inserting into the black box LLA being used.
Our proof begins with several structural results, from which the analysis follows almost immediately.
First, we show that to determine the overall asymptotic cost it is sufficient to consider the set of actual LLAs after all inserts are completed—relabels during merges are a lower-order term.
Lemma 2. For any sequence of insertions, if  $\mathcal{P}_F$  is the set of actual LLAs after all operations are completed, and  $C(\cdot)$  is an admissible cost function for the LLAs, then the total number of element movements incurred by the learnedLLA is  $O\left(\sum_{P\in \mathcal{P}_F}|P|\cdot C(|P|)\right)$ .
Proof. For a LLA  $P \in \mathcal{P}_F$ , consider all potential LLAs that are descendants of  $P$  in  $T$ . Every  $P'$  that is ever an actual LLA must be a descendant of exactly one  $P \in \mathcal{P}_F$ .
First, we analyze the cost of all merges in descendants of  $P$ . The total capacity of all LLAs that are a descendant of  $P$  is  $|P| \log |P|$ . Since a merge operation has linear cost, the total number of element movements during all merges is  $O(|P| \log |P|)$ .
Now, we must bound the cost of all inserts to descendants of  $P$ . Consider all inserts into all LLAs  $P_{1}, P_{2}, \ldots, P_{d}$  that are a descendant of  $P$ . Let  $k_{i}$  be the number of inserts into some such LLA  $P_{i}$ ; the total cost of these inserts is  $O(k_{i}C(|P_{i}|)) \leq O(k_{i}C(|P|))$ . All inserts into descendants of  $P$  are ultimately stored in  $P$ ; thus,  $\sum_{i=1}^{g} k_{i} \leq 3|P|$ . Then the total cost of all inserts into LLAs that are a descendant of  $P$  is  $\sum_{i=1}^{g} O(k_{i}C(|P|)) = O(|P|C(|P|))$ .
Summing between the merge and insert cost, and using  $C(|P|) + \log |P| = O(C(|P|))$  because  $C(\cdot)$  is admissible, we obtain a total number of element movements of  $O\left(\sum_{P \in \mathcal{P}_F} |P| \cdot C(|P|) + |P| \log |P|\right) \leq O\left(\sum_{P \in \mathcal{P}_F} |P| \cdot C(|P|)\right)$ .
Lemma 3. If  $P$  is an actual LLA, then there exists an element  $x_{j}$  stored in  $P$  with  $\eta_{j} \geq |P| / 2$ .
Proof. Since  $P$  was formed by merging its two children  $P_{i}$  and  $P_{k}$  in  $T$ , the density of either  $P_{i}$  or  $P_{k}$  must have been at least  $1/2$ ; without loss of generality assume it was  $P_{i}$ .
Let  $r_1, r_1 + 1, \ldots, r_2$  be the sequence of  $|P_i|$  ranks assigned to  $P_i$ . Let  $x_s$  and  $x_\ell$  be the smallest and largest items in  $P_i$  respectively, with predicted ranks  $\widetilde{r_s}$  and  $\widetilde{r_\ell}$ . We must have  $\widetilde{r_s} \geq r_1$  since the predecessor of  $x_s$  is not in  $P_i$ ,  $x_s$  must be placed in the LLA whose assigned ranks contain  $\widetilde{r_s}$ , or in the LLA containing its successor (if  $\widetilde{r_s}$  is larger than any rank assigned to the LLA containing its successor). Similarly,  $\widetilde{r_\ell} \leq r_2$ . Therefore,  $\widetilde{r_\ell} - \widetilde{r_s} \leq |P_i| - 1$ . There are at least  $3|P_i|$  items in  $P_i$  as its density is at least  $1/2$ . Thus,  $r_\ell - r_s \geq 3|P_i| - 1$ . Thus, either  $|\widetilde{r_s} - r_s| \geq |P_i|$  or  $|\widetilde{r_\ell} - r_\ell| \geq |P_i|$ . Noting that  $|P_i| = |P|/2$ , either  $\eta_s \geq |P|/2$  or  $\eta_\ell \geq |P|/2$ .
Theorem 4. For any sequence of at most  $n$  insertions  $\sigma$  with maximum error  $\eta$ , the learned LLA using LLAs with admissible cost function  $C(\cdot)$  incurs  $O(nC(\eta))$  total element movements.
Proof. By Lemma 3, any  $P \in \mathcal{P}_F$  must have an element with error at least  $|P|/2$ . Thus, any  $P \in \mathcal{P}_F$  must have  $|P| \leq 2\eta$ . The actual LLAs partition the ranks, so  $\sum_{P \in \mathcal{P}_F} |P| = n$ . By Lemma 2, and since  $C(\cdot)$  is admissible, the total number of element movements over  $\sigma$  is  $O\left(\sum_{P \in \mathcal{P}_F} |P| \cdot C(|P|)\right) \leq O\left(\sum_{P \in \mathcal{P}_F} |P| \cdot C(2\eta)\right) \leq O(nC(\eta))$ .
Corollary 5. Using the classic PMA as an LLA [4, 30], the learnedLLA is deterministic and achieves  $O(\log^2 \eta)$  amortized element movements. Using the best-known randomized LLA [3], the learnedLLA achieves  $O(\log^{3/2} \eta)$  amortized element movements.
3.3 Optimality for Deterministic Learned List Labeling
The learnedLLA is clearly optimal on the two extremes: perfect predictions  $(\eta = 0)$  and completely erroneous predictions  $(\eta = n)$ . When predictions are perfect, the learnedLLA does not need to move any element. When  $\eta = n$ , it is easy to create instances where the predictions give no information about the rank of each item (e.g. if each prediction is  $\widetilde{r_i} = 1$ ). Bulánek et al. [10] showed that, without predictions, any deterministic LLA with  $O(n)$  slots can be forced to perform  $\Omega (\log^2 n)$  amortized element movements. Thus, the learnedLLA is optimal when  $\eta = n$ .
However, it is not clear if the above idea extends to intermediate error. In this section, we show that for deterministic LLAs, the learnedLLA is in fact optimal for all  $\eta$ .
The intuition behind our proof is to split the learned list labeling problem into a sequence of  $\Omega (n / \eta)$  subproblems of size  $\pmb{\eta}$ , each handled by its own LLA algorithm. The given predictions within each subproblem are only accurate to within  $\pmb{\eta}$ —therefore, an adversary can force each LLA to get no information about an element's position within the LLA, and by [10] perform no better than  $\Omega (\log^2\pmb {\eta})$  amortized element movements.
The challenge is that the LLAs for the subproblems are not actually separate: we can't force the data structure to allocate  $O(\eta)$  space to each LLA and keep this allocation static throughout the execution. The data structure may move elements between adjacent LLAs or shift the number of slots available to each, an ability that disrupts a black-box lower bound.
Lower bound summary. First, we define a new problem, shifted list labeling, which generalizes online list labeling to allow labels that "shift" over time rather than be confined to  $\{1,\dots ,m\}$ . We show that this generalization does not increase cost: a data structure for shifted list labeling implies a data structure for online list labeling with the same asymptotic space and amortized cost.
Then, we partition the universe  $U$  into  $n / \eta$  blocks. The adversary only inserts an element from a block if its elements are spread over at most  $O(\eta)$  slots. Thus, the labels of the elements inserted from a particular block form a solution to the shifted list labeling problem—this addresses the challenge of handling separate LLA subproblems. We apply the lower bound of [10] to items inserted from each block; therefore, any block from which  $\eta$  items are inserted must have  $\Omega (\eta \log^2\eta)$  amortized cost.
Finally, the adversary implements a cleanup step which inserts extra elements to ensure that the predictions of all elements are accurate to within  $\eta$ . We show that a constant fraction of the blocks had  $\eta$  elements inserted adversarially before the cleanup step. Each of these blocks must have had  $\Omega(\eta \log^2 \eta)$  element movements; summing we obtain the following theorem.
Theorem 6. For any  $n$  and  $\eta$ , there exists an adversary that inserts  $n$  items, each with error at most  $\eta$ , such that any deterministic data structure for learned list labeling with capacity  $n$  and  $m = cn$  slots incurs  $\Omega (n\log^2\eta)$  total element movements.
3.4 Stochastic Predictions with Adversarial Insert Order
In this section, we assume that each predicted rank is the result of adding random noise to the true rank. In particular, given an unknown distribution  $\mathcal{D}$ , for any item  $x_{i}$ , we have  $\widetilde{r_i} = r_i + e_i$  where each  $e_i$  is sampled independently from  $\mathcal{D}$ . This means that the error of each item is essentially sampled from  $\mathcal{D}$ , as  $\pmb{\eta}_i = |e_i|$ . The arrival order of the elements can still be chosen adversarially, with full knowledge of the predicted ranks. In particular, we can describe our input model with the following adversary. The adversary first chooses  $n$  elements of the set  $S$  from the universe  $U$ . For each element  $x_{i}\in S$ , the error  $e_i$  is then sampled from  $\mathcal{D}$ . Finally, the adversary can look at the predicted ranks and choose any insert order of elements in  $S$ . In this model, we prove the following.
Theorem 7. Consider a learnedLLA data structure using a black box LLA algorithm with admissible cost  $C(\cdot)$ . The total element movements incurred over a sequence of at most  $n$  insertions with stochastic predictions is  $O\left(n \cdot C((|\mu| + s^2)^2)\right)$ .
As in Corollary 5, we immediately obtain a deterministic learnedLLA with  $O(\log^2 (|\mu | + s))$  expected amortized cost, and a randomized learnedLLA with  $O(\log^{3 / 2}(|\mu | + s))$  expected amortized cost.
This section presents experimental results on real data sets. The goal is to show that the theory is predictive of practice. In particular, the aim is to establish the following.
- The learnedLLA improves performance over baseline data structures. Moreover, for the common use case of temporal data, learning from the past leads to future improvements.
- Predictions made on only a small amount of past data lead to improved performance.
- The learnedLLA is robust to large errors in the predictions.
Experimental setup. We compare the performance of two strong baselines with our algorithm. The first baseline is the Packed-Memory Array (PMA) [4] and the second is the Adaptive Packed-Memory Array (APMA) [7]. We tested LearnedLLA with both of these LLAs as the black box LLA; we call these the LearnedLLA + PMA and LearnedLLA + APMA. To ensure all algorithms have the same memory, we implemented the PMA (resp. APMA) as a LearnedLLA using a PMA (resp. APMA) as a black box, placing each element into the first black box LLA.
We use datasets from SNAP Large Network Dataset Collection [37]. All the datasets we use are temporal. The timestamp is used for the arrival order, and an element feature is used for the value.
To generate the predictions for LearnedLLA, we use a contiguous subsequence  $L_{\mathrm{train}}$  of the input in temporal order as our training data. Our test data  $L_{\mathrm{test}}$  is a contiguous subsequence of the input that comes right after  $L_{\mathrm{train}}$ , again in temporal order.
We use two different algorithms for obtaining predictions:
- predictor $_1$  ( $L_{\text{train}}, L_{\text{test}}$ ): For each element  $x \in L_{\text{test}}$ , this function first finds the rank of  $x$  in  $L_{\text{train}}$ , and then it scales it by  $|L_{\text{test}}| / |L_{\text{train}}|$ . Finally, it returns these predictions.
- predictor $_2(L_{train}, L_{test})$ : Let  $a$  be the slope of the best-fit line for points  $\{(i, L_{train}[i])\}_{1 \leq i \leq |L_{train}|}$ . First, this function adds  $a \cdot (d + i \cdot (\frac{|L_{test}|}{|L_{train}|} - 1))$  to the  $i$ 'th element in  $L_{train}$ , for each  $1 \leq i \leq |L_{train}|$ , to obtain  $L_{train}'$ , where  $d$  is the difference between the starting points of training and test data in our input sequence. Then it returns predictor $_1(L_{train}', L_{test})$
Let  $L_{\mathrm{train}}^{1}$  and  $L_{\mathrm{train}}^{2}$  be the first and second halves of  $L_{\mathrm{train}}$ , respectively. To obtain the final predictions for LearnedLLA, we calculate the predictions  $P_{i} \coloneqq \mathrm{predictor}_{i}(L_{\mathrm{train}}^{1}, L_{\mathrm{train}}^{2})$  for  $i = 1, 2$ ,
and run LearnedLLA on  $L_{\text{train}}^2$  with both of these predictions. If  $P_{i^*}$  performs better, we use predictor  $_{i^*}(L_{\text{train}}, L_{\text{test}})$  as the final predictions for  $L_{\text{test}}$ .
Experimental results. See Table 1 for performance on several real data sets. Due to limited space, we only include plots on one data set: Gowalla [14], a location-based social networking website where users share their locations by checking in. The latitudes of each user are used as elements in the input sequence. The remaining plots are given in the supplementary materials and the full version of the paper.
Gowalla (LocationID)	Gowalla (Latitude)	MOOC	AskUbuntu	email-Eu-core
PMA	7.14	14.56	19.22	24.56	21.49
APMA	7.38	15.63	16.70	10.84	21.43
LearnedLLA + PMA	3.36	6.06	11.99	14.27	16.55
LearnedLLA + APMA	3.36	6.15	12.13	8.49	16.55
Table 1: Amortized cost of LLAs on several real datasets. In all cases, we use the first and second  $2^{17} = 131072$  entries as training and test data, respectively.
Figure 1a shows the amortized cost versus the test data size. For several values for  $k$ , we use the first and second  $n = 2^k$  portions of the input as training data and test data, respectively. Figure 1b shows the performance versus the training data size to illustrate how long it takes for the LearnedLLA to learn. The x-axis is the ratio of the training data size to the test data size (in percentage). Figure 1c is a robustness experiment showing performance versus the noise added to predictions, using half of the data as training. In this experiment, we first generate predictions by the algorithm described above, and then we sample  $t$  percent of the predictions uniformly at random and make their error as large as possible (the predicted rank is modified to 1 or  $n$ , whichever is farthest from the current calculated rank). We repeat the experiment five times, each time resampling the dataset, and report the mean and standard deviation of the results of these experiments.
(b)
Figure 1: Latitudes in the Gowalla dataset. In Fig. 1a, we use the first and second  $n = 2^k$  entries as training data and test data (resp.) for several values of  $k$ . In Fig. 1b, the test data has a fixed size  $n = 2^{17}$  elements, and we increase the size of the training data. In Fig. 1c, training data and test data have size  $2^{16}$ , and we increase the percentage of "bad" predictions. The lines are the means of the 5 experiments, and the clouds around them show the standard deviation.
Discussion. Results in Table 1 show that in most cases, using even a simple algorithm to predict ranks can lead to significant improvements in the performance over the baselines; in some cases by over  $50\%$ . Whenever the APMA outperforms the PMA, the learnedLLA inherits this performance gain when using the APMA as a black box, showing its versatility. Furthermore, as illustrated in Figure 1b, a small amount of past data—in some cases as small as  $5\%$ —is needed to see a significant separation between the performance of our method and the baseline LLAs. Finally, Figure 1c suggests that our algorithm is robust to bad predictions. In particular, in this experiment, the maximum error is as large as possible and a significant fraction of the input have errors in their ranks, yet the LearnedLLA is still able to improve over baseline performance. We remark that when an enormous number of predictions are completely erroneous, the method can have performance worse than baselines.
In the supplementary materials and the full version of the paper, we include more details about the above experiments and datasets, as well as more experiments which further support our conclusions.
In this paper, we show how to use learned predictions to build a learned list labeling array. We analyze our data structure using the learning-augmented algorithms model. This is the first application of the model to bound the theoretical performance of a data structure. We show that the new data structure optimally makes use of the predictions. Moreover, our experiments establish that the theory is predictive of practical performance on real data.
An exciting line of work is to determine what other data structures can have improved theoretical performance using predictions. A feature of the list labeling problem that makes it amenable to the learning-augmented algorithms model is that its cost function and online nature is similar to the competitive analysis model, where predictions have been applied successfully to many problems. Other data structure problems with similar structure are natural candidates to consider.
Samuel McCauley was supported in part by NSF CCF 2103813. Benjamin Moseley was supported in part by a Google Research Award, an Infor Research Award, a Carnegie Bosch Junior Faculty Chair, National Science Foundation grants CCF-2121744 and CCF-1845146 and U. S. Office of Naval Research grant N00014-22-1-2702. Aidin Niaparast was supported in part by U. S. Office of Naval Research under award number N00014-21-1-2243 and the Air Force Office of Scientific Research under award number FA9550-20-1-0080. Shikha Singh was supported in part by NSF CCF 1947789.
[1] Michael A. Bender, Jonathan W. Berry, Rob Johnson, Thomas M. Kroeger, Samuel McCauley, Cynthia A. Phillips, Bertrand Simon, Shikha Singh, and David Zage. Anti-persistence on persistent storage: History-independent sparse tables and dictionaries. In Proc. 35th ACM Symposium on Principles of Database Systems (PODS), pages 289-302, 2016.
[2] Michael A Bender, Richard Cole, Erik D Demaine, Martin Farach-Colton, and Jack Zito. Two simplified algorithms for maintaining order in a list. In Proc. 10th Annual European Symposium on Algorithms (ESA), pages 152-164. Springer, 2002.
[3] Michael A Bender, Alex Conway, Martin Farach-Colton, Hanna Komlós, William Kuszmaul, and Nicole Wein. Online list labeling: Breaking the  $\log^{3/2} n$  barrier. In Proc. 63rd Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 980-990. IEEE, 2022.
[4] Michael A Bender, Erik D Demaine, and Martin Farach-Colton. Cache-oblivious B-trees. In Proc. 41st Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 399-409, 2000.
[5] Michael A Bender, Martin Farach-Colton, and Miguel A Mosteiro. Insertion sort is  $\mathrm{O}(\mathrm{n}\log \mathrm{n})$ . Theory of Computing Systems (TOCS), 39:391-397, 2006.
[6] Michael A Bender, Jeremy T Fineman, Seth Gilbert, and Bradley C Kuszmaul. Concurrent cache-oblivious B-trees. In Proc. of the 17th Annual ACM Symposium on Parallelism in Algorithms and Architectures (SPAA), pages 228-237, 2005.
[7] Michael A Bender and Haodong Hu. An adaptive packed-memory array. ACM Transactions on Database Systems (TODS), 32(4):26:1-26:43, 2007.
[8] Ioana O Bercea, Jakob Bæk Tejs Houen, and Rasmus Pagh. Daisy bloom filters. arXiv preprint arXiv:2205.14894, 2022.
[9] Antonio Boffa, Paolo Ferragina, and Giorgio Vinciguerra. A learned approach to design compressed rank/select data structures. ACM Transactions on Algorithms (TALG), 18(3):1-28, 2022.
[10] Jan Bulánek, Michal Koucký, and Michael Saks. Tight lower bounds for the online labeling problem. In Proc. 44th Annual ACM Symposium on Theory of Computing (STOC), pages 1185-1198, 2012.
[11] Jan Bulánek, Michal Koucký, and Michael Saks. On randomized online labeling with polynomially many labels. In Proc. 40th International Colloquium Automata, Languages, and Programming (ICALP), pages 291-302. Springer, 2013.
[52] Dan E Willard. Maintaining dense sequential files in a dynamic environment. In Proc. 14th Annual ACM Symposium on Theory of Computing (STOC), pages 114-121, 1982.
[53] Dan E Willard. Good worst-case algorithms for inserting and deleting records in dense sequential files. In ACM SIGMOD Record, volume 15:2, pages 251-260, 1986.
[54] Dan E Willard. A density control algorithm for doing insertions and deletions in a sequentially ordered file in a good worst-case time. Information and Computation, 97(2):150-204, 1992.
[55] Jiacheng Wu, Yong Zhang, Shimin Chen, Jin Wang, Yu Chen, and Chunxiao Xing. Updatable learned index with precise positions. Proceedings of the VLDB Endowment, 14(8):1276-1288, 2021.