SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Atsuki Sato Yusuke Matsui
a_sato@hal.t.u-tokyo.ac.jp matsui@hal.t.u-tokyo.ac.jp
A Bloom filter is a memory-efficient data structure for approximate membership queries used in numerous fields of computer science. Recently, learned Bloom filters that achieve better memory efficiency using machine learning models have attracted attention. One such filter, the partitioned learned Bloom filter (PLBF), achieves excellent memory efficiency. However, PLBF requires a  $\mathcal{O}(N^3 k)$  time complexity to construct the data structure, where  $N$  and  $k$  are the hyperparameters of PLBF. One can improve memory efficiency by increasing  $N$ , but the construction time becomes extremely long. Thus, we propose two methods that can reduce the construction time while maintaining the memory efficiency of PLBF. First, we propose fast PLBF, which can construct the same data structure as PLBF with a smaller time complexity  $\mathcal{O}(N^2 k)$ . Second, we propose fast PLBF++, which can construct the data structure with even smaller time complexity  $\mathcal{O}(Nk\log N + Nk^2)$ . Fast PLBF++ does not necessarily construct the same data structure as PLBF. Still, it is almost as memory efficient as PLBF, and it is proved that fast PLBF++ has the same data structure as PLBF when the distribution satisfies a certain constraint. Our experimental results from real-world datasets show that (i) fast PLBF and fast PLBF++ can construct the data structure up to 233 and 761 times faster than PLBF, (ii) fast PLBF can achieve the same memory efficiency as PLBF, and (iii) fast PLBF++ can achieve almost the same memory efficiency as PLBF. The codes are available at https://github.com/atsukisato/FastPLBF.
Membership query is a problem of determining whether a given query  $q$  is contained within a set  $S$ . Membership query is widely used in numerous areas, including networks and databases. One can correctly answer the membership query by keeping the set  $S$  and checking whether it contains  $q$ . However, this approach is memory intensive because we need to maintain  $S$ .
A Bloom filter [1] is a memory-efficient data structure that answers approximate membership queries. A Bloom filter uses hash functions to compress the set into a bitstring and then answers the queries using the bitstring. When a Bloom filter answers  $q \in S$ , it can be wrong (false positives can arise); when it answers  $q \notin S$ , it is always correct (no false negatives arise). A Bloom filter has been incorporated into numerous systems, such as networks [2, 3, 4], databases [5, 6, 7], and cryptocurrencies [8, 9].
A traditional Bloom filter does not care about the distribution of the set or the queries. Therefore, even if the distribution has a characteristic structure, a Bloom filter cannot take advantage of it. Kraska et al. proposed learned Bloom filters (LBFs), which utilize the distribution structure using a machine learning model [10]. LBF achieves a better trade-off between memory usage and false positive rate (FPR) than the original Bloom filter. Since then, several studies have been conducted on various LBFs [11, 12, 13].
Figure 1: PLBF partitions the score space into  $k$  regions and assigns backup Bloom filters with different FPRs to each region.
Figure 2: PLBF divides the score space into  $N$  segments and then clusters the  $N$  segments into  $k$  regions. PLBF uses dynamic programming to find the optimal way to cluster segments into regions.
Partitioned learned Bloom filter (PLBF) [14] is a variant of the LBF. PLBF effectively uses the distribution and is currently one of the most memory-efficient LBFs. To construct PLBF, a machine learning model is trained to predict whether the input is included in the set. For a given element  $x$ , the machine learning model outputs a score  $s(x) \in [0, 1]$ , indicating the probability that  $S$  includes  $x$ . PLBF divides the score space  $[0, 1]$  into  $N$  equal segments and then appropriately clusters the  $N$  segments into  $k (< N)$  regions using dynamic programming (DP). Then,  $k$  backup Bloom filters with different FPRs are assigned to each region. We can obtain a better (closer to optimal) data structure with a larger  $N$ . However, PLBF builds DP tables  $\mathcal{O}(N)$  times, and each DP table requires  $\mathcal{O}(N^2 k)$  time complexity to build, so the total time complexity amounts to  $\mathcal{O}(N^3 k)$ . Therefore, the construction time increases rapidly as  $N$  increases.
We propose fast PLBF, which is constructed much faster than PLBF. Fast PLBF constructs the same data structure as PLBF but with  $\mathcal{O}(N^2 k)$  time complexity by omitting the redundant construction of DP tables. Furthermore, we propose fast PLBF++, which can be constructed faster than fast PLBF, with a time complexity of  $\mathcal{O}(Nk\log N + Nk^2)$ . Fast PLBF++ accelerates DP table construction by taking advantage of a characteristic that DP tables often have. We proved that fast PLBF++ constructs the same data structure as PLBF when the probability of  $x\in S$  and the score  $s(x)$  are ideally well correlated. Our contributions can be summarized as follows.
- We propose fast PLBF, which can construct the same data structure as PLBF with less time complexity.
- We propose fast PLBF++, which can construct the data structure with even less time complexity than fast PLBF. Fast PLBF++ constructs the same data structure as PLBF when the probability of  $x \in S$  and the score  $s(x)$  are ideally well correlated.
- Experimental results show that fast PLBF can construct the data structure up to 233 times faster than PLBF and achieves the same memory efficiency as PLBF.
- Experimental results show that fast PLBF++ can construct the data structure up to 761 times faster than PLBF and achieves almost the same memory efficiency as PLBF.
2 Preliminaries: PLBF
First, we define terms to describe PLBF [14]. Let  $S$  be a set of elements for which the Bloom filter is to be built, and let  $\mathcal{Q}$  be the set of elements not included in  $S$  that is used when constructing PLBF  $(S \cap \mathcal{Q} = \emptyset)$ . The elements included in  $S$  are called keys, and those not included are called nonkeys. To build PLBF, a machine learning model is trained to predict whether a given element  $x$  is included in the set  $S$  or  $\mathcal{Q}$ . For a given element  $x$ , the machine learning model outputs a score  $s(x) \in [0, 1]$ . The score  $s(x)$  indicates "how likely is  $x$  to be included in the set  $S$ ".
Next, we explain the design of PLBF. PLBF partitions the score space  $[0,1]$  into  $k$  regions and assigns backup Bloom filters with different FPRs to each region (Figure 1). Given a target overall FPR,
$F \in (0,1)$ , we optimize  $\pmb{t} \in \mathbb{R}^{k + 1}$  and  $\pmb{f} \in \mathbb{R}^k$  to minimize the total memory usage. Here,  $\pmb{t}$  is a vector of thresholds for partitioning the score space into  $k$  regions, and  $\pmb{f}$  is a vector of FPRs for each region, satisfying  $t_0 = 0 < t_1 < \dots < t_k = 1$  and  $0 < f_{i} \leq 1$  ( $i = 1 \dots k$ ).
Next, we explain how PLBF finds the optimal  $\pmb{t}$  and  $\pmb{f}$ . PLBF divides the score space  $[0,1]$  into  $N(>k)$  segments and then finds the optimal  $\pmb{t}$  and  $\pmb{f}$  using DP. Deciding how to cluster  $N$  segments into  $k$  consecutive regions corresponds to determining the threshold  $\pmb{t}$  (Figure 2). We denote the probabilities that the key and non-key scores are contained in the  $i$ -th segment by  $g_{i}$  and  $h_{i}$ , respectively. After we cluster the segments (i.e., determine the thresholds  $\pmb{t}$ ), we denote the probabilities that the key and non-key scores are contained in the  $i$ -th region by  $G_{i}$  and  $H_{i}$ , respectively (e.g.,  $g_{1} + g_{2} + g_{3} = G_{1}$  in Figure 2).
We can find the thresholds  $t$  that minimize memory usage by solving the following problem for each  $j = k \dots N$  (see the appendix for details); we find a way to cluster the 1st to  $(j - 1)$ -th segments into  $k - 1$  regions while maximizing
$$
\sum_ {i = 1} ^ {k - 1} G _ {i} \log_ {2} \left(\frac {G _ {i}}{H _ {i}}\right). \tag {1}
$$
PLBF solves this problem by building a  $j \times k$  DP table  $\mathrm{DP}_{\mathrm{KL}}^j[p][q]$  ( $p = 0 \ldots j - 1$  and  $q = 0 \ldots k - 1$ ) for each  $j$ .  $\mathrm{DP}_{\mathrm{KL}}^j[p][q]$  denotes the maximum value of  $\sum_{i=1}^{q} G_i \log_2\left(\frac{G_i}{H_i}\right)$  one can get when you cluster the 1st to  $p$ -th segments into  $q$  regions. To construct PLBF, one must find a clustering method that achieves  $\mathrm{DP}_{\mathrm{KL}}^j[j - 1][k - 1]$ .  $\mathrm{DP}_{\mathrm{KL}}^j$  can be computed recursively as follows:
$$
\mathrm {D P} _ {\mathrm {K L}} ^ {j} [ p ] [ q ] = \left\{ \begin{array}{l l} 0 & (p = 0 \wedge q = 0) \\ - \infty & ((p = 0 \wedge q > 0) \vee (p > 0 \wedge q = 0)) \\ \max  _ {i = 1 \dots p} \left(\mathrm {D P} _ {\mathrm {K L}} ^ {j} [ i - 1 ] [ q - 1 ] + d _ {\mathrm {K L}} (i, p)\right) & (\text {e l s e}), \end{array} \right. \tag {2}
$$
where the function  $d_{\mathrm{KL}}(i_l,i_r)$  is the following function defined for integers  $i_l$  and  $i_r$  satisfying  $1\leq i_l\leq i_r\leq N$ :
$$
d _ {\mathrm {K L}} \left(i _ {l}, i _ {r}\right) = \left(\sum_ {i = i _ {l}} ^ {i _ {r}} g _ {i}\right) \log_ {2} \left(\frac {\sum_ {i = i _ {l}} ^ {i _ {r}} g _ {i}}{\sum_ {i = i _ {l}} ^ {i _ {r}} h _ {i}}\right). \tag {3}
$$
The time complexity to construct this DP table is  $\mathcal{O}(j^2 k)$ . Then, by tracing the recorded transitions backward from  $\mathrm{DP}_{\mathrm{KL}}^j [j - 1][k - 1]$ , we obtain the best clustering with a time complexity of  $\mathcal{O}(k)$ . As the DP table is constructed for each  $j = k\dots N$ , the overall complexity is  $\mathcal{O}(N^3 k)$ . The pseudo-code for PLBF construction is provided in the appendix.
We can divide the score space more finely with a larger  $N$  and thus obtain a near-optimal  $\pmb{t}$ . However, the time complexity increases rapidly with increasing  $N$ .
3 Fast PLBF
We propose fast PLBF, which constructs the same data structure as PLBF more quickly than PLBF by omitting the redundant construction of DP tables. Fast PLBF uses the same design as PLBF and finds the best clustering (i.e.,  $t$ ) and FPRs (i.e.,  $f$ ) to minimize memory usage.
PLBF constructs a DP table for each  $j = k \dots N$ . We found that this computation is redundant and that we can also use the last DP table  $\mathrm{DP}_{\mathrm{KL}}^N$  for  $j = k \dots N - 1$ . This is because the maximum value of  $\sum_{i=1}^{k-1} G_i \log_2\left(\frac{G_i}{H_i}\right)$  when clustering the 1st to  $(j-1)$ -th segments into  $k-1$  regions is equal to  $\mathrm{DP}_{\mathrm{KL}}^N[j-1][k-1]$ . We can obtain the best clustering by tracing the transitions backward from  $\mathrm{DP}_{\mathrm{KL}}^N[j-1][k-1]$ . The time complexity of tracing the transitions is  $\mathcal{O}(k)$ , which is faster than constructing the DP table.
This is a simple method, but it is a method that only becomes apparent after some organization on the optimization problem of PLBF. PLBF solves  $N - k + 1$  optimization problems separately, i.e., for each of  $j = k,\dots ,N$ , the problem of "finding the optimal  $t$  and  $f$  when the  $k$ -th region consists of
Figure 3: The cases (a) where there is no "crossing" of the recorded transitions when computing  $\mathrm{DP}_{\mathrm{KL}}[p][q](p = 1\ldots N - 1)$  from  $\mathrm{DP}_{\mathrm{KL}}[p][q - 1](p = 0\ldots N - 2)$  and (b) where there is. The arrows indicate which transition took the maximum value. Empirically, the probability of "crossing" is small.
Figure 4: Computing  $\mathrm{DP}_{\mathrm{KL}}[p][q]$  ( $p = 1 \dots N - 1$ ) from  $\mathrm{DP}_{\mathrm{KL}}[p][q - 1]$  ( $p = 0 \dots N - 2$ ) via the matrix  $A$ . The computation is the same as solving the matrix problem for matrix  $A$ . When the score distribution is ideal,  $A$  is a monotone matrix.
Figure 5: Example of a matrix problem for a  $4 \times 5$  monotone matrix. The position of the maximum value in each row (filled in gray) is moved to the "lower right".
the  $j,\ldots ,N$  th segments" is solved separately. Fast PLBF, on the other hand, solves the problems for  $j = k,\dots ,N - 1$  faster by reusing the computations in the problem for  $j = N$ . The reorganization of the problem in the appendix makes it clear that this reuse does not change the answer.
The pseudo-code for fast PLBF construction is provided in the appendix. The time complexity of building  $\mathrm{DP}_{\mathrm{KL}}^N$  is  $\mathcal{O}(N^2 k)$ , and the worst-case complexity of subsequent computations is  $\mathcal{O}(Nk^2)$ . Because  $N > k$ , the total complexity is  $\mathcal{O}(N^2 k)$ , which is faster than  $\mathcal{O}(N^3 k)$  for PLBF, although fast PLBF constructs the same data structure as PLBF.
Fast PLBF extends the usability of PLBF. In any application of PLBF, fast PLBF can be used instead of PLBF because fast PLBF can be constructed quickly without losing the accuracy of PLBF. Fast PLBF has applications in a wide range of computing areas [15, 5], and is significantly superior in applications where construction is frequently performed. Fast PLBF also has the advantage of simplifying hyperparameter settings. When using PLBF, the hyperparameters  $N$  and  $k$  must be carefully determined, considering the trade-off between accuracy and construction speed. With fast PLBF, on the other hand, it is easy to determine the appropriate hyperparameters because the construction time is short enough, even if  $N$  and  $k$  are set somewhat large.
4 Fast PLBF++
We propose fast PLBF++, which can be constructed even faster than fast PLBF. Fast PLBF++ accelerates the construction of the DP table  $\mathrm{DP}_{\mathrm{KL}}^N$  by taking advantage of a characteristic that DP tables often have. When the transitions recorded in computing  $\mathrm{DP}_{\mathrm{KL}}^N[p][q] (p = 1 \dots N - 1)$  from  $\mathrm{DP}_{\mathrm{KL}}^N[p][q - 1] (p = 0 \dots N - 2)$  are represented by arrows, as in Figure 3, we find that the arrows rarely "cross" (at locations other than endpoints). In other words, the transitions tend to have few intersections (Figure 3(a)) rather than many (Figure 3(b)). Fast PLBF++ takes advantage of this characteristic to construct the DP table with less complexity  $\mathcal{O}(Nk\log N)$ , thereby reducing the total construction complexity to  $\mathcal{O}(Nk\log N + Nk^2)$ .
Figure 6: A divide-and-conquer algorithm for solving a matrix problem for a monotone matrix: (a) An exhaustive search is performed on the middle row, and (b) the result is used to narrow down the search area. (c) This is repeated recursively.
(a)
(b)
Figure 7: When the number of regions is fixed at  $q$  and the number of segments is increased by 1, the optimal  $t_{q-1}$  remains unchanged or increases if  $A$  is a monotone matrix (a). When  $A$  is not a monotone matrix (b), the optimal  $t_{q-1}$  may decrease.
First, we define the terms to describe fast PLBF++. For simplicity,  $\mathrm{DP}_{\mathrm{KL}}^N$  is denoted as  $\mathrm{DP}_{\mathrm{KL}}$  in this section. The  $(N - 1) \times (N - 1)$  matrix  $A$  is defined as follows:
$$
A _ {p i} = \left\{ \begin{array}{l l} - \infty & (i = p + 1, p + 2, \dots , N - 1) \\ \mathrm {D P} _ {\mathrm {K L}} [ i - 1 ] [ q - 1 ] + d _ {\mathrm {K L}} (i, p) & (\text {e l s e}). \end{array} \right. \tag {4}
$$
Then, from the definition of  $\mathrm{DP}_{\mathrm{KL}}$
$$
\mathrm {D P} _ {\mathrm {K L}} [ p ] [ q ] = \max  _ {i = 1 \dots N - 1} A _ {p i}. \tag {5}
$$
The matrix  $A$  represents the intermediate calculations involved in determining  $\mathrm{DP}_{\mathrm{KL}}[p][q]$  from  $\mathrm{DP}_{\mathrm{KL}}[p][q - 1]$  (Figure 4).
Following Aggarwal et al. [16], we define the monotone matrix and matrix problem as follows.
Definition 4.1. Let  $B$  be an  $n \times m$  real matrix, and we define a function  $J: \{1 \dots n\} \to \{1 \dots m\}$ , where  $J(i)$  is the  $j \in \{1 \dots m\}$  such that  $B_{ij}$  is the maximum value of the  $i$ -th row of  $B$ . If there is more than one such  $j$ , let  $J(i)$  be the smallest. A matrix  $B$  is called a monotone matrix if  $J(i_1) \leq J(i_2)$  for any  $i_1$  and  $i_2$  that satisfy  $1 \leq i_1 < i_2 \leq n$ . Finding the maximum value of each row of a matrix is called a matrix problem.
An example of a matrix problem for a monotone matrix is shown in Figure 5. Solving the matrix problem for a general  $n \times m$  matrix requires  $\mathcal{O}(nm)$  time complexity because all matrix values must be checked. Meanwhile, if the matrix is known to be a monotone matrix, the matrix problem for this matrix can be solved with a time complexity of  $\mathcal{O}(n + m\log n)$  using the divide-and-conquer algorithm [16]. Here, the exhaustive search for the middle row and the refinement of the search range are repeated recursively (Figure 6).
We also define an ideal score distribution as follows.
Definition 4.2. A score distribution is ideal if the following holds:
$$
\frac {g _ {1}}{h _ {1}} \leq \frac {g _ {2}}{h _ {2}} \leq \dots \leq \frac {g _ {N}}{h _ {N}}. \tag {6}
$$
An ideal score distribution implies that the probability of  $x \in S$  and the score  $s(x)$  are ideally well correlated. In other words, an ideal score distribution means that the machine learning model learns the distribution ideally.
"Few crossing transitions" in Figure 3 indicates that  $A$  is a monotone matrix or is close to it. It is somewhat intuitive that  $A$  is a monotone matrix or is close to it. This is because the fact that  $A$  is a monotone matrix implies that the optimal  $t_{q-1}$  does not decrease when the number of regions is fixed at  $q$  and the number of segments increases by 1 (Figure 7). It is intuitively more likely that  $t_{q-1}$  remains unchanged or increases, as in Figure 7(a), than that  $t_{q-1}$  decreases, as in Figure 7(b). Fast PLBF++ takes advantage of this insight to rapidly construct DP tables.
From Equation (5), determining  $\mathrm{DP}_{\mathrm{KL}}[p][q]$  ( $p = 1 \dots N - 1$ ) from  $\mathrm{DP}_{\mathrm{KL}}[p][q - 1]$  ( $p = 0 \dots N - 2$ ) is equivalent to solving the matrix problem of matrix  $A$ . When  $A$  is a monotone matrix, the divide-and-conquer algorithm can solve this problem with  $\mathcal{O}(N \log N)$  time complexity. (The same algorithm
(a) Malicious URLs Dataset
(b) EMBER Dataset
Figure 8: Histograms of the score distributions of keys and nonkeys.
(a) Malicious URLs Dataset
Figure 9: Ratio of keys to nonkeys.
can obtain a not necessarily correct solution even if  $A$  is not a monotone matrix.) By computing  $\mathrm{DP}_{\mathrm{KL}}[p][q]$  with this algorithm sequentially for  $q = 1 \dots k - 1$ , we can construct a DP table with a time complexity of  $\mathcal{O}(Nk\log N)$ . The worst-case complexity of the subsequent calculations is  $\mathcal{O}(Nk^2)$ , so the total complexity is  $\mathcal{O}(Nk\log N + Nk^2)$ . This is the fast PLBF++ construction algorithm, and this is faster than fast PLBF, which requires  $\mathcal{O}(N^2k)$  computations.
Fast PLBF++ does not necessarily have the same data structure as PLBF because  $A$  is not necessarily a monotone matrix. However, as the following theorem shows, we can prove that  $A$  is a monotone matrix under certain conditions.
Theorem 4.3. If the score distribution is ideal,  $A$  is a monotone matrix.
The proof is given in the appendix. When the distribution is not ideal, matrix  $A$  is not necessarily a monotone matrix, but as mentioned above, it is somewhat intuitive that  $A$  is close to a monotone matrix. In addition, as will be shown in the next section, experiment results from real-world datasets whose distribution is not ideal show that fast PLBF++ is almost as memory efficient as PLBF.
This section evaluates the experimental performance of fast PLBF and fast PLBF++. We compared the performances of fast PLBF and fast PLBF++ with four baselines: Bloom filter [1], Ada-BF [11], sandwiched learned Bloom filter (sandwiched LBF) [12], and PLBF [14]. Similar to PLBF, Ada-BF is an LBF that partitions the score space into several regions and assigns different FPRs to each region. However, Ada-BF relies heavily on heuristics for clustering and assigning FPRs. Sandwiched LBF is an LBF that "sandwiches" a machine learning model with two Bloom filters. This achieves better memory efficiency than the original LBF by optimizing the size of two Bloom filters.
To facilitate the comparison of different methods or hyperparameters results, we have slightly modified the original PLBF framework. The original PLBF was designed to minimize memory usage under the condition of a given false positive rate. However, this approach makes it difficult to compare the results of different methods or hyperparameters. This is because both the false positive rate at test time and the memory usage vary depending on the method and hyperparameters, which often makes it difficult to determine the superiority of the results. Therefore, in our experiments, we used a framework where the expected false positive rate is minimized under the condition of memory usage. This approach makes it easy to obtain two results with the same memory usage and compare them by
(b) EMBER Dataset
Figure 10: Construction time.
the false positive rate at test time. See the appendix for more information on how this framework modification will change the construction method of PLBFs.
Datasets: We evaluated the algorithms using the following two datasets.
- Malicious URLs Dataset: As in previous papers [11, 14], we used Malicious URLs Dataset [17]. The URLs dataset comprises 223,088 malicious and 428,118 benign URLs. We extracted 20 lexical features such as URL length, use of shortening, number of special characters, etc. We used all malicious URLs and 342,482 (80%) benign URLs as the training set, and the remaining benign URLs as the test set.
- EMBER Dataset: We used the EMBER dataset [18] as in the PLBF research. The dataset consists of 300,000 malicious and 400,000 benign files, along with the features of each file. We used all malicious files and 300,000 (75%) benign files as the train set and the remaining benign files as the test set.
While any model can be used for the classifier, we used LightGBM [19] because of its speed in training and inference, as well as its memory efficiency and accuracy. The sizes of the machine learning model for the URLs and EMBER datasets are  $312\mathrm{Kb}$  and  $1.19\mathrm{Mb}$ , respectively. The training time of the machine learning model for the URLs and EMBER datasets is 1.09 and 2.71 seconds, respectively. The memory usage of LBF is the total memory usage of the backup Bloom filters and the machine learning model. Figure 8 shows a histogram of each dataset's score distributions of keys and non-keys. We can see that the frequency of keys increases and that of non-keys decreases as the score increases. In addition, Figure 9 plots  $g_{i} / h_{i}$  ( $i = 1\dots N$ ) when  $N = 1,000$ . We can see that  $g_{i} / h_{i}$  tends to increase as  $i$  increases, but the increase is not monotonic (i.e., the score distribution is not ideal).
5.1 Construction time
We compared the construction times of fast PLBF and fast PLBF++ with those of existing methods. Following the experiments in the PLBF paper, hyperparameters for PLBF, fast PLBF, and fast PLBF++ were set to  $N = 1,000$  and  $k = 5$ .
Figure 10 shows the construction time for each method. The construction time for learned Bloom filters includes not only the time to insert keys into the Bloom filters but also the time to train the machine learning model and the time to compute the optimal parameters ( $t$  and  $f$  in the case of PLBF).