SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sepehr Assadi<sup>1</sup> Vihan Shah<sup>1</sup> Chen Wang<sup>2</sup>
$^{1}$ University of Waterloo  $^{2}$ Rutgers University
sepehr@assadi.info
vihan.shah@uwaterloo.ca
wc497@cs.rutgers.edu
Correlation clustering is a fundamental optimization problem at the intersection of machine learning and theoretical computer science. Motivated by applications to big data processing, recent years have witnessed a flurry of results on this problem in the streaming model. In this model, the algorithm needs to process the input  $n$ -vertex graph by making one or few passes over the stream of its edges and using a limited memory, much smaller than the input size.
All previous work on streaming correlation clustering has focused on semistreaming algorithms with  $\Omega(n)$  memory, whereas in this work, we study streaming algorithms with much smaller memory requirements of only polylog $(n)$  bits. This stringent memory requirement is in the same spirit of classical streaming algorithms that instead of recovering a full solution to the problem—which can be prohibitively large with such small memory as is the case in our problem—, aimed to learn certain statistical properties of their inputs. In our case, this translates to determining the “(correlation) clusterability” of input graphs, or more precisely, estimating the cost of the optimal correlation clustering solution.
As our main result, we present two novel algorithms that in only polylog(n) space are able to estimate the optimal correlation clustering cost up to some constant multiplicative factor plus some extra additive error. One of the algorithms outputs a 3-multiplicative approximation plus  $o(n^{2})$  additive approximation, and the other one further reduces the additive error at the cost of increasing the multiplicative factor to some large constant. We then present new lower bounds that justify the mix of both multiplicative and additive error approximations in our algorithms.
Correlation clustering is a fundamental optimization problem at the intersection of machine learning and theoretical computer science. This problem was introduced by the work of  $\mathrm{[BBC04]}^1$ , with motivation to document clustering, as follows: we have a complete graph  $G = (V,E)$  whose edges are labeled by either  $(+)$  or  $(-)$ , and the objective, known as disagreement minimization, is to cluster the vertices so that the number of  $(+)$  edges across clusters and  $(-)$  edges inside the same clusters are minimized. Correlation clustering has since found broad applications in areas such as document categorization [BBC04], webpage segmentation [CKP08], microscopy imaging [ZYH14], and community detection [VGW18, SDE+21], to name a few.
There is an abundant body of literature studying polynomial time algorithms for correlation clustering. [BBC04, CMSY15] showed that there exists a 2.06-approximation algorithm in polynomial time and
<sup>1</sup>Historically, some forms of the problem were mentioned in earlier works like [GW89].
the problem is NP-hard and even APX-hard. Furthermore, [ACN05] gave a simple (combinatorial) poly-time 3-approximation algorithm that is widely used in practice. Very recently, breakthrough results by [CLN22, CALLN23] achieved 1.73 approximation in polynomial time. In addition, efficient algorithms are also explored for several variants of this problem like the agreement maximization objective [BBC04, Swa04, CGW03], weighted graphs [Swa04, CGW03, EF03], fixed number of clusters [GG06], fair clustering [AEKM20, SZ22], and others.
In recent years, with the rapid development of the 'big data era', there has been a growing interest in algorithms for correlation clustering under sublinear models. In general, learning algorithms under sublinear models are able to output the answer without processing or storing the entire input. For instance, [BGK13, AW22] gave algorithms to (approximately) learn the clustering in sublinear time, and  $\mathrm{[CLM^{+}21}$ , BCMT22] designed algorithms in the Massively Parallel Computation (MPC) model. Another widely popular model of sublinear algorithms—the focus of our paper—is the graph streaming model. In this model, the edges of the input graph are given to the algorithm one by one in a stream and the memory of the algorithm is desired to be substantially smaller than the input. Here,  $\mathrm{[ACG^{+}15]}$  designed a streaming algorithm with  $\tilde{O}(n)$  memory² that achieves 3-approximation with  $O(\log \log(n))$  passes – here,  $n$  is the number of vertices of the graph and thus the input size is  $\Theta(n^2)$  bits. The number of passes of these algorithms were later improved to  $\mathcal{O}(1)$  by  $\mathrm{[CLM^{+}21]}$  and a single pass by [AW22], albeit with much larger, yet still a constant, approximation factor. Very recently, [BCMT23] further improved the approximation ratio of single-pass streaming algorithms to a 5-approximation in polynomial time and  $(1 + \varepsilon)$ -approximation in exponential-time.
The aforementioned line of work in  $\left[\mathrm{ACG}^{+}15, \mathrm{CLM}^{+}21, \mathrm{AW22}, \mathrm{BCMT23}\right]$  focused on the  $\widetilde{O}(n)$ -memory regime, otherwise known as the semi-streaming memory. Allowing for  $\Omega(n)$  memory in these algorithms is necessary given that even outputting the solution, namely, the clustering of the input labeled graph, requires this much memory. Yet, in many applications,  $\Omega(n)$  memory can still be quite large, and the implementations can require significant resource. In such cases, it is highly desirable to determine the "clusterability" of the input graph before running the actual clustering algorithm. If even the optimal cost for the input is high, it implies the clustering cannot provide any meaningful outcome, and we should not waste resources on these instances. This type of "value estimation" problem is extensively studied in the streaming literature, see e.g. [KKS15, GT19,  $\mathrm{ACL}^{+}22$ , BOS22, DKPP22] for several examples in this context. This raises the following fundamental question:
How well can we estimate the optimal correlation clustering cost with polylog(n)-space streaming algorithms?
Despite the vast body of work on streaming correlation clustering in general, this question has received almost no attention so far. Indeed, to the best of our knowledge, the only prior work here is that of  $\left[\mathrm{ACG}^{+}15\right]$  who proved that any (finite) purely multiplicative factor approximation of the cost is not possible in  $o(n)$  space. This result however does not rule out any additive error approximation (say, in the spirit of [BGK13] for local query algorithms).
We remedy this state of affairs in this paper. Our main algorithmic results show that one can obtain a 3-approximation plus  $o(n^{2})$  additive approximation to the cost of optimal correlation clustering in only polylog(n) space – the additive approximation can be further reduced at the cost of increasing the multiplicative approximation to some large constant. We then complement these results with new streaming lower bounds that further justify the necessity of the additive errors in our algorithms. Throughout, we will present algorithms in the manner of insertion-only streams. However, we shall note that our algorithms can be extended to dynamic streams wherein the edges of the graph can be both inserted and deleted during the stream.
To state our results, we need the following notation. Let OPT be the optimal cost for correlation clustering. We say an algorithm achieves an  $(\alpha, \beta)$ -approximation of OPT if it gives an  $\alpha$ -multiplicative with a  $\beta$  additive approximation, namely, outputs a number ALG such that
$$
O P T \leq A L G \leq \alpha \cdot O P T + \beta .
$$
Here, and throughout, we use  $\tilde{O} (\cdot)$  to hide polylog terms on the parameters.
Our first main result is a single-pass streaming algorithm that achieves an  $(O(1),\delta n^2)$ -approximation with high probability in poly  $(\log n,1 / \delta)$  space.
Result 1. There is a single-pass streaming algorithm that outputs an  $(O(1),\delta n^2)$ -approximation of the optimal correlation clustering cost with high probability and uses  $O(\log (n) / \delta^5)$  space.
Here, and throughout, with high probability means with probability at least  $1 - 1/n$ .
The  $\delta^{-5}$ -dependence of our algorithm ensures that by setting  $\delta = n^{-0.19}$  we can reduce the additive error down to  $n^{1.81} = o(n^2)$  and still achieve an  $o(n)$ -space algorithm. On the flip side however, the leading constant multiplicative factor is quite large in this algorithm - as we will see shortly, the algorithm is built on the sparse-dense decomposition idea from [AW22], which inevitably incurs a (worst-case) constant of at least  $10^7$ . Our next result addresses this drawback: we present another algorithm that achieves a  $(3,\delta n^2)$ -approximation in expectation with polylog  $n$  space at the cost of an exponential dependence on  $\delta$  in the space.
Result 2. There is a single-pass streaming algorithm that outputs a  $(3,\delta n^2)$ -approximation of the optimal correlation clustering cost in expectation and uses  $2^{O(1 / \delta)}\cdot \mathrm{polylog}(n)$  space.
Compared to the algorithm in Result 1, the algorithm in Result 2 is more suitable for instances whose optimal correlation clustering costs are large (e.g.  $\mathsf{OPT} = \Omega(n^2)$ ). In such a case, we can pick  $\delta$  to be a small constant, and achieve a  $(3 + \delta)$ -multiplicative approximation with  $\text{polylog}(n)$  space.
Since both of our upper bounds contain the  $\delta n^2$  additive error, we would naturally wonder to what extent these additive terms are necessary. We present two new lower bounds that partially address this question, and show that the additive errors are necessary to a large extent. Our first lower bound shows that if only additive error is allowed, there is no polylog  $(n)$ -space streaming algorithm in a single pass achieves additive error substantially better than  $O(n^{2})$ .
Result 3. No single-pass streaming algorithm with polylog(n) space can output a  $(1,n^{2 - \varepsilon})$  approximation of the optimal correlation clustering cost with a sufficiently large constant probability of success for some  $\varepsilon = o(1)$ .
Result 3 can also be interpreted as the additive counterpart of the lower bounds in  $\left[\mathrm{ACG}^{+}15\right]$ , which focus instead on the purely multiplicative approximation. Since our upper bound allows both approximations, we further provide a second lower bound showing that a  $\Theta(n)$  additive error is necessary even if both multiplicative and additive errors are allowed.
Result 4. No single-pass streaming algorithm with even  $o(\sqrt{n})$  space can output a  $(1.19, O(n))$  approximation of the optimal correlation clustering cost with a sufficiently large constant probability of success.
We now discuss the applicable scenarios for our algorithms. To test instances with low vs. high correlation clustering costs (e.g.  $o(n^{2})$  vs.  $\Omega(n^{2})$ ), it suffices to run our algorithm in Result 1 with  $\delta = 1 / \log(n)$  which uses  $\widetilde{O}(1)$  space. On the other hand, to separate instances that are 'partially clusterable' (e.g. optimal cost of  $n^{2} / 1000$ ) vs. instances that are 'not clusterable at all' (e.g. optimal cost of  $n^{2} / 5$ ), it suffices to run our algorithm in Result 2 with  $\delta = O(1)$  a small constant which uses  $\widetilde{O}(1)$  space. The only case our algorithms are not able to deal with is to separate multiple 'well-clusterable' instances which does not seem that motivated in practice either.
Experiments To further validate our algorithms, we conduct experiments of our algorithms with the stochastic block model (SBM) extensively studied in the literature (e.g., [HLL83, ABH16, ZT23, Abb17a]). For correlation clustering, we can use a variate of the model that plants clusters with sizes  $\Omega(n)$  and samples (+) edges between the vertices in the same cluster with a large probability and (-) edges with a low probability, and vice versa. Our implementation on the simulations of graph streams from the Stochastic Block Model shows that our algorithms consistently obtain approximations within a factor of 3 for the optimal clustering cost, while only storing  $0.04\% \sim 3.6\%$  total edges when the graphs is moderately large. Furthermore, our algorithms are able to distinguish instances that are "well-clusterable" vs. "badly-clusterable" using a very small memory.
${}^{3}{0.04}\%$  fraction of edges are obtained by implementing the algorithm of Result 2 in a two-pass manner
We now introduce the notation and the problem definition. For standard technical tools, please see the supplementary material.
Notation. We use  $G = (V, E^{+} \cup E^{-})$  to denote a labeled complete graph arriving in a stream. For any vertex  $v$ , we denote by  $N^{+}(v)$  all vertices that have an (+) edge from  $v$ , and we let  $N^{+}[v] = \{v\} \cup N^{+}(v)$ . Similarly, we denote by  $N^{-}(v)$  all vertices that have an (-) edge from  $v$ , and we let  $N^{-}[v] = \{v\} \cup N^{-}(v)$ . We use  $N^{+}(u) \triangle N^{+}(v)$  (resp.  $N^{-}(u) \triangle N^{-}(v)$ ) to denote the disjoint neighborhoods of  $u$  and  $v$ , i.e.  $N^{+}(u) \triangle N^{+}(v) = (N^{+}(u) \cup N^{+}(v)) \setminus (N^{+}(u) \cap N^{+}(v))$ . For a fixed set of vertices  $A \subseteq V$ , we further let  $E^{+}(v, A)$  be the set of (+) edges between  $v$  and vertices in  $A$ , and  $E^{-}(v, A)$  be the set of (-) edges between  $v$  and vertices in  $A$ .
We use  $G^{+} = (V, E^{+})$  to denote the positive subgraph of  $G$  with all the (+) edges, and we define  $G^{-} = (V, E^{-})$  analogously. Note that since we work with labelled complete graph, the information of  $G^{-}$ edges can be uniquely inferred from the positive subgraph  $G^{+}$ . As such, when we work with  $G^{+}$ only, we call a (+) edge  $(u, v) \in E^{+}$ as an edge in  $G^{+}$ . Similarly, we call a (-) edge  $(u, v) \in E^{-}$ an non-edge in  $G^{+}$ . We may omit  $G^{+}$ when the context is clear.
For a fixed cluster  $\mathcal{C}$  on a labeled complete graph  $G$ , we use cost  $(\mathcal{C})$  to denote the total cost of correlation clustering on  $G$  by  $\mathcal{C}$  (we omit  $G$  since it is obvious by the context). Furthermore, we slightly abuse the notation to reload cost  $(\cdot)$  as a function of cost on subgraphs of  $G$  in the following occasions: 1) For an induced subgraph  $H \subseteq G$ , we let cost  $(\mathcal{C}, H)$  be the cost of  $\mathcal{C}$  induced by the edges whose both endpoints are in  $H$ ; 2) For two vertex sets  $A \cup B \subseteq V$ , we let cost  $(\mathcal{C}, (A, B))$  be the cost of  $\mathcal{C}$  induced by the edges with exactly one endpoint in  $A$  and one endpoint in  $B$ .
Finally, for a single edge  $(u,v)$ , we use the notation edge-cost  $(\mathcal{C},(u,v))$  to denote the cost induced by a single edge  $(u,v)$  for clustering  $\mathcal{C}$ . We may write a short-hand notation edge-cost  $(\mathcal{C},(u,v))$  when the context is clear that  $\mathcal{C}$  is used.
We now give the formal description of the problem.
Problem 1 (Correlation Clustering Value Estimation). Given a labeled complete graph  $G = (V, E^{-} \cup E^{+})$  and a clustering  $\mathcal{C}$  that partitions  $V$  into disjoint sets of vertices  $C_{1}, C_{2}, \dots, C_{k}$ , the cost of disagreement minimization correlation clustering on  $\mathcal{C}$  is defined as
$$
\mathbf {c o s t} (\mathcal {C}) := \left| \left\{(u, v) \in E ^ {+} \mid \exists i \neq j s. t. u \in C _ {i}, v \in C _ {j} \right\} \right| + \left| \left\{(u, v) \in E ^ {-} \mid \exists i s. t. u, v \in C _ {i} \right\} \right|
$$
Let OPT be the minimum cost over all possible clusterings. The Correlation Clustering Value Estimation problem asks for a number ALG such that  $\mathrm{OPT} \leq \mathrm{ALG} \leq f(\mathrm{OPT})$  for some function  $f(x) \geq x$ . If  $f(\mathrm{OPT}) = \alpha \cdot \mathrm{OPT} + \beta \cdot n^2$  for  $\alpha$  and  $\beta$ , we say that ALG is a  $(\alpha, \beta)$ -approximation of the value of correlation clustering in this scenario.
We study algorithms for correlation clustering value estimation under the graph streaming model, where edges arrive one after another in a stream with the labels. We give a more formal definition of the streaming model (and the more general dynamic streaming model) in the supplementary material.
3 An Algorithm based on Sparse-dense Decomposition
In this section, we present our first streaming algorithm that achieves a  $(O(1),\delta n^2)$ -approximation for testing the value of correlation clustering in  $O(\mathrm{polylog}(n) / \mathrm{poly}(\delta))$  space, as long as  $\delta \geq 1 / \mathrm{polylog}(n)$ . We utilize the idea in [AW22] to test a  $(O(1),\delta n^2)$ -approximate value of the sparse-dense decomposition-based correlation clustering cost, which in turn is an  $O(1)$  approximation of the optimal cost. Our algorithm uses a memory of  $O(\mathrm{polylog}(n) / \mathrm{poly}(\delta))$  bits, which is efficient for large-scale inputs. More formally, the guarantee of our algorithm is as follows.
Theorem 1. There is a (dynamic) streaming algorithm that with high probability gives a  $(O(1),\delta n^{2})$  approximation for the correlation clustering value and take space  $O\left(\frac{\log^2(n)}{\delta^5}\right)$  words.
On a high level, our algorithm for Theorem 1 uses the idea to approximate the value of optimal correlation clustering with sparse-dense decomposition (see supplementary material). It is shown in
[AW22] that once we can find such a decomposition on  $G^{+}$ , we can achieve an  $O(1)$ -approximation by simply putting every sparse vertex into a singleton cluster and gathering each almost-clique  $K_{i}$  in a separate cluster. Therefore, an approximation of the number of the aforementioned edges in a fixed decomposition will result in a good estimation of the cost.
However, since the sparse-dense decomposition is not unique, it is unclear how to estimate the edges for a fixed decomposition. On the other hand, an algorithm with polylog(n) space is necessarily oblivious of the decomposition since it takes  $\Theta (n)$  bits to write it down. To overcome the problem, we forgo the strict notion of the sparse-dense decomposition, and utilize the notion of  $\varepsilon$ -sparse edges and  $\varepsilon$ -dense non-edges (of  $G^{+}$ ) instead. We now formally define these notions as follows.
Definition 1. Fix an arbitrary graph  $G = (V, E)$  (which is not necessarily labeled and complete) and a vertex pair  $(u, v) \in G$ , we say
1.  $(u,v)$  is an  $\varepsilon$ -sparse edge (resp. non-edge) if  $(u,v) \in E$  (resp.  $(u,v) \notin E$ ) and  $|N(v) \triangle N(u)| \geq \varepsilon \cdot \max \{\deg(u), \deg(v)\}$ .
2.  $(u,v)$  is an  $\varepsilon$ -dense edge (resp. non-edge) if  $(u,v) \in E$  (resp.  $(u,v) \notin E$ ) and  $|N(v) \triangle N(u)| \leq \varepsilon \cdot \max \{\deg(u), \deg(v)\}$ .
Note that the definitions of the  $\varepsilon$ -sparse and  $\varepsilon$ -dense edges/non-edges are generic and not restricted to the labeled complete graph (or even the correlation clustering application). Similarly, the sketching tools we design in this section are also generic: we will use them in the context of correlation clustering later. We now prove the existence of the following tools.
Lemma 3.1. There exists a dynamic streaming algorithm Tool-spr with parameters  $\varepsilon, \delta$  that given any graph  $G = (V, E)$  in a stream and a pair of vertices  $u, v \in V$ , satisfying the promise  $\deg^{+}(u) \geq \delta n$  and  $\deg^{+}(v) \geq \delta n$ , with high probability outputs 'Yes' if  $(u, v)$  is at least  $\varepsilon$ -sparse and "No" if  $(u, v)$  is not  $\frac{\varepsilon}{8}$ -sparse using  $O(1 / \varepsilon^{2} \delta)$  words of space.
Lemma 3.2. There exists a dynamic streaming algorithm Tool-dns with parameters  $\varepsilon, \delta$  that given any graph  $G = (V, E)$  in a stream and a pair of vertices  $u, v \in V$ , satisfying the promise  $\deg^{+}(u) \geq \delta n$  and  $\deg^{+}(v) \geq \delta n$ , with high probability outputs 'Yes' if  $(u, v)$  is at most  $\varepsilon$ -dense and "No" if  $(u, v)$  is not  $8\varepsilon$ -dense using  $O(1 / \varepsilon^{2}\delta)$  words of space.
We defer the proofs of Lemmas 3.1 and 3.2 to the supplementary material, and use them as blackboxes in the rest of the paper. We now discuss how to use the  $\varepsilon$ -sparse edges and  $\varepsilon$ -dense non-edges (defined in Definition 1) in  $G^{+}$  to estimate the correlation clustering cost. In what follows, we use  $E_{\varepsilon\text{-sparse}}^{+}$  to denote the set of  $\varepsilon$ -sparse (positive) edges, and  $E_{\varepsilon\text{-dense}}^{-}$  to denote the set of  $\varepsilon$ -dense non-edges. Furthermore, for each vertex  $v$ , we use  $E_{\varepsilon\text{-dense}, v}^{-}$  to denote the set of  $\varepsilon$ -dense non-edges incident on  $v$ . We let  $m_{\varepsilon\text{-sparse}}^{+} := |E_{\varepsilon\text{-sparse}}^{+}|$  denote the number of  $\varepsilon$ -sparse edges,  $m_{\varepsilon\text{-dense}}^{-} := |E_{\varepsilon\text{-dense}}^{-}|$  denote the number of  $\varepsilon$ -dense non-edges and let  $m_{\varepsilon\text{-dense}, v}^{-} := |E_{\varepsilon\text{-dense}, v}^{-}|$  denote the number of  $\varepsilon$ -dense non-edges incident on  $v$ . Finally, we define  $\widehat{m}_{\varepsilon\text{-dense}}^{-}$  as follows.
Definition 2. For each vertex  $v$  define  $\widehat{m}_{\varepsilon\text{-dense},v}^{-} \coloneqq \min \{m_{\varepsilon\text{-dense},v}^{-}, \deg^{+}(v)\}$ . Furthermore, let  $\widehat{m}_{\varepsilon\text{-dense}}^{-} \coloneqq \sum_{v \in V} \widehat{m}_{\varepsilon\text{-dense},v}^{-}$ .
The intuition behind  $\widehat{m}_{\varepsilon\text{-dense},v}^{-}$  is to count the non-edges (of  $G^{+}$ ) in  $E_{\varepsilon\text{-dense},v}^{-}$  for at most  $\deg^{+}(v)$  times. Our estimator for  $\varepsilon$ -dense non-edges will estimate  $\widehat{m}_{\varepsilon\text{-dense}}^{-}$  instead of  $m_{\varepsilon\text{-dense}}^{-}$  for the following reason: it suffices to estimate  $\widehat{m}_{\varepsilon\text{-dense}}^{-}$  since the number of non-edges inside each almost-clique is at most  $\deg^{+}(v)$ ; on the other hand, if we estimate all  $\varepsilon$ -dense non-edges, there could be a very large overhead since the non-edges between almost-cliques are also counted. We note that if  $\widehat{m}_{\varepsilon\text{-dense},v}^{-} = m_{\varepsilon\text{-dense},v}^{-}$  for all  $v \in V$  then  $\widehat{m}_{\varepsilon\text{-dense}}^{-}$  is twice  $m_{\varepsilon\text{-dense}}^{-}$  (since we are double counting edges). But this is a 2-approximation in the worst case, and we do this to make calculations easier.
We prove the following lemmas that establish the connections between the aforementioned sets of edges and the edges from the sparse-dense decomposition. Due to space limits, we directly present the properties of our estimators, and defer the proof to the supplementary material.
Lemma 3.3. Suppose  $G = (V, E)$  is any labeled graph and  $V = V_{\text{sparse}} \sqcup K_1 \sqcup \ldots \sqcup K_k$  is an  $\varepsilon$ -sparse-dense decomposition of  $G^+$  for  $0 \leq \varepsilon \leq 1/360$  and  $\eta_0 \leq 1/20$ . Let  $C_{SDD}$  be the cost of
correlation clustering when the clusters are the almost cliques and sparse vertices are in singleton clusters. Then we have  $\frac{2}{\eta_0 \cdot \varepsilon} \cdot m_{\eta_0 \varepsilon\text{-sparse}}^+ + m_{\varepsilon\text{-sparse}}^+ + \widehat{m}_{4\varepsilon\text{-dense}}^- \geq C_{SDD} \geq OPT.$
We upper bound the cost of  $m_{\varepsilon \text{-sparse}}^{+}$  and  $\widehat{m}_{\varepsilon \text{-dense}}^{-}$  by a function of OPT using a charging argument.