SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gregory Dexter
gdexter@purdue.edu
Petros Drineas
pdrineas@purdue.edu
Taisuke Yasuda
taisukey@cs.cmu.edu
Sketching algorithms have recently proven to be a powerful approach both for designing low-space streaming algorithms as well as fast polynomial time approximation schemes (PTAS). In this work, we develop new techniques to extend the applicability of sketching-based approaches to the sparse dictionary learning and the Euclidean  $k$ -means clustering problems. In particular, we initiate the study of the challenging setting where the dictionary/clustering assignment for each of the  $n$  input points must be output, which has surprisingly received little attention in prior work. On the fast algorithms front, we obtain a new approach for designing PTAS's for the  $k$ -means clustering problem, which generalizes to the first PTAS for the sparse dictionary learning problem. On the streaming algorithms front, we obtain new upper bounds and lower bounds for dictionary learning and  $k$ -means clustering. In particular, given a design matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  in a turnstile stream, we show an  $\tilde{O}(nr / \epsilon^2 + dk / \epsilon)$  space upper bound for  $r$ -sparse dictionary learning of size  $k$ , an  $\tilde{O}(n / \epsilon^2 + dk / \epsilon)$  space upper bound for  $k$ -means clustering, as well as an  $\tilde{O}(n)$  space upper bound for  $k$ -means clustering on random order row insertion streams with a natural "bounded sensitivity" assumption. On the lower bounds side, we obtain a general  $\tilde{\Omega}(n / \epsilon + dk / \epsilon)$  lower bound for  $k$ -means clustering, as well as an  $\tilde{\Omega}(n / \epsilon^2)$  lower bound for algorithms which can estimate the cost of a single fixed set of candidate centers.
A classic idea in machine learning and signal processing for efficiently handling large datasets is to approximate them by simpler or more structured surrogate datasets. Many methods in this direction have long been considered, including low rank approximation, which approximates a given dataset by one that lies on a low-dimensional subspace,  $k$ -means clustering, which approximates a given dataset by at most  $k$  distinct points, and sparse dictionary learning [Olshausen and Field 1997], which approximates a given dataset by linear combinations of elements of a small dictionary of size  $k$  with  $r$ -sparse coefficient vectors (i.e., a vector with at most  $r$  nonzero entries). We focus on the latter two problems in this work:
Definition 1.1 (r-sparse dictionary learning). Let  $\{a^i\}_{i=1}^n \subseteq \mathbb{R}^d$  be a set of  $n$  vectors in  $d$  dimensions, and let  $\mathbf{A} \in \mathbb{R}^{n \times d}$  be the matrix with the  $i$ th row set to  $a^i$ . Then for a matrix  $\mathbf{X} \in \mathbb{R}^{n \times k}$  with
$r$ -sparse rows and a dictionary  $\mathbf{D} \in \mathbb{R}^{k \times d}$ , we define the dictionary learning cost to be
$$
\operatorname {c o s t} (\mathbf {X}, \mathbf {D}) := \| \mathbf {X D} - \mathbf {A} \| _ {F} ^ {2}
$$
In the  $r$ -sparse dictionary learning problem, we seek to minimize  $\mathrm{cost}(\mathbf{X}, \mathbf{D})$  over all  $\mathbf{X} \in \mathcal{X}$  and  $\mathbf{D} \in \mathbb{R}^{k \times d}$ , where  $\mathcal{X}$  denotes the set of all  $n \times k$  matrices with  $r$ -sparse rows.
Definition 1.2 (Euclidean  $k$ -means clustering). Let  $\{a^i\}_{i=1}^n \subseteq \mathbb{R}^d$  be a set of  $n$  vectors in  $d$  dimensions, and let  $\mathbf{A} \in \mathbb{R}^{n \times d}$  be the matrix with the  $i$ th row set to  $a^i$ . Then, for a matrix  $\mathbf{X} \in \mathbb{R}^{n \times k}$  with standard basis vectors in its rows and a set of centers  $\mathbf{C} \in \mathbb{R}^{k \times d}$ , we define the  $k$ -means clustering cost to be
$$
\operatorname {c o s t} (\mathbf {X}, \mathbf {C}) := \| \mathbf {X C} - \mathbf {A} \| _ {F} ^ {2}.
$$
In the  $k$ -means clustering problem, we seek to minimize  $\mathrm{cost}(\mathbf{X},\mathbf{C})$  over all  $\mathbf{X}\in \mathcal{X}$  and  $\mathbf{C}\in \mathbb{R}^{k\times d}$ , where  $\mathcal{X}$  denotes the set of all  $n\times k$  matrices with standard basis vectors as rows.
While dictionary learning and clustering have found extraordinary success in various applications in practice, they are known to be computationally difficult problems to solve [Mahajan et al., 2012, Natarajan, 1995], and thus there has been intense focus on developing approximation algorithms and heuristics for these problems, such as those based on greedy methods [Lloyd, 1982, Das and Kempe, 2011] or convex relaxations [Donoho and Elad, 2003, Fuchs, 2004, Cohen-Addad et al., 2022a].
In this work, we study algorithms for sparse dictionary learning and  $k$ -means clustering in two distinct settings via a unified set of techniques based on sketching. Sketching [Woodruff, 2014b], broadly speaking, refers to techniques for compressing large matrices by linear maps, and includes methods such as oblivious sketching and nonuniform sampling. Classically, sketching has been applied to design low-memory algorithms in the streaming setting, when the input is presented to the algorithm as a sequence of updates. More recently, sketching has been shown to be invaluable for designing fast algorithms as well. In particular, there has been a line of work which shows how sketching techniques can be applied to obtain polynomial time approximation schemes (PTAS) for a variety of NP-hard problems ranging from clustering [Feldman et al., 2007] to weighted low rank approximation [Razenshteyn et al., 2016] to tensor decompositions [Song et al., 2019]. We study such sketching-based algorithms for sparse dictionary learning and Euclidean  $k$ -means clustering, both in the offline setting where we obtain the first PTAS for sparse dictionary learning, as well as in the turnstile streaming and other streaming models. In particular, in the streaming setting, we initiate the study of solving these problems in the setting where the algorithm must output the assignment of the points to the dictionary/clustering, which has received surprisingly little attention in prior work.
1.1.1 PTAS for dictionary learning and clustering
We start with a discussion of our results on designing fast PTAS's. Our main contribution that we highlight from this section is the first PTAS for sparse dictionary learning, which also gives a new and simple approach towards designing a PTAS for  $k$ -means clustering.
A typical approach for designing PTAS's for shape fitting problems such as dictionary learning and clustering is to first find a smaller instance whose solution approximates the original instance, and then to solve the smaller instance using any algorithm, where even an inefficient algorithm will be tractable due to the smaller size of the instance. A representative work which takes such an approach for the  $k$ -means clustering problem is that of Feldman et al. [2007], which uses coresets to implement the first step of finding a smaller instance. Here, coresets for  $k$ -means clustering are a weighted subset of the original data points such that the cost of any candidate set of centers approximates the cost when applied to the original dataset. Furthermore, the size of this coreset can be taken to be poly( $k / \epsilon$ ), and thus solving for an optimal set of centers on this subset of points can be done in time independent of the number of points  $n$ . Due to this natural approach, there has been a long line of work on obtaining smaller coresets for  $k$ -means clustering [Feldman and Langberg, 2011; Braverman et al., 2016; Bachem et al., 2018; Cohen-Addad et al., 2021; 2022b; c].
On the other hand, for the sparse dictionary learning problem, similar results are strikingly lacking. The only previous work we are aware of is a coreset construction for the sparse dictionary learning problem due to Feldman et al. [2013]. However, the construction of the coreset in this work requires an algorithm for computing an approximately optimal dictionary, which prevents its use in designing
fast PTAS's to solve the dictionary learning problem in the first place. To address this problem, we first show that a completely different coreset technique due to Tukan et al. [2022] for the projective clustering problem can in fact be applied to the sparse dictionary learning problem. Notably, this technique uses John ellipsoids to construct coresets rather than using a nearly optimal solution to the dictionary learning problem, and thus avoids computing approximately optimal dictionaries. In turn, this allows us to obtain the first PTAS for the dictionary learning problem. Our argument additionally combines this coreset construction with a sparsity-counting technique together with polynomial system solvers Renegar [1992a b] to efficiently solve a smaller version of the original problem. Our techniques also yield a new PTAS for  $k$ -means clustering, which is arguably simpler than prior approaches such as the algorithm of Feldman et al. [2007]. We give a full discussion of our results and techniques for our PTAS for sparse dictionary learning in Section ②
1.1.2 Dictionary learning and clustering on streams
As our next contribution, we study algorithms for dictionary learning and clustering in turnstile streams and other related models of streaming. In the turnstile streaming model, the input undergoes arbitrary entrywise insertions and deletions:
Definition 1.3 (Turnstile stream). We say that an input matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  is presented in a turnstile stream if  $\mathbf{A}$  is initialized to 0 and receives entrywise updates  $\mathbf{A}_{i,j} \gets \mathbf{A}_{i,j} + \Delta$  for  $\Delta \in \mathbb{R}$ .
We initiate a systematic study of the dictionary learning and clustering problems in the setting where the assignment of the points to their sparse set of dictionary elements or clusters must be output together with the dictionary/cluster centers. Indeed, even for the popular Euclidean  $k$ -means clustering problem, almost all prior work that we are aware of only focus on outputting either only the cluster partitions, or the centers, but do not study the problem of recovering both. We address this problem by providing a dimensionality reduction technique that applies to  $k$ -means, sparse dictionary learning, and more generally to any problem of the form  $\min_{\mathbf{X} \in \mathcal{X}, \mathbf{D} \in \mathbb{R}^{k \times d}} \| \mathbf{X}\mathbf{D} - \mathbf{A} \|_F^2$ .
A typical approach for designing low-space streaming algorithms for clustering is to apply the standard Johnson-Lindenstraus lemma [Johnson and Lindenstrauss 1984, Boutsidis et al. 2010, Cohen et al. 2015, Becchetti et al. 2019, Makarychev et al. 2019]. This result states that if  $\mathbf{G} \in \mathbb{R}^{d \times s}$  is an appropriately scaled dense sub-Gaussian matrix for  $s = O(\epsilon^{-2} \log (k / \epsilon))$ , then for any partition of  $\mathbf{A}$  into  $k$  clusters, the  $k$ -means clustering cost of  $\mathbf{AG}$  approximates the  $k$ -means clustering cost of  $\mathbf{A}$  up to a  $(1 \pm \epsilon)$  factor. Furthermore,  $\mathbf{AG}$  can be efficiently maintained in the turnstile streaming model (Definition 1.3) using just  $ns = \tilde{O}(\epsilon^{-2} n)$  space, due to the linearity of the sketch  $\mathbf{G}$ . Note however that, naively, we cannot retrieve the corresponding centers of a clustering found by this method, since we have only stored the  $s$ -dimensional sketches of the  $n$  points, and additional information must be stored in order to retrieve  $d$ -dimensional cluster centers which achieve a  $(1 + \epsilon)$  approximation. In fact, we note in Theorem 4.1 that there is in fact a  $\tilde{\Omega}(dk / \epsilon)$  space lower bound if we wish to output centers  $\mathbf{C} \in \mathbb{R}^{k \times d}$  which achieve a  $(1 + \epsilon)$  approximation, so the sketch  $\mathbf{AG}$  is provably insufficient for outputting both a nearly optimal assignment  $\mathbf{X}$  and centers  $\mathbf{C}$  when  $n = \tilde{o}(\epsilon dk)$ . We give a full discussion of our approaches for sketching and streaming algorithms for  $k$  means clustering and dictionary learning and how we overcome this problem in Sections 2 and 3.
On the other hand, a study of lower bounds for the  $k$ -means clustering problem in the streaming setting when the assignment of points must be output is notably lacking in prior work as well. The main challenge in this setting is in obtaining the right dependence on  $n$  and  $\epsilon$ . Indeed, an  $\Omega(n)$  lower bound is immediate, since the size of the output is at least  $\Omega(n)$  when we need to output assignments of the  $n$  points to its appropriate cluster (in fact, we show in Theorems 4.3 and 4.4 that an  $\Omega(n)$  lower bound follows even for outputting a constant factor approximation of the cost or centers). On the other hand, the previous upper bound using the Johnson-Lindenstrauss lemma to compute a nearly optimal assignment to clusters requires  $\tilde{O}(\epsilon^{-2}n)$  bits of space. Note that there are many lower bounds that show that roughly  $\epsilon^{-2}$  dimensions are required to apply the Johnson-Lindenstrauss lemma in various settings [Nelson and Nguyen, 2014], [Kane et al., 2010], [Larsen and Nelson, 2016, 2017], [Makarychev et al., 2019]. However, it is not clear whether or not this implies that  $\epsilon^{-2}$  bits must be stored for all  $n$  points in order to cluster them to a  $(1 + \epsilon)$ -approximately optimal clustering solution. Indeed, it may be possible that  $\epsilon^{-2}$  bits are required only for much fewer than  $n$  points, while the vast majority of the  $n$  input points requires only  $\tilde{O}(n)$  bits of space to assign to an approximately optimal center.
We present two lower bounds to partially address the question of impossibility results for assigning points to clusters in turnstile streams. Our main lower bound result is the following, which establishes an  $\tilde{\Omega} (\epsilon^{-1}n)$  lower bound to output a  $(1 + \epsilon)$ -nearly optimal clustering. While this does not match the upper bound given by the Johnson-Lindenstrauss lemma, it shows that we cannot hope for a  $\tilde{O}(n)$  upper bound in the turnstile streaming model in general.
Theorem 1.1 (Informal restatement of Theorem C.1). Let  $k = d = \tilde{O}(1/\epsilon)$ . Suppose a turnstile streaming algorithm outputs centers  $\{\hat{c}^j\}_{j=1}^k \subseteq \mathbb{R}^d$  as well as assignments of  $n$  points to the  $k$  centers, which achieves a  $(1 + \epsilon)$ -approximately optimal solution to the  $k$ -means clustering problem. Then, the algorithm must use at least  $\tilde{\Omega}(n/\epsilon)$  bits of space over any constant number of passes.
As a second lower bound result, we also show that the Johnson-Lindenstrauss lemma is nearly tight if we require our algorithm to give a nearly optimal assignment of the input points to a fixed set of candidate centers. That is, we show in Theorem 4.2 that there is a fixed set of centers such that, if a turnstile streaming algorithm can assign each of the  $n$  input points to a cluster such that the cost is at most  $(1 + \epsilon)$  times the cost of the optimal assignment, then at least  $\Omega (\epsilon^{-2}n)$  bits must be stored. A more detailed discussion of our lower bounds is given in Section 4.
Finally, we show that under some natural settings, one can obtain upper bounds that circumvent the lower bounds presented above. Indeed, we show that if we work in the random order row arrival streaming model, in which the input stream corresponds to the rows of  $\mathbf{A}$  that arrive in a uniformly random order, then we can obtain upper bounds that depend on the maximum sensitivity of the input stream, and in particular, we obtain an upper bound using only  $\tilde{O}(n)$  bits of space if the maximum sensitivity is sufficiently small (Theorem 4.5). Here, a bounded sensitivity assumption states that there are no points that can take up a significant fraction of the objective function, and can also be interpreted as a way to formalize a "well-clustered" instance.
2 Fixed parameter PTAS for sparse dictionary learning
2.1 PTAS for  $r$ -sparse dictionary learning
In this section, we provide an algorithm which solves the  $r$ -sparse dictionary learning problem (Definition 1.1) in time polynomial in the input matrix size  $(n)$  and dimension  $(d)$  up to  $\epsilon$ -relative error, for fixed  $k$  and  $\epsilon$ . Additionally, we show that a similar approach can be used to provide an algorithm for  $k$ -means (Definition 1.2) that matches the current best dependency on  $n, d, \epsilon$  and  $k$  up to lower terms. First, we introduce a dimensionality reduction method that applies to both problems.
2.2 Dimensionality reduction
Our first step is to reduce the dimensionality of the given problem. Since the only difference between  $k$ -means and sparse dictionary learning is the constraint on the left factor,  $\mathbf{X}$ , we can use the same sketching approach to reduce both problems. Consider the following general definition:
General problem: Let  $\mathcal{X} \subset \mathbb{R}^{n \times k}$  and  $\mathbf{A} \in \mathbb{R}^{n \times d}$ . Let  $k \ll n, d$ . Define the optimal solution as:
$$
\left(\mathbf {X} ^ {*}, \mathbf {D} ^ {*}\right) = \underset {\mathbf {X} \in \mathcal {X}, \mathbf {D} \in \mathbb {R} ^ {k \times d}} {\operatorname {a r g m i n}} \| \mathbf {X D} - \mathbf {A} \| _ {F} ^ {2} \tag {1}
$$
The following theorem states that one may efficiently reduce the dimensionality of  $\mathbf{A}$  in sparse dictionary learning or  $k$ -means. We briefly sketch the ideas behind the reduction. Intuitively, the regression guarantee of Theorem 3.1 in Clarkson and Woodruff [2009] states that if  $\mathbf{S}$  is a rank  $k \ll d$ $\ell_2$ -embedding matrix, then  $\tilde{\mathbf{D}} = \operatorname*{argmin}_{\mathbf{D} \in \mathbb{R}^{k \times d}} \| \mathbf{S}(\mathbf{X}^*\mathbf{D} - \mathbf{A}) \|_F^2$  will be a good approximation to the optimal solution of the original problem. While we do not know  $\mathbf{X}^*$ , this guarantee implies that there is an approximately optimal dictionary,  $\tilde{\mathbf{D}}$ , in the row space of SA. We can then restrict the optimization problem to consider only dictionaries in this lower dimensional space. Therefore, we only need to consider the error residual in this lower dimensional space, so we may reduce the dimension of the problem by applying an affine-embedding matrix  $\mathbf{T}$  and then applying SVD to find the dominant singular subspace of SAT. Finally, we project the rows of  $\mathbf{A}$  to this dominant subspace. We can then solve the lower dimensional problem and map the solution to the original space.
Theorem 2.1. There is an algorithm which solves the problem in (1) up to  $\epsilon \in (0,1)$  relative error with constant probability in  $\mathcal{O}(\mathsf{nnz}(\mathbf{A}) + (n + d)\mathrm{poly}(k / \epsilon))$  time plus the time needed to solve:
$$
\min _ {\mathbf {X} \in \mathcal {X}, \mathbf {D} \in \mathbb {R} ^ {k \times s}} \| \mathbf {X D} - \mathbf {A} ^ {\prime} \| _ {F} ^ {2},
$$
to within  $\epsilon$ -relative error for  $s = \mathcal{O}(k\log (k) / \epsilon)$  and some  $\mathbf{A}' \in \mathbb{R}^{n \times s}$  with constant probability.
In the rest of this section, we assume that  $d = \mathrm{poly}(k / \epsilon)$  for clearer exposition, since the above theorem implies we can reduce to this case efficiently.
2.3 Algorithm for sparse dictionary learning
The first component of our algorithm for sparse dictionary learning is a coreset construction that reduces the size of the problem from  $n$  to a size that is logarithmic in  $n$ . We achieve this by first leveraging an existing coreset construction for projective clustering by Tukan et al. [2022]. In the  $(\ell, m)$ -projective clustering problem, the goal is to find a set of  $\ell$ $m$ -dimensional subspaces that minimizes the sum of the squared Euclidean distances of the input vectors  $\{a^i\}_{i=1}^n$  to the closest subspace. Observe that, in the  $r$ -sparse dictionary problem, the minimum cost of a dictionary is the sum of the squared Euclidean distances of the input vectors to the  $\binom{k}{r}$  subspaces spanned by any subset of  $r$  vectors of the  $k$  vectors in the dictionary. Hence, a coreset which preserves the projective clustering cost when  $\ell = \binom{k}{r}$  will also preserve the cost of a dictionary in sparse dictionary learning.
After applying the coreset, we have reduced the size of the sparse dictionary problem to be at most logarithmic in  $n$ . This allows us to guess the sparsity pattern of the optimal left factor  $\mathbf{X}^*$ , since at most  $r$  entries in each row of  $\mathbf{X}^*$  may be nonzero. For each guess of the sparsity pattern of  $\mathbf{X}^*$ , we can find an approximately optimal solution under this constraint by recognizing this as a polynomial optimization problem. We apply the decision algorithm of Renegar [1992a] using binary search to determine each entry of  $\mathbf{D}$  and the nonzero entries of  $\mathbf{X}$  as done in Razenshteyn et al. [2016]. At some point we guess the sparsity pattern of  $\mathbf{X}^*$ , and hence attain an  $\epsilon$ -relative error solution to the sparse dictionary problem. The next theorem formally states the assumptions and guarantees of our algorithm, which is formalized in Algorithm [1] in the appendix.
Theorem 2.2. For an input for the  $r$ -sparse dictionary learning problem (Definition [1,1]) with error tolerance  $\epsilon \in (0,1)$  such that the entries of  $\mathbf{A}$  have bounded bit complexity, Algorithm [1] returns  $\tilde{\mathbf{X}} \in \mathcal{X}$  and  $\tilde{\mathbf{D}} \in \mathbb{R}^{k \times d}$  satisfying:
$$
\| \tilde {\mathbf {X}} \tilde {\mathbf {D}} - \mathbf {A} \| _ {F} \leq (1 + \epsilon) \min  _ {\mathbf {X} \in \mathcal {X}, \mathbf {D} \in \mathbb {R} ^ {k \times d}} \| \mathbf {X D} - \mathbf {A} \| _ {F},
$$
in poly(n) time with constant probability, when  $k$ ,  $r$ , and  $1 / \epsilon$  are bounded by a constant.
2.4 Algorithms for  $k$ -means
The same general approach of applying dimensionality reduction and a coreset construction along with guessing the sparsity pattern of  $\mathbf{X}^*$  can be used to achieve a fixed-parameter PTAS for  $k$ -means as well. However, we can achieve an improved time complexity matching the current best dependency on  $k$  and  $\epsilon$  up to lower order terms by further reducing the problem using results on leverage score sampling. Specifically, we combine Theorem 17 in Woodruff [2014b] and Theorem 3.1 in Clarkson and Woodruff [2009] to prove the following lemma.
Lemma 2.1. There is a set of matrices  $S \subset \mathbb{R}^{s \times n}$  with exactly one non-zero entry per column such that for any  $\mathbf{A} \in \mathbb{R}^{n \times k}$  and  $\mathbf{B} \in \mathbb{R}^{n \times d}$ , there exists  $\mathbf{S} \in S$ , so that if:
$$
\tilde {\mathbf {X}} = \operatorname * {a r g m i n} _ {\mathbf {X} \in \mathbb {R} ^ {k \times d}} \| \mathbf {S} (\mathbf {A X} - \mathbf {B}) \| _ {F} a n d \mathbf {X} ^ {*} = \operatorname * {a r g m i n} _ {\mathbf {X} \in \mathbb {R} ^ {k \times d}} \| \mathbf {A X} - \mathbf {B} \| _ {F},
$$
$$
\left\| \mathbf {A} \tilde {\mathbf {X}} - \mathbf {B} \right\| _ {F} \leq (1 + \epsilon) \left\| \mathbf {A} \mathbf {X} ^ {*} - \mathbf {B} \right\| _ {F}.
$$
Furthermore,  $S$  depends only on  $n$ ,  $k$ , and  $\epsilon$ ; and  $|\mathcal{S}| = n^{\mathcal{O}\left(\frac{k\log k}{\epsilon}\right)}$ .
<sup>1</sup>If  $k$  and  $r$  are not assumed to be constant, then the time complexity is  $\exp ((8k^{3r})^{O(k^{2r + 1})}\log n)$
After applying a coreset construction to reduce the  $k$ -means problem to size  $\mathrm{poly}(k / \epsilon)$ , we can efficiently apply the above lemma to then reduce the problem to size  $\tilde{\mathcal{O}}(k / \epsilon)$ . Then, we brute force over all possible left-factors to find  $\mathbf{X}^*$ . The following theorem states our results formally.