SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Updated Construction / Update Complexity of Lemma 3.2,  $F = R, Y = [1]$ : Algorithm 3 constructs the partial Gram matrix for each leaf node  $v$  in the segment tree. Each nonzero in the segment  $U[S_0(v):S_1(v),\cdot]$  contributes time  $O(R)$  during line 3 of Algorithm 3 to update a single row and column of  $G^v$ . Summed over all leaves, the cost of line 3 is  $O(\mathrm{nnz}(U)R)$ . The remainder of the construction procedure updates the partial Gram matrices of all internal nodes. Since there are at most  $O\left(\lceil \mathrm{nnz}(U) / R^2 \rceil\right)$  internal nodes and the addition on line 6 costs  $O(R^2)$  per node, the remaining steps of the construction procedure cost  $O(\mathrm{nnz}(U))$ , a lower-order term. The construction time is therefore  $O(\mathrm{nnz}(U)R)$ .
Since we store a single partial Gram matrix of size  $R^2$  at each of  $O\left(\lceil \mathrm{nnz}(U) / R^2 \rceil\right)$  internal nodes, the space complexity of our modified data structure is  $O(\mathrm{nnz}(U))$ .
Finally, the data structure update time in case a single element of  $U$  is modified does not change from Theorem 1.1. Since the depth of the segment tree  $\lceil \mathrm{nnz}(U) / R^2 \rceil + 1$  is upper-bounded by  $\lceil I / R \rceil + 1$ , the runtime of the update procedure in Algorithm 5 stays the same.
Updated Sampling Complexity of Lemma 3.2,  $F = R, Y = [1]$ : The procedure "RowSample" in Algorithm 4 now conducts a traversal of a tree of depth  $O(\lceil \mathrm{nnz}(U) / R^2 \rceil)$ . As a result, we
can still upper-bound the number of calls to procedure  $\tilde{m}$  as  $[I / F]$ . The runtime of procedure  $\tilde{m}$  is unchanged. The runtime of procedure  $\tilde{q}$  for leaf node  $c$  is dominated by the matrix-vector multiplication  $U[S_0(c):S_1(c),:]\cdot h$ . This runtime is  $O(\mathrm{nnz}(U[S_0(c):S_1(c),:]))\leq O(R^2)$ . Putting these facts together, the sampling complexity of the data structure in Lemma 3.2 does not change under our proposed modifications for  $F = R,Y = [1]$ .
Updated Construction Complexity of Theorem 1.1: Algorithm 1 now requires  $O\left(R \sum_{j=1}^{N} \mathrm{nz}(U_j)\right)$  construction time and  $O\left(\sum_{j=1}^{N} \mathrm{nz}(U_j)\right)$  storage space, summing the costs for the updated structure from Lemma 3.2 over all matrices  $U_1, \ldots, U_N$ . The sampling complexity of these data structures is unaffected by the modifications, which completes the proof of the corollary.
A.9 Alternating Least Squares CP Decomposition
CP Decomposition. CP decomposition represents an  $N$ -dimensional tensor  $\tilde{\mathcal{T}} \in \mathbb{R}^{I_1 \times \ldots \times I_n}$  as a weighted sum of generalized outer products. Formally, let  $U_1, \ldots, U_N$  with  $U_j \in \mathbb{R}^{I_j \times R}$  be factor matrices with each column having unit norm, and let  $\sigma \in \mathbb{R}^R$  be a nonnegative coefficient vector. We call  $R$  the rank of the decomposition. The tensor  $\tilde{\mathcal{T}}$  that the decomposition represents is given elementwise by
$$
\tilde {\mathcal {T}} \left[ i _ {1}, \dots , i _ {N} \right] := \left\langle \sigma^ {\top}, U _ {1} \left[ i _ {1}, : \right], \dots , U _ {N} \left[ i _ {N}, : \right] \right\rangle = \sum_ {r = 1} ^ {R} \sigma \left[ r \right] U _ {1} \left[ i _ {1}, r \right] \dots U _ {N} \left[ i _ {N}, r \right],
$$
which is a generalized inner product between  $\sigma^{\top}$  and rows  $U_{j}[i_{j},:]$  for  $1\leq j\leq N$ . Given an input tensor  $\mathcal{T}$  and a target rank  $R$ , the goal of approximate CP decomposition is to find a rank- $R$  representation  $\tilde{\mathcal{T}}$  that minimizes the Frobenius norm  $\left\| \mathcal{T} - \tilde{\mathcal{T}}\right\| _F$ .
Definition of Matricization. The matricization  $\mathrm{mat}(\mathcal{T},j)$  flattens tensor  $\mathcal{T}\in \mathbb{R}^{I_1\times \ldots \times I_N}$  into a matrix and isolates mode  $j$  along the row axis of the output. The output of matricization has dimensions  $I_{j}\times \prod_{k\neq j}I_{k}$ . We take the formal definition below from a survey by Kolda and Bader [13]. The tensor entry  $\mathcal{T}[i_1,\dots,i_N]$  is equal to the matricization entry  $\mathrm{mat}(\mathcal{T},j)[i_N,u]$ , where
$$
u = 1 + \sum_{\substack{k = 1\\ k\neq j}}^{N}\bigl(i_{k} - 1\bigr)\prod_{\substack{m = 1\\ m\neq j}}^{k - 1}I_{m}.
$$
Details about Alternating Least Squares. Let  $U_{1}, \ldots, U_{N}$  be factor matrices of a low-rank CP decomposition,  $U_{k} \in \mathbb{R}^{I_{k} \times R}$ . We use  $U_{\neq j}$  to denote  $\bigodot_{k=N,k \neq j}^{k=1} U_{k}$ . Note the inversion of order here to match indexing in the definition of matricization above. Algorithm 6 gives the non-randomized alternating least squares algorithm CP-ALS that produces a decomposition of target rank  $R$  given input tensor  $\mathcal{T} \in \mathbb{R}^{I_{1} \times \ldots \times I_{N}}$  in general format. The random initialization on line 1 of the algorithm can be implemented by drawing each entry of the factor matrices  $U_{j}$  according to a standard normal distribution, or via a randomized range finder [11]. The vector  $\sigma$  stores the generalized singular values of the decomposition. At iteration  $j$  within a round, ALS holds all factor matrices except  $U_{j}$  constant and solves a linear-least squares problem on line 6 for a new value for  $U_{j}$ . In between least squares solves, the algorithm renormalizes the columns of each matrix  $U_{j}$  to unit norm and stores their original norms in the vector  $\sigma$ . Appendix A.11 contains more details about the randomized range finder and the convergence criteria used to halt iteration.
We obtain a randomized algorithm for sparse tensor CP decomposition by replacing the exact least squares solve on line 6 with a randomized method according to Theorem 2.1. Below, we prove Corollary 3.3, which derives the complexity of the randomized CP decomposition algorithm.
Proof of Corollary 3.3. The design matrix  $U_{\neq j}$  for optimization problem  $j$  within a round of ALS has dimensions  $\prod_{k \neq j} I_k \times R$ . The observation matrix  $\mathrm{mat}(\mathcal{T}, j)^{\top}$  has dimensions  $\prod_{k \neq j} I_k \times I_j$ . To achieve error threshold  $1 + \varepsilon$  with probability  $1 - \delta$  on each solve, we draw  $J = \tilde{O}\left(R / (\varepsilon \delta)\right)$  rows from both the design and observation matrices and solve the downsampled problem (Theorem 2.1).
Algorithm 6 CP-ALS(T,R)
1: Initialize  $U_{j}\in \mathbb{R}^{I_{j}\times R}$  randomly for  $1\leq j\leq N$    
2: Renormalize  $U_{j}(:,i]$ $= \| U_j(:,i]\| _2$ $1\leq j\leq N,1\leq i\leq R.$    
3: Initialize  $\sigma \in \mathbb{R}^R$  to [1].   
4: while not converged do   
5: for  $j = 1\dots N$  do   
6:  $U_{j}\coloneqq \arg \min_{X}\left\| U_{\neq j}\cdot X^{\top} - \mathrm{mat}(\mathcal{T},j)^{\top}\right\|_{F}$    
7:  $\sigma [i] = \| U_j(:,i]\| _2$ $1\leq i\leq R$    
8: Renormalize  $U_{j}(:,i]$ $= \| U_j(:,i]\| _2$ $1\leq i\leq R$    
9: return  $[\sigma ;U_1,\dots ,U_N]$
These rows are sampled according to the leverage score distribution on the rows of  $U_{\neq j}$ , for which we use the data structure in Theorem 1.1. After a one-time initialization cost  $O(\sum_{j=1}^{N} I_j R^2)$  before the ALS iteration begins, the complexity to draw  $J$  samples (assuming  $I_j \geq R$ ) is
$$
O \left(N R ^ {3} + J \sum_ {k \neq j} R ^ {2} \log I _ {k}\right) = \tilde {O} \left(N R ^ {3} + \frac {R}{\varepsilon \delta} \sum_ {k \neq j} R ^ {2} \log I _ {k}\right).
$$
The cost to assemble the corresponding subset of the observation matrix is  $O(JI_{j}) = \tilde{O}(RI_{j} / (\varepsilon \delta))$ . The cost to solve the downsampled least squares problem is  $O(JR^{2}) = \tilde{O}(I_{j}R^{2} / (\varepsilon \delta))$ , which dominates the cost of forming the subset of the observation matrix. Finally, we require additional time  $O(I_{j}R^{2})$  to update the sampling data structure (Theorem 1.1 part 1). Adding these terms together and summing over  $1 \leq j \leq N$  gives
$$
\begin{array}{l} \tilde {O} \left(\frac {1}{\varepsilon \delta} \cdot \sum_ {j = 1} ^ {N} \left[ I _ {j} R ^ {2} + \sum_ {k \neq j} R ^ {3} \log I _ {k} \right]\right) \tag {21} \\ = \tilde {O} \left(\frac {1}{\varepsilon \delta} \cdot \sum_ {j = 1} ^ {N} \left[ I _ {j} R ^ {2} + (N - 1) R ^ {3} \log I _ {j} \right]\right). \\ \end{array}
$$
Rounding  $N - 1$  to  $N$  and multiplying by the number of iterations gives the desired complexity. When  $I_{j} < R$  for any  $j$ , the complexity changes in Theorem 1.1 propagate to the equation above. The column renormalization on line 8 of the CP-ALS algorithm contributes additional time  $O\left(\sum_{j = 1}^{N}I_{j}R\right)$  per round, a lower-order term.
A.10 Experimental Platform and Sampler Parallelism
We provide two implementations of our sampler. The first is a slow reference implementation written entirely in Python, which closely mimics our pseudocode and can be used to test correctness. The second is an efficient implementation written in C++, parallelized in shared memory with OpenMP and Intel Thread Building Blocks.
Each Perlmutter CPU node (our experimental platform) is equipped with two sockets, each containing an AMD EPYC 7763 processor with 64 cores. All benchmarks were conducted with our efficient  $\mathrm{C + + }$  implementation using 128 OpenMP threads. We link our code against Intel Thread Building blocks to call a multithreaded sort function when decomposing sparse tensors. We use OpenBLAS 0.3.21 to handle linear algebra with OpenMP parallelism enabled, but our code links against any linear algebra library implementing the CBLAS and LAPACKE interfaces.
Our proposed data structure samples from the exact distribution of leverage scores of the Khatri-Rao product, thereby enjoying better sample efficiency than alternative approaches such as CP-ARLS-LEV [15]. The cost to draw each sample, however, is  $O(R^2 \log H)$ , where  $H$  is the number of rows in the Khatri-Rao product. Methods such as row-norm-squared sampling or CP-ARLS-LEV can draw each
sample in time  $O(\log H)$  after appropriate preprocessing. Therefore, efficient parallelization of our sampling procedure is required for competitive performance, and we present two strategies below.
1. Asynchronous Thread Parallelism: The KRPSampleDraw procedure in Algorithm 2 can be called by multiple threads concurrently without data races. The simplest parallelization strategy divides the  $J$  samples equally among the threads in a team, each of which makes calls to KRPSampleDraw asynchronously. This strategy works well on a CPU, but is less attractive on a SIMT processor like a GPU where instruction streams cannot diverge without significant performance penalties.
2. Synchronous Batch Parallelism As an alternative to the asynchronous strategy, suppose for the moment that all leaves have the same depth in each segment tree. Then for every sample, STSample makes a sequence of calls to  $\tilde{m}$ , each updating the current node by branching left or right in the tree. The length of this sequence is the depth of the tree, and it is followed by a single call to the function  $\tilde{q}$ . Observe that procedure  $\tilde{m}$  in Algorithm 4 can be computed with a matrix-vector multiplication followed by a dot product. The procedure  $\tilde{q}$  of Algorithm 4 requires the same two operations if  $F = 1$  or  $Y = [1]$ . Thus, we can create a batched version of our sampling procedure that makes a fixed length sequence of calls to batched gemv and dot routines. All processors march in lock-step down the levels of each segment tree, each tracking the branching paths of a distinct set of samples. The MAGMA linear algebra library provides a batched version of gemv [10], while a batched dot product can be implemented with an ad hoc kernel. MAGMA also offers a batched version of the symmetric rank- $k$  update routine syrk, which is helpful to parallelize row sampler construction (Algorithm 3). When all leaves in the tree are not at the same level, the bottom level of the tree can be handled with a special sequence of instructions making the required additional calls to  $\tilde{m}$ .
Our CPU code follows the batch synchronous design pattern. To avoid dependency on GPU-based MAGMA routines in our CPU prototype, portions of the code that should be batched BLAS calls are standard BLAS calls wrapped in a for loop. These sections can be easily replaced when the appropriate batched routines are available.
A.11 Sparse Tensor CP Experimental Configuration
Table 3: Sparse Tensors from FROSTT collection.
TENSOR	DIMENSIONS	NONZEROS	PREP.	INIT.
UBER PICKUPS	183 × 24 × 1,140 × 1,717	3,309,490	NONE	IID
ENRON EMAILS	6,066 × 5,699 × 244,268 × 1,176	54,202,099	LOG	RRF
NELL-2	12,092 × 9,184 × 28,818	76,879,419	LOG	IID
AMAZON REVIEWS	4,821,207 × 1,774,269 × 1,805,187	1,741,809,018	NONE	IID
REDDIT-2015	8,211,298 × 176,962 × 8,116,559	4,687,474,081	LOG	IID
Table 3 lists the nonzero counts and dimensions of sparse tensors in our experiments [25]. We took the log of all values in the Enron, NELL-2, and Reddit-2015 tensors. Consistent with established practice, this operation damps the effect of a few high magnitude tensor entries on the fit metric [15].
The factor matrices for the Uber, Amazon, NELL-2, and Reddit experiments were initialized with i.i.d. entries from the standard normal distribution. As suggested by Larsen and Kolda [15], the Enron tensor's factors were initialized with a randomized range finder [11]. The range finder algorithm initializes each factor matrix  $U_{j}$  as  $\mathrm{mat}(\mathcal{T}, j)S$ , a sketch applied to the mode-  $j$  matricization of  $\mathcal{T}$  with  $S \in \mathbb{R}^{\prod_{k \neq j} I_k \times R}$ . Larsen and Kolda chose  $S$  as a sparse sampling matrix to select a random subset of fibers along each mode. We instead used an i.i.d. Gaussian sketching matrix that was not materialized explicitly. Instead, we exploited the sparsity of  $\mathcal{T}$  and noted that at most  $\mathrm{nnz}(\mathcal{T})$  columns of  $\mathrm{mat}(\mathcal{T}, j)$  were nonzero. Thus, we computed at most  $\mathrm{nnz}(\mathcal{T})$  rows of the random sketching matrix  $S$ , which were lazily generated and discarded during the matrix multiplication without incurring excessive memory overhead.
ALS was run for a maximum of 40 rounds on all tensors except for Reddit, which was run for 80 rounds. The exact fit was computed every 5 rounds (defined as 1 epoch), and we used an early
stopping condition to terminate runs before the maximum round count. The algorithm was terminated at epoch  $T$  if the maximum fit in the last 3 epochs did not exceed the maximum fit from epoch 1 through epoch  $T - 3$  by tolerance  $10^{-4}$ .
Hybrid CP-ARLS-LEV deterministically includes rows from the Khatri-Rao product whose probabilities exceed a threshold  $\tau$ . The ostensible goal of this procedure is to improve diversity in sample selection, as CP-ARLS-LEV may suffer from many repeat draws of high probability rows. We replicated the conditions proposed in the original work by selecting  $\tau = 1 / J$  [15].
Individual trials of non-randomized (exact) ALS on the Amazon and Reddit tensors required several hours on a single Perlmutter CPU node. To speed up our experiments, accuracy measurements for exact ALS in Figure 3 were carried out using multi-node SPLATT, The Surprisingly ParalleL spArse Tensor Toolkit [24], on four Perlmutter CPU nodes. The fits computed by SPLATT agree with those computed by our own non-randomized ALS implementation. As a result, Figure 3 verifies that our randomized algorithm STS-CP produces tensor decompositions with accuracy comparable to those by highly-optimized, state-of-the-art CP decomposition software. We leave a distributed-memory implementation of our randomized algorithms to future work.
A.12 Efficient Computation of Sketch Distortion
The definition of  $\sigma$  in this section is different from its definition in the rest of this work. The condition number  $\kappa$  of a matrix  $M$  is defined as
$$
\kappa (M) := \frac {\sigma_ {\max } (M)}{\sigma_ {\min } (M)}
$$
where  $\sigma_{\min}(M)$  and  $\sigma_{\max}(M)$  denote the minimum and maximum nonzero singular values of  $M$ . Let  $A$  be a Khatri-Rao product of  $N$  matrices  $U_1, \ldots, U_N$  with  $\prod_{j=1}^{N} I_j$  rows,  $R$  columns, and rank  $r \leq R$ . Let  $A = Q\Sigma V^\top$  be its reduced singular value decomposition with  $Q \in \mathbb{R}^{\Pi_j I_j \times r}$ ,  $\Sigma \in \mathbb{R}^{r \times r}$ , and  $V \in \mathbb{R}^{r \times R}$ . Finally, let  $S \in \mathbb{R}^{J \times \Pi_j I_j}$  be a leverage score sampling matrix for  $A$ . Our goal is to compute  $\kappa(SQ)$  without fully materializing either  $A$  or its QR decomposition. We derive
$$
\begin{array}{l} \kappa (S Q) = \kappa \left(S Q \Sigma V ^ {\top} V \Sigma^ {- 1}\right) \tag {22} \\ = \kappa (S A V \Sigma^ {- 1}) \\ \end{array}
$$
The matrix  $SA \in \mathbb{R}^{J \times R}$  is efficiently computable using our leverage score sampling data structure. We require time  $O(JR^2)$  to multiply it by  $V\Sigma^{-1}$  and compute the singular value decomposition of the product to get the condition number. Next observe that  $A^\top A = V\Sigma^2 V^\top$ , so we can recover  $V$  and  $\Sigma^{-1}$  by eigendecomposition of  $A^\top A \in \mathbb{R}^{R \times R}$  in time  $O(R^3)$ . Finally, recall the formula
$$
A ^ {\top} A = \bigotimes_ {j = 1} ^ {N} U _ {j} ^ {\top} U _ {j}
$$
used at the beginning of Section 3 that enables computation of  $A^\top A$  in time  $O\left(\sum_{j=1}^{N} I_j R^2\right)$  without materializing the full Khatri-Rao product. Excluding the time to form  $SA$  (which is given by Theorem 1.1),  $\kappa(SQ)$  is computable in time
$$
O \left(J R ^ {2} + R ^ {3} + \sum_ {j = 1} ^ {N} I _ {j} R ^ {2}\right).
$$
Plugging  $\kappa (SQ)$  into Equation (12) gives an efficient method to compute the distortion.
A.13 Supplementary Results
A.13.1 Comparison Against Standard CP Decomposition Packages
Table 4 compares the runtime per ALS round for our algorithm against existing common software packages for sparse tensor CP decomposition. We compared our algorithm against Tensorly version 0.81 [14] and Matlab Tensor Toolbox version 3.5 [3]. We compared our algorithm against both non-randomized ALS and a version of CP-ARLS-LEV in Tensor Toolbox.
Table 4: Average time (seconds) per ALS round for our method vs. standard CP decomposition packages. OOM indicates an out-of-memory error. All experiments were conducted on a single LBNL Perlmutter CPU node. Randomized algorithms were benchmarked with  $2^{16}$  samples per least-squares solve.
METHOD	UBER	ENRON	NELL-2	AMAZON	REDDIT
TENSORLY, SPARSE BACKEND	64.2	OOM	759.6	OOM	OOM
MATLAB TTOOLBOX STANDARD	11.6	294.4	177.4	>3600	OOM
MATLAB TTOOLBOX CP-ARLS-LEV	0.5	1.4	1.9	34.2	OOM
STS-CP (ours)	0.2	0.5	0.6	3.4	26.0
As demonstrated by Table 4, our implementation exhibits more than  $1000\mathrm{x}$  speedup over Tensorly and  $295\mathrm{x}$  over Tensor Toolbox (non-randomized) for the NELL-2 tensor. STS-CP enjoys a dramatic speedup over Tensorly because the latter explicitly materializes the Khatri-Rao product, which is prohibitively expensive given the large tensor dimensions (see Table 3).
STS-CP consistently exhibits at least  $2.5\mathrm{x}$  speedup over the version of CP-ARLS-LEV in Tensor Toolbox, with more than  $10\mathrm{x}$  speedup on the Amazon tensor. To ensure a fair comparison with CP-ARLS-LEV, we wrote an improved implementation in  $\mathrm{C}++$  that was used for all other experiments.
A.13.2 Probability Distribution Comparison
Figure 7 provides confirmation on a small test problem that our sampler works as expected. For the Khatri-Rao product of three matrices  $A = U_{1} \odot U_{2} \odot U_{3}$ , it plots the true distribution of leverage scores against a normalized histogram of 50,000 draws from the data structure in Theorem 1.1. We choose  $U_{1}, U_{2}, U_{3} \in \mathbb{R}^{8 \times 8}$  initialized i.i.d. from a standard normal distribution with  $1\%$  of all entries multiplied by 10. We observe excellent agreement between the histogram and the true distribution.
Figure 7: Comparison of true leverage score distribution with histogram of 50,000 samples drawn from  $U_{1} \odot U_{2} \odot U_{3}$ .
A.13.3 Fits Achieved for  $J = 2^{16}$
Table 5 gives the fits achieved for sparse tensor decomposition for varying rank and algorithm (presented graphically in Figure 4). Uncertainties are one standard deviation across 8 runs of ALS.
A.13.4 Fit as a Function of Time
Figures 8a and 8b shows the fit as a function of time for the Amazon Reviews and NELL2 tensors. The hybrid version of CP-ARLS-LEV was used for comparison in both experiments. As in section 4.3, thick lines are averages of the running max fit across 4 ALS trials, shown by the thin dotted lines. For Amazon, the STS-CP algorithm makes faster progress than CP-ARLS-LEV at all tested sample counts.
For the NELL-2 tensor, STS-CP makes slower progress than CP-ARLS-LEV for sample counts up to  $J = 163,840$ . On average, these trials with CP-ARLS-LEV do not achieve the same final fit as STS-CP. CP-ARLS-LEV finally achieves a comparable fit to STS-CP when the former uses  $J = 196,608$  samples, compared to  $J = 65,536$  for our method.
Table 5: Fits Achieved by Randomized Algorithms for Sparse Tensor Decomposition,  $J = 2^{16}$ , and non-randomized ALS. The best result among randomized algorithms is bolded. "CP-ARLS-LEV-H" refers to the hybrid version of CP-ARLS-LEV and "Exact" refers to non-randomized ALS.
TENSOR	R	CP-ARLS-LEV	CP-ARLS-LEV-H	STS-CP (OURS)	EXACT
UBER	25	.187 ± 2.30E-03	.188 ± 2.11E-03	.189 ± 1.52E-03	.190 ± 1.41E-03
50	.211 ± 1.72E-03	.212 ± 1.27E-03	.216 ± 1.18E-03	.218 ± 1.61E-03
75	.218 ± 1.76E-03	.218 ± 2.05E-03	.230 ± 9.24E-04	.232 ± 9.29E-04
100	.217 ± 3.15E-03	.217 ± 1.69E-03	.237 ± 2.12E-03	.241 ± 1.00E-03
125	.213 ± 1.96E-03	.213 ± 2.47E-03	.243 ± 1.78E-03	.247 ± 1.52E-03
ENRON	25	.0881 ± 1.02E-02	.0882 ± 9.01E-03	.0955 ± 1.19E-02	.0978 ± 8.50E-03
50	.0883 ± 1.72E-02	.0920 ± 6.32E-03	.125 ± 1.03E-02	.132 ± 1.51E-02
75	.0899 ± 6.10E-03	.0885 ± 6.39E-03	.149 ± 1.25E-02	.157 ± 4.87E-03
100	.0809 ± 1.26E-02	.0787 ± 1.00E-02	.164 ± 5.90E-03	.176 ± 4.12E-03
125	.0625 ± 1.52E-02	.0652 ± 1.00E-02	.182 ± 1.04E-02	.190 ± 4.35E-03
NELL-2	25	.0465 ± 9.52E-04	.0467 ± 4.61E-04	.0470 ± 4.69E-04	.0478 ± 7.20E-04
50	.0590 ± 5.33E-04	.0593 ± 4.34E-04	.0608 ± 5.44E-04	.0618 ± 4.21E-04
75	.0658 ± 6.84E-04	.0660 ± 3.95E-04	.0694 ± 2.96E-04	.0708 ± 3.11E-04
100	.0700 ± 4.91E-04	.0704 ± 4.48E-04	.0760 ± 6.52E-04	.0779 ± 5.09E-04
125	.0729 ± 8.56E-04	.0733 ± 7.22E-04	.0814 ± 5.03E-04	.0839 ± 8.47E-04
AMAZON	25	.338 ± 6.63E-04	.339 ± 6.99E-04	.340 ± 6.61E-04	.340 ± 5.78E-04
50	.359 ± 1.09E-03	.360 ± 8.04E-04	.366 ± 7.22E-04	.366 ± 1.01E-03
75	.367 ± 1.82E-03	.370 ± 1.74E-03	.382 ± 9.13E-04	.382 ± 5.90E-04
100	.366 ± 3.05E-03	.371 ± 2.53E-03	.392 ± 6.67E-04	.393 ± 5.62E-04
125	.358 ± 6.51E-03	.364 ± 4.22E-03	.400 ± 3.67E-04	.401 ± 3.58E-04
REDDIT	25	.0581 ± 1.02E-03	.0583 ± 2.78E-04	.0592 ± 3.07E-04	.0596 ± 4.27E-04
50	.0746 ± 1.03E-03	.0738 ± 4.85E-03	.0774 ± 7.88E-04	.0783 ± 2.60E-04
75	.0845 ± 1.64E-03	.0849 ± 8.96E-04	.0909 ± 5.49E-04	.0922 ± 3.69E-04
100	.0904 ± 1.35E-03	.0911 ± 1.59E-03	.101 ± 6.25E-04	.103 ± 7.14E-04
125	.0946 ± 2.13E-03	.0945 ± 3.17E-03	.109 ± 7.71E-04	.111 ± 7.98E-04
(a) Amazon
Figure 8: Fit as a function of time,  $R = 100$ .
(b) NELL-2
A.13.5 Speedup of STS-CP and Practical Usage Guide
Timing Comparisons. For each tensor, we now compare hybrid CP-ARLS-LEV and STS-CP on the time required to achieve a fixed fraction of the fit achieved by non-randomized ALS. For each tensor and rank in the set  $\{25, 50, 75, 100, 125\}$ , we ran both algorithms using a range of sample counts. We tested STS-CP on values of  $J$  from the set  $\{2^{15}x \mid 1 \leq x \leq 4\}$  for all tensors. CP-ARLS-LEV required a sample count that varied significantly between datasets to hit the required thresholds, and we report the sample counts that we tested in Table 6. Because CP-ARLS-LEV has poorer sample complexity than STS-CP, we tested a wider range of sample counts for the former algorithm.