SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our low rank approximation algorithms use a matrix vector multiplication subroutine  $f(UV)z_{i}$  for  $O(\mathrm{poly}(k / \epsilon))$  different vectors  $z_{i}$  as their main dimensionality reduction technique for subquadratic time upper bounds. This can be used to directly translate lower bounds for low rank approximation to implicit matrix vector multiplication for a wide range of scalar functions that extends previous work beyond the exponential function. We emphasize that these matrix-vector product lower bounds also imply that our LRA lower bounds are non-trivial and significantly strengthen those provided by previous works.
Theorem 4.3 (LRA reduces to Matrix Vector Products). Let  $U, V, f$  be as in Theorem 3.2. Then, any possibly randomized algorithm that for any vector  $z \in \mathbb{R}^n$ , outputs  $f(UV)z$  up to  $1/poly(n)$  entrywise error, with constant probability, requires  $\Omega((nd)^{1 - o(1)})$  time, under Assumption 1. This holds even when  $U = V^\top$ .
A Missing Lower Bound Proofs
Proof of Theorem 3.2. Let  $U, V^T \in \{-1,0,1\}^{n \times r}$  be an instance of entrywise transformed low rank approximation with  $g$ . WLOG, by scaling our solution, we can let  $c_p = 1$ . Let  $B = \mathrm{poly}(n)$  be sufficiently large, and let  $\tilde{U} = U / \sqrt{B}$  and  $\tilde{V} = V / \sqrt{B}$ . For each  $i,j \in [n]$ , for  $B$  sufficiently large and using that  $r = n^{o(1)}$ ,
$$
\begin{array}{l} {g (\tilde {U} _ {i} \tilde {V} _ {j})} = {f (| \tilde {U} _ {i} \tilde {V} _ {j} |) = f (| U _ {i} V _ {j} | / B) = \frac {| U _ {i} V _ {j} | ^ {p}}{B ^ {p}} - O \left(\frac {| U _ {i} V _ {j} | ^ {p + 1}}{B ^ {p + 1}}\right)} \\ { = } { \frac { | U _ { i } V _ { j } | ^ { p } } { B ^ { p } } - O \left( \frac { r ^ { p } } { B ^ { p + 1 } } \right) = h ( \tilde { U } _ { i } \tilde { V } _ { j } ) - O \left( \frac { r ^ { p } } { B ^ { p + 1 } } \right) . } \\ \end{array}
$$
where  $h(x) = |x|^p$ . Hence, if  $\tilde{U}', \tilde{V}'$  are such that
$$
\left\| \tilde {U} ^ {\prime} \cdot \tilde {V} ^ {\prime} - g (\tilde {U} \cdot \tilde {V}) \right\| _ {F} ^ {2} \leq \Delta \| [ g (\tilde {U} \cdot \tilde {V}) ] _ {k} - g (\tilde {U} \cdot \tilde {V}) \| _ {F} ^ {2},
$$
then by the triangle inequality,
$$
\begin{array}{l} \| \tilde {U} ^ {\prime} \cdot \tilde {V} ^ {\prime} - h (\tilde {U} \cdot \tilde {V}) \| _ {F} \leq \| g (\tilde {U} \cdot \tilde {V}) - h (\tilde {U} \cdot \tilde {V}) \| _ {F} + \sqrt {\Delta} \| [ g (\tilde {U} \cdot \tilde {V}) ] _ {k} - g (\tilde {U} \cdot \tilde {V}) \| _ {F} \\ \leq O \left(\frac {n r ^ {p}}{B ^ {p + 1}}\right) + \sqrt {\Delta} \| [ g (\tilde {U} \cdot \tilde {V}) ] _ {k} - g (\tilde {U} \cdot \tilde {V}) \| _ {F}. \\ \end{array}
$$
Setting  $U' = \sqrt{B} \cdot \tilde{U}'$  and  $V' = \sqrt{B} \cdot \tilde{V}'$  and scaling both sides by  $B$ , we have
$$
\begin{array}{l} \left\| U ^ {\prime} \cdot V ^ {\prime} - h (U \cdot V) \right\| _ {F} \leq O \left(\frac {n r ^ {p}}{B ^ {p}}\right) + B \sqrt {\Delta} \left\| [ g (\tilde {U} \cdot \tilde {V}) ] _ {k} - g (\tilde {U} \cdot \tilde {V}) \right\| _ {F} \\ \leq O \left(\frac {n r ^ {p}}{B ^ {p}}\right) + B \sqrt {\Delta} \| [ h (\tilde {U} \cdot \tilde {V}) ] _ {k} - g (\tilde {U} \cdot \tilde {V}) \| _ {F} \\ \leq O \left(\frac {n r ^ {p}}{B ^ {p}}\right) + B \sqrt {\Delta} \| [ h (\tilde {U} \cdot \tilde {V}) ] _ {k} - h (\tilde {U} \cdot \tilde {V}) \| _ {F} \\ + B \sqrt {\Delta} \| h (\tilde {U} \cdot \tilde {V}) - g (\tilde {U} \cdot \tilde {V}) \| _ {F} \\ = O \left(\frac {\sqrt {\Delta} n r ^ {p}}{B ^ {p}}\right) + B \sqrt {\Delta} \| [ h (\tilde {U} \cdot \tilde {V}) ] _ {k} - h (\tilde {U} \cdot \tilde {V}) \| _ {F} \\ = O \left(\frac {\sqrt {\Delta} n r ^ {p}}{B ^ {p}}\right) + \sqrt {\Delta} \| [ h (U \cdot V) ] _ {k} - h (U \cdot V) \| _ {F}, \\ \end{array}
$$
where in the second inequality we used that  $[h(\tilde{U} \cdot \tilde{V})]_k$  has rank  $k$  whereas  $[g(\tilde{U} \cdot \tilde{V})]_k$  is the best rank- $k$  approximation to  $g(\tilde{U} \cdot \tilde{V})$  in Frobenius norm. Note that the third line is triangle inequality and the fourth line follows our entrywise approximation bounds.
Now by AM-GM, using that  $a \leq b + c$  implies that  $a^2 \leq 2b^2 + 2c^2$  for  $a, b, c \geq 0$ , we have
$$
\| U ^ {\prime} \cdot V ^ {\prime} - h (U \cdot V) \| _ {F} ^ {2} = O \left(\frac {\Delta n ^ {2} r ^ {2 p}}{B ^ {2 p}}\right) + 2 \Delta \| [ h (U \cdot V) ] _ {k} - h (U \cdot V) \| _ {F} ^ {2}.
$$
Thus,  $U' \cdot V'$  is a rank- $k$  approximation to  $h(U \cdot V)$  with additive error  $\frac{1}{\mathrm{poly}(n)}$ , by setting  $B$  to be large enough, and relative error  $2\Delta$ . The total time to find  $U'$  and  $V'$  is the same as the time to solve entrywise transformed low rank approximation with respect to the function  $g$  on inputs  $\tilde{U}$  and  $\tilde{V}$ , and thus by Theorem 3.1 is at least  $(nd)^{1 - o(1)}$ . Note that applying the case  $U = V^\top$  in Theorem 3.1 establishes this theorem when  $\tilde{U} = \tilde{V}^\top$ .
Proof of Theorem 3.3. Suppose  $A$  and  $B$  are the input sets to the Apx-Max-IP $_{n,d}$  Problem of Assumption 4 with parameter  $s = r$ . We let the rows of  $U$  be the points in  $A$  and we let the columns of  $V$  be the points in  $B$ . As in the proof of Theorem 3.1, we have  $f(U_i^T V_j) = (U_i^T V_j)^p$  so  $f(U \cdot V) = U'' \cdot V''$ . By Remark 5, there are at most  $n^{o(1)}$  entries of  $f(U \cdot V)$  that are at least  $(m/100)^p$ .
Hence, if we set  $k$  to be an appropriate value in  $n^{o(1)}$ , then
$$
\left\| \left[ f (U \cdot V) \right] _ {k} - f (U \cdot V) \right\| _ {F} ^ {2} \leq (n d) (m / 1 0 0) ^ {2 p},
$$
since one possible rank- $k$  approximation is just to zero out the at most  $n^{o(1)}$  entries of  $f(U \cdot V)$  that are at least  $(m / 100)^{p}$ . By the correctness guarantee, with constant probability,
$$
\left\| f (U \cdot V) W W ^ {\top} - f (U \cdot V) \right\| _ {F} ^ {2} \leq \Delta \cdot (n d) (m / 1 0 0) ^ {2 p} = O (n d) (m / 1 0 0) ^ {2 p}. \tag {A.1}
$$
We may assume WLOG that  $p \geq C \log n$  for the moment, for a sufficiently large constant  $C > 0$  since a trivial lower bound for outputting an LRA is  $\Omega(n)$ , so the inclusion of the  $2^{\Omega(p)}$  term in the lower bound allows us to make this assumption. Therefore, we conclude that  $f(U \cdot V)$  is in the column span of  $W$  up to  $1 / \mathrm{poly}(n)$  error. Consequently, consider the maximum entry  $m$ , which we can assume is at least 1, as otherwise all points in  $A$  would have disjoint support from those in  $B$  but this can be verified in  $O(nr) = n^{1 + o(1)}$  time, contradicting the  $n^{1 + \Omega(1)}$  lower bound for  $d = n^{\Omega(1)}$  in Assumption 4.
It follows that if the maximum  $m$  occurs in the  $(i,j)$ -th entry of  $f(U\cdot V) = U''V''$ , then there exists a vector in the column span of  $U''$  for which the  $i$ -th entry is  $[m^p (1 - \mathrm{poly}(n)),m^p (1 + \mathrm{poly}(n))]$  and all other entries are in the range  $[-m^{p} / \mathrm{poly}(n),m^{p} / \mathrm{poly}(n)]$  since  $(1 / 100)^{p} = 1 / \mathrm{poly}(n)$ . Consequently, since the column span of  $W$  is close to the column span of  $U''$  up to  $1 / \mathrm{poly}(n)$  error, we see that the column span of  $W$  also contains these almost-sparse vectors that have  $n^{o(1)}$  large entries. As argued before similarly in Theorem 3.1, the rows of these large dot products forces the row leverage scores of  $W$  to be  $\Omega (1)$ . Since  $W$  has rank  $k = n^{o(1)}$ , in  $n^{1 + o(1)}$  time we can find the set  $S$  of  $n^{o(1)}$  row leverage scores of  $W$  that are  $\Omega (1)$ . We can then explicitly compute all dot products between pairs of vectors in  $A\cap S$  and  $B$  in  $n^{1 + o(1)}$  total time, at which point we can output the maximum dot product, which includes  $m$ . Therefore by Assumption 4, the total time to find  $W$  is at least  $(nd)^{1 - o(1)}$ .
Proof of Theorem 4.3. This proof follows by reducing low rank approximation to matrix multiplication by the same algorithm as in Theorem 4.1. Specifically, let us set  $k = n^{o(1)}$  and  $n = d$  and note that our low rank approximation algorithm's runtime is dominated by computing  $S \cdot f(UV)$  and  $f(UV) \cdot R$ , where  $S, R$  are matrices with  $\mathrm{poly}(k / \epsilon)$  rows and columns, respectively. In fact, those operations are the only terms that incur a polynomial dependence on  $n$ .
Therefore, suppose there exists such a matrix multiplication algorithm that takes time  $O(n^{2 - c})$  for some  $c$ . Then this would directly imply that computing both  $S \cdot f(UV), f(UV) \cdot R$  takes  $O(n^{2 - c})$  time as  $[Sf(UV)]^{\top} = f(V^{\top}U^{\top})S^{\top}$ . By our guarantees in Theorem 47 of Woodruff [2014], this implies a constant relative error LRA in time  $O(n^{2 - c})$ , which contradicts Theorem 3.2.
B Missing Upper Bound Proofs
Proof of Theorem 4.2. Again,  $f(U_i V_j) = (U_i V_j)^p$  so we may rewrite  $f(U \cdot V) = U'' V''$ , where each row of  $U'' \in \mathbb{R}^{n \times r^p}$  is the Khatri-Rao product of itself  $p$  times, and each column of  $V'' \in \mathbb{R}^{r^p \times d}$  is the Khatri-Rao product of itself  $p$  times.
Using Theorem 1 of Ahle et al. [2020], we see that with probability 0.9, our approximate matrix product guarantees hold for  $U''$ ,  $V''$  such that if  $\Pi$  is an  $m \times r^p$  TensorSRHT matrix with  $m = \Theta(p / \epsilon^2)$ , then
$$
\| U ^ {\prime \prime} \Pi^ {\top} \Pi V ^ {\prime \prime} - U ^ {\prime \prime} V ^ {\prime \prime} \| _ {F} \leq \epsilon \| U ^ {\prime \prime} \| _ {F} \| V ^ {\prime \prime} \| _ {F}
$$
Therefore, we can use the approximate low rank sketches again to approximately solve:
$$
\min  _ {U V} \| U ^ {\prime \prime} \Pi^ {\top} \Pi V ^ {\prime \prime} - U V \| _ {F} ^ {2}
$$
Specifically, let  $S$  be a random Gaussian sketching matrix with  $O(k / \epsilon)$  rows. We know that these matrices satisfy the  $(\sqrt{\epsilon / k}, 9 / 10, l)$ -JL property Woodruff [2014]. Furthermore, let  $R$  be a random
Gaussian matrix with  $O(m / \epsilon^2)$  columns, so we know that it is a  $(1 + O(\epsilon))\ell_2$  subspace embedding of the row space of  $SU''\Pi^\top$ , which has rank at most  $m$ . Let  $U''' = U''\Pi^\top$  and let  $\Pi V'' = V'''$ . Then, by Theorem 47 of Woodruff [2014], the following is true with constant probability
$$
\| (U ^ {\prime \prime \prime} V ^ {\prime \prime \prime} R) (S U ^ {\prime \prime \prime} V ^ {\prime \prime \prime} R) ^ {+} (S U ^ {\prime \prime \prime} V ^ {\prime \prime \prime}) - U ^ {\prime \prime \prime} V ^ {\prime \prime \prime} \| _ {F} \leq (1 + \epsilon) \cdot \min  _ {U, V} \| U V - U ^ {\prime \prime \prime} V ^ {\prime \prime \prime} \| _ {F}
$$
Finally we bound the runtime of computing this product. Note that we may compute  $U'''$  and  $V'''$  in time  $O(np(m + r))$  time. Then, computing  $SU'''$ ,  $V'''R$  can be done in  $nm \cdot (k / \epsilon + m / \epsilon^2)$  time. Lastly, the remaining products can be computed in  $\text{poly}(kp / \epsilon)$  and the final rank  $k$  decomposition can be computed and we can find  $U'$ ,  $V'$  such that
$$
\| U ^ {\prime} V ^ {\prime} - U ^ {\prime \prime \prime} V ^ {\prime \prime \prime} \| _ {F} \leq (1 + \epsilon) \min  _ {U, V} \| U V - U ^ {\prime \prime \prime} V ^ {\prime \prime \prime} \| _ {F}
$$
Now, let  $U^{\star}, V^{\star}$  be the optimal rank  $k$  decomposition of the original problem of  $f(UV)$ , then by the guarantees of approximate matrix product and the triangle inequality,  $\| U^{\star}V^{\star} - U''V'' \|_F \geq \| U^{\star}V^{\star} - U''\Pi^\top\Pi V'' \|_F - \epsilon \| U'' \|_F \| V'' \|_F$ .
$$
\begin{array}{l} \| U ^ {\prime} V ^ {\prime} - U ^ {\prime \prime} \Pi^ {\top} \Pi V ^ {\prime \prime} \| _ {F} \leq (1 + \epsilon) \| U ^ {\star} V ^ {\star} - U ^ {\prime \prime} \Pi^ {\top} \Pi V ^ {\prime \prime} \| _ {F} \\ \leq (1 + \epsilon) \| U ^ {\star} V ^ {\star} - U ^ {\prime \prime} V ^ {\prime \prime} \| _ {F} + 2 \epsilon \| U ^ {\prime \prime} \| _ {F} \| V ^ {\prime \prime} \| _ {F} \\ \end{array}
$$
We end by rewriting  $\| U''\| _F^2 = \mathrm{Tr}(U''U''^\top) = \sum_{i = 1}^n (U_i^\top U_i)^p = \sum_i\| U_i\| _2^{2p}$  and similarly for  $\| V''\| _F^2$  and then applying AM-GM and noting that  $(1 + \epsilon)^{2} = 1 + O(\epsilon)$ .