SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As explained in Section 3.1, as a base case we have an automatic procedure to compute the transpose or adjoint of any operator  $\mathbf{A}$  via autodiff. However, we also incorporate the following rules. For sum we have  $(\mathbf{A} + \mathbf{B})^{*} = \mathbf{A}^{*} + \mathbf{B}^{*}$  and  $(\mathbf{A} + \mathbf{B})^{\mathsf{T}} = \mathbf{A}^{\mathsf{T}} + \mathbf{B}^{\mathsf{T}}$ . For product we have  $(\mathbf{AB})^{*} = \mathbf{B}^{*}\mathbf{A}^{*}$  and  $(\mathbf{AB})^{\mathsf{T}} = \mathbf{B}^{\mathsf{T}}\mathbf{A}^{\mathsf{T}}$ . For Kronecker product we have  $(\mathbf{A}\otimes \mathbf{B})^{*} = \mathbf{A}^{*}\otimes \mathbf{B}^{*}$  and  $(\mathbf{A}\otimes \mathbf{B})^{\mathsf{T}} = \mathbf{A}^{\mathsf{T}}\otimes \mathbf{B}^{\mathsf{T}}$ . For the Kronecker sum we have  $(\mathbf{A}\oplus \mathbf{B})^{*} = \mathbf{A}^{*}\oplus \mathbf{B}^{*}$  and  $(\mathbf{A}\oplus \mathbf{B})^{\mathsf{T}} = \mathbf{A}^{\mathsf{T}}\oplus \mathbf{B}^{\mathsf{T}}$ . In terms of block composition we have
$$
\left(\left[ \begin{array}{c c} \mathbf {A} & \mathbf {B} \\ \mathbf {C} & \mathbf {D} \end{array} \right]\right) ^ {*} = \left[ \begin{array}{c c} \mathbf {A} ^ {*} & \mathbf {C} ^ {*} \\ \mathbf {B} ^ {*} & \mathbf {D} ^ {*} \end{array} \right] \quad \text {a n d} \quad \left(\left[ \begin{array}{c c} \mathbf {A} & \mathbf {B} \\ \mathbf {C} & \mathbf {D} \end{array} \right]\right) ^ {\intercal} = \left[ \begin{array}{c c} \mathbf {A} ^ {\intercal} & \mathbf {C} ^ {\intercal} \\ \mathbf {B} ^ {\intercal} & \mathbf {D} ^ {\intercal} \end{array} \right].
$$
Finally for the annotated operators we have the following rules.  $\mathbf{A}^{*} = \mathbf{A}$  if  $\mathbf{A}$  is self-adjoint and  $\mathbf{A}^{\top} = \mathbf{A}$  if  $\mathbf{A}$  is symmetric.
A.1.5 Pseudo-inverse
As a base case, if we need to compute  $\mathbf{A}^{+}$ , we may use  $\mathrm{SVD}(\mathbf{A}) = \mathbf{U}, \boldsymbol{\Sigma}, \mathbf{V}$  and therefore set  $\mathbf{A}^{+} = \mathbf{U} \boldsymbol{\Sigma}^{+} \mathbf{V}^{*}$ , where  $\boldsymbol{\Sigma}^{+}$ inverts the nonzero diagonal scalars. If the size of  $\mathbf{A}$  is too large, then we may use randomized SVD. Yet, it is uncommon to simply want  $\mathbf{A}^{+}$ , usually we want to solve a least-squares problem and therefore we can use solvers that are not as expensive to run as SVD. For the compositional operators we have the following identities. For product  $(\mathbf{AB})^{+} = (\mathbf{A}^{+} \mathbf{AB})^{+} (\mathbf{ABB}^{+})^{+}$ and for Kronecker product we have  $(\mathbf{A} \otimes \mathbf{B})^{+} = \mathbf{A}^{+} \otimes \mathbf{B}^{+}$ . For block diagonal we have
$$
\left(\left[ \begin{array}{c c} \mathbf {A} & \mathbf {0} \\ \mathbf {0} & \mathbf {D} \end{array} \right]\right) ^ {+} = \left[ \begin{array}{c c} \mathbf {A} ^ {+} & \mathbf {0} \\ \mathbf {0} & \mathbf {D} ^ {+} \end{array} \right].
$$
Finally, we have some identities that are mathematically trivial but that are necessary when recursively exploiting structure as that would save computation. For example, if  $\mathbf{Q}$  is unitary we know that  $\mathbf{Q}^{+} = \mathbf{Q}$  and similarly when  $\mathbf{Q}$  is orthonormal. If  $\mathbf{A}$  is self-adjoint, then  $\mathbf{A}^{+} = \mathbf{A}^{-1}$  and also if it is symmetric and PSD.
A.2 Derived Functions
Interestingly, the previous core functions allow us to derive multiple rules from the previous ones. To illustrate, we have that  $\operatorname{Tr}(\mathbf{A}) = \sum_{i}\operatorname{Diag}(\mathbf{A})_{i}$ . Additionally, if  $\mathbf{A}$  is PSD we have that  $f(\mathbf{A}) = \mathbf{V}_{\mathbf{A}}f(\boldsymbol{\Lambda}_{\mathbf{A}})\mathbf{V}_{\mathbf{A}}^{-1}$  and if  $\mathbf{A}$  is both symmetric and PSD then  $f(\mathbf{A}) = \mathbf{V}_{\mathbf{A}}f(\boldsymbol{\Lambda}_{\mathbf{A}})\mathbf{V}_{\mathbf{A}}^{\top}$ . where in both cases we used Eigs  $(\mathbf{A}) = \boldsymbol{\Lambda}_{\mathbf{A}},\mathbf{V}_{\mathbf{A}}$ . Some example functions for PSD matrices are  $\mathrm{Sqrt}(\mathbf{A}) = \mathbf{V}_{\mathbf{A}}\boldsymbol{\Lambda}_{\mathbf{A}}^{1 / 2}\mathbf{V}_{\mathbf{A}}^{-1}$  or  $\mathrm{Log}(\mathbf{A}) = \mathbf{V}_{\mathbf{A}}\log \boldsymbol{\Lambda}_{\mathbf{A}}\mathbf{V}_{\mathbf{A}}^{-1}$ . Which also this rules allow us to define  $\mathrm{LogDet}(\mathbf{A}) = \mathrm{Tr}(\mathrm{Log}(\mathbf{A}))$
A.3 Other matrix identities
We emphasize that there are a myriad more matrix identities that we do not intentionally include such as  $\operatorname{Tr}(\mathbf{A} + \mathbf{B}) = \operatorname{Tr}(\mathbf{A}) + \operatorname{Tr}(\mathbf{B})$  or  $\operatorname{Tr}(\mathbf{AB}) = \operatorname{Tr}(\mathbf{BA})$  when  $\mathbf{A}$  and  $\mathbf{B}$  are squared. These
additional cases are not part of our dispatch rules as either they are automatically computed from other rules (as in the first example) or they do not yield any computational savings (as in the second example).
B Features in CoLA
B.1 Doubly stochastic diagonal and trace estimation
Singly Stochastic Trace Estimator Consider the traditional stochastic trace estimator:
$$
\overline {{\operatorname {T r}}} [ \text {B a s e} ] (\mathbf {A}) = \frac {1}{n} \sum_ {j = 1} ^ {n} \mathbf {z} _ {j} ^ {\intercal} \mathbf {A} \mathbf {z} _ {j} \tag {1}
$$
with each  $\mathbf{z}_j\sim \mathcal{N}(\mathbf{0},\mathbf{I}_D)$  where  $\mathbf{A}$  is a  $D\times D$  matrix. When  $\mathbf{A}$  is itself a sum  $\mathbf{A} = \frac{1}{m}\sum_{i = 1}^{m}\mathbf{A}_{i}$ , we can expand the trace as  $\overline{\mathrm{Tr}} [\mathrm{Base}](\mathbf{A}) = \frac{1}{mn}\sum_{j = 1}^{n}\sum_{i = 1}^{m}\mathbf{z}_j^\top \mathbf{A}_iz_j$ , with probe variables shared across elements of the sum.
Consider the quadratic form  $Q \coloneqq \mathbf{z}^{\mathsf{T}}\mathbf{A}\mathbf{z}$ , which for Gaussian random variables has a cumulant generating function of  $K_{Q}(t) = \log \mathbb{E}[e^{tQ}] = -\frac{1}{2}\log \det(\mathbf{I} - 2t\mathbf{A})$ . From the generating function we can derive the mean and variance of this estimator:  $\mathbb{E}[Q] = K_{Q}'(0) = \operatorname{Tr}(\mathbf{A})$  and  $\operatorname{Var}[Q] = K_{Q}''(0) = 2\operatorname{Tr}(\mathbf{A}^2)$ . Since  $\overline{\operatorname{Tr}}[\operatorname{Base}](\mathbf{A})$  is a sum of independent random draws of  $Q$ , we see:
$$
\mathbb {E} \left[ \overline {{\operatorname {T r}}} [ \text {B a s e} ] (\mathbf {A}) \right] = \operatorname {T r} (\mathbf {A}) \quad \text {a n d} \quad \operatorname {V a r} \left[ \overline {{\operatorname {T r}}} [ \text {B a s e} ] (\mathbf {A}) \right] = \frac {2}{n} \operatorname {T r} \left(\mathbf {A} ^ {2}\right). \tag {2}
$$
Doubly Stochastic Trace Estimator For the doubly stochastic estimator, we choose probe variables which are sampled independently for each element of the sum:
$$
\overline {{\operatorname {T r}}} [ \operatorname {S u m} ] (\mathbf {A}) = \frac {1}{n m} \sum_ {j = 1} ^ {n} \sum_ {i = 1} ^ {m} \mathbf {z} _ {i j} ^ {\intercal} \mathbf {A} _ {i} \mathbf {z} _ {i j}. \tag {3}
$$
Separating out the elements of the sum, we can write the estimator as  $\overline{\mathrm{Tr}}[\mathrm{Sum}](\mathbf{A}) = \frac{1}{n}\sum_{j=1}^{n}R_{j}$  where  $R_{j}$  are independent random samples of the value  $R = \frac{1}{m}\sum_{i=1}^{m}\mathbf{z}_{i}^{\top}\mathbf{A}_{i}\mathbf{z}_{i}$ . The cumulant generating function is merely  $K_{R}(t) = \sum_{i=1}^{m}K_{Q_{i}}(t/m)$  where  $Q_{i} = \mathbf{z}^{\top}\mathbf{A}_{i}\mathbf{z}$ . Taking derivatives we find that,
$$
\mathbb {E} [ R ] = K _ {R} ^ {\prime} (0) = \frac {1}{m} \sum_ {i = 1} ^ {m} \operatorname {T r} \left(\mathbf {A} _ {i}\right) = \operatorname {T r} (\mathbf {A}), \tag {4}
$$
$$
\operatorname {V a r} [ R ] = K _ {R} ^ {\prime \prime} (0) = \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} 2 \operatorname {T r} \left(\mathbf {A} _ {i} ^ {2}\right) = \frac {2}{m} \operatorname {T r} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \mathbf {A} _ {i} ^ {2}\right) \tag {5}
$$
Assuming bounded moments on  $\mathbf{A}_i$ , then both  $\mathbf{A} = \frac{1}{m}\sum_{i}\mathbf{A}_{i}$  and  $S(\mathbf{A}) = \frac{1}{m}\sum_{i}\mathbf{A}_{i}^{2}$  will converge to fixed values as  $m\to \infty$ . Given that  $\overline{\mathrm{Tr}}[\mathrm{Sum}](\mathbf{A}) = \frac{1}{n}\sum_{j=1}^{n}R_{j}$ , we can now write the mean and variance of the doubly stochastic estimator:
$$
\mathbb {E} \left[ \overline {{\operatorname {T r}}} [ \operatorname {S u m} ] (\mathbf {A}) \right] = \operatorname {T r} (\mathbf {A}) \quad \text {a n d} \quad \operatorname {V a r} \left[ \overline {{\operatorname {T r}}} [ \operatorname {S u m} ] (\mathbf {A}) \right] = \frac {2}{m n} \operatorname {T r} (S (\mathbf {A})). \tag {6}
$$
As the error of the estimator can be bounded by the square root of the variance, showing that while the error for  $\overline{\mathrm{Tr}}[\mathrm{Base}]$  is  $O(1 / \sqrt{n})$  (even when applied to sum structures), whereas the error for  $\overline{\mathrm{Tr}}[\mathrm{Sum}]$  is  $O(1 / \sqrt{nm})$ , a significant asymptotic variance reduction.
The related stochastic diagonal estimator
$$
\overline {{\operatorname {D i a g}}} [ \operatorname {S u m} ] (\mathbf {A}) = \frac {1}{n m} \sum_ {j = 1} ^ {n} \sum_ {i = 1} ^ {m} \mathbf {z} _ {i j} \odot \mathbf {A} _ {i} \mathbf {z} _ {i j}. \tag {7}
$$
achieves the same  $O(1 / \sqrt{nm})$  convergence rate, though we omit this derivation for brevity as it is follows the same steps.
In Figure 5 we empirically how our doubly stochastic diagonal estimator outperforms the standard Hutchinson estimator.
Figure 5: Improved convergence of doubly stochastic diagonal estimator. Convergence of our doubly stochastic diagonal estimator in evaluating the diagonal of the UCI Buzz empirical covariance matrix (batch size = 100). Shown is the relative error of the estimate vs the number of passes through the  $n$  data points of the dataset. Our diagonal estimator has lower variance and converges faster than the standard Hutchinson estimator.
(a) Backwards pass runtime
(b) Backwards pass memory
Figure 6: Our autograd rules allow for fast and memory efficient backpropagation. For two different linear algebra operations  $\mathbf{A}_{\theta}^{-1}\mathbf{b}$  and  $\log |\mathbf{A}_{\theta}|$ , we show the runtime and peak memory utilization required to compute the derivatives as we increase the size of the problem. In all plots, we compare CoLA's autograd rules against the autograd default of backpropagating through each iteration of the solver (unrolled autodiff). Notably, using the custom autograd rules allows us to save substantial memory and runtime when performing the backwards pass.
B.2 Autograd rules for iterative algorithms
For machine learning applications, we want to seamlessly interweave linear algebra operations with automatic differentiation. The most basic strategy is to simply let the autograd engine trace through the operations and backpropagate accordingly. However, when using iterative methods like conjugate gradients or Lanczos, this naive approach is extremely memory inefficient and, for problems with many iterations, the cost can be prohibitive (as seen in Figure 6). However, the linear algebra operations corresponding to inverse, eigendecomposition and trace estimation have simple closed form derivatives which we can implement to avoid the prohibitive memory consumption and reduce runtime.
Simply put, for an operation like  $f = \mathsf{CGSolve}$ , CGSolve(A, b) = A $^{-1}$ b we must define a Vector Jacobian Product: VJP(f, (A, b), v) = (vT  $\frac{\partial f}{\partial A}$ , vT  $\frac{\partial f}{\partial b}$ ). However, for matrix-free linear operators, we cannot afford to store the dense matrix A, and thus neither can we store the gradients with respect to each of its elements! Instead we must (recursively) consider how the linear operator was constructed in terms of its differentiable arguments. In other words, we must flatten the tree structure of possibly nested differentiable arguments into a vector:  $\theta = \text{flatten}[A]$ . For example for A = Kron(Diag( $\theta_1$ ), Conv( $\theta_2$ )), flatten[A] = [\theta_1, \theta_2]. From this perspective, we consider A as a container or tree of its arguments  $\theta$ , and define vT  $\frac{\partial f}{\partial A} := \text{unflatten}[v^T \frac{\partial f}{\partial \theta}]$  which coincides with the usual definition for dense matrices. Applying to inverses, we can now write a simple VJP:
$$
\mathbf {v} ^ {\mathrm {T}} \frac {\partial f}{\partial \mathbf {A}} = \text {u n f l a t t e n} [ \operatorname {V J P} (\theta \mapsto \operatorname {u n f l a t t e n} (\theta) \mathbf {A} ^ {- 1} \mathbf {b}, \theta , \mathbf {A} ^ {- 1} \mathbf {v}) ] \tag {8}
$$
for  $\mathbf{v}^{\intercal}\frac{\partial f}{\partial\boldsymbol{\theta}} = \mathbf{v}^{\intercal}(\mathbf{A}^{-1})^{\intercal}(\partial_{\boldsymbol{\theta}}\mathbf{A}_{\boldsymbol{\theta}})\mathbf{A}^{-1}\mathbf{b}$ , and we will adopt this notation below for brevity. Doing so gives a memory cost which is constant in the number of solver iterations, and proportional to the memory used in the forward pass. Below we list the autograd rules for some of the iterative routines that we implement in CoLA with their VJP definitions.
1.  $\mathbf{y} = \operatorname{Solve}(\mathbf{A}, \mathbf{b})$ :  $\mathbf{w}^{\mathsf{T}} \frac{\partial \mathbf{y}}{\partial \theta} = -(\mathbf{A}^{-1} \mathbf{w})^{\mathsf{T}} (\partial_{\theta} \mathbf{A}_{\theta})(\mathbf{A}^{-1} \mathbf{b})$
2.  $\lambda, \mathbf{V} = \operatorname{Eig}\mathbf{s}(\mathbf{A})$ :  $\mathbf{w}^{\mathsf{T}}\frac{\partial\lambda}{\partial\theta} = \mathbf{w}^{\mathsf{T}}\operatorname{Diag}\bigl (\mathbf{V}^{-1}(\partial_{\theta}\mathbf{A}_{\theta})\mathbf{V}\bigr)$
3.  $\lambda, \mathbf{V} = \operatorname{Eig}\mathbf{s}(\mathbf{A})$ :  $\mathbf{w}^{\top} \frac{\partial \mathbf{v}_i}{\partial \theta} = \mathbf{w}^{\top} (\lambda_i \mathbf{I} - \mathbf{A})^{+} \partial_{\theta} \mathbf{A}_{\theta} \mathbf{v}_i$
4.  $y = \log |\mathbf{A}|:$ $\frac{\partial y}{\partial\pmb{\theta}} = \mathrm{Tr}\bigl (\mathbf{A}^{-1}\partial_{\pmb{\theta}}\mathbf{A}_{\pmb{\theta}}\bigr)$
5.  $\mathbf{y} = \mathrm{Diag}(\mathbf{A})$ :  $\mathbf{w}^{\mathsf{T}}\frac{\partial\mathbf{y}}{\partial\theta} = \mathbf{w}^{\mathsf{T}}\mathrm{Diag}\left(\partial_{\theta}\mathbf{A}_{\theta}\right)$
In Figure 6 we show the practical benefits of our autograd rules. We take gradients of different linear solves  $\mathbf{A}_{\theta}^{-1}\mathbf{b}$  that were derived using conjugate gradients (CG), where each solve required an increasing number of CG iterations.
C Algorithmic Details
In this section we expand upon three different points introduced in the main paper. For the first point we argue why SVRG leads to gradients with reduced variants. For the second points we display all the iterative methods that we use as base algorithms in CoLA. Finally, for the third point we expand upon CoLA's strategy for dealing with the different numerical precisions that we support.
C.1 SVRG
In simplest form, SVRG [23] performs gradient descent with the varianced reduced gradient
$$
\mathbf {w} \leftarrow \mathbf {w} - \eta \left(g _ {i} (\mathbf {w}) - g _ {i} \left(\mathbf {w} _ {0}\right) + g \left(\mathbf {w} _ {0}\right)\right) \tag {9}
$$
where  $g_{i}$  represents the stochastic gradient evaluated at only a single element or minibatch of the sum, and  $g(\mathbf{w}_0)$  is the full batch gradient evaluated at the anchor point  $\mathbf{w}_0$  which is recomputed at the end of each epoch with an updated anchor.
With different loss functions, we can use this update rule to solve symmetric or non-symmetric linear systems, to compute the top eigenvectors or even find the nullspace of a matrix. Despite the fact that the corresponding objectives are not strongly convex in the last two cases, it has been shown that
Symmetric Solve A w = b	Top-k Eigenvectors A W = W Λ	Nullspace A W = 0
gi(w)	Ai w - b	-Ai W + WW^T W [55]	Ai W [16]
gradient descent and thus SVRG will converge at this exponential rate [55, 16]. Below we list the gradients that enable us to solve different linear algebra problems: In each of the three cases listed above, we can recognize that if the average of all the gradients  $g(w)$  is 0, then the corresponding linear algebra solution has been recovered.
While it may seem that we need to take three complete passes through  $\{\mathbf{A}_i\}$  per SVRG epoch (due to the three terms in Equation 9), we can reduce this cost to two complete passes exploiting the fact that the gradients are linear in the matrix object, replacing  $\mathbf{A}_i\mathbf{W} - \mathbf{A}_i\mathbf{W}_0$  with  $\mathbf{A}_i(\mathbf{W} - \mathbf{W}_0)$  where appropriate. In all of the Sum structure experiments where we leverage SVRG, the x-axis measures the total number of passes through  $\{\mathbf{A}_i\}_{i=1}^m$ , two for each epoch for SVRG.
C.2 Iterative methods
In Table 5 we list the different iterative methods (base cases) that we use for different linear algebraic operations as well as for different types of linear operators. As seen in Table 5, there are many alternatives to our base cases, however we opted for algorithms that are known to be performant, that are well-studied and that are popular amongst practitioners. A comprehensive explanation of our bases cases and their alternatives can be found in Golub and Loan [21] and Saad [44].
Table 4: SVRG gradients for solving different linear algebra problems.
Linear Algebra Op	Base Case	Alternatives
Ax = b (non-symmetric)	GMRES	BiCGSTAB, LGMRES, QMR
Ax = b (self-adjoint)	MINRES	GMRES
Ax = b (PSD)	CG	GMRES
Eigs(A) (non-symmetric)	Arnoldi	IRAM, Bi-Lanczos
Eigs(A) (self-adjoint)	Lanczos	LOBPCG
A+	CG	LSQR, LSMR
A = UΣV*	Lanczos, rSVD	Jacobi-Davidson
f(A) (self-adjoint)	SLQ	Arnoldi
Table 5: CoLA's base case iterative algorithm and some alternatives. We now expand on the acronyms. GMRES: Generalized Minimum RESidual, BiCGSTAB: BiConjugate Gradient STABi-lized, QMR: Quasi-Minimal Residual, MINRES: MINimum RESidual, CG: Conjugate Gradients, IRAM: Implicitly Restarted Arnoldi Method, LOBPCG: Locally Optimal Block Preconditioned Conjugate Gradients, Bi-Lanczos: Bidiagonal Lanczos, CGS: Conjugate Gradient Squared, LSQR: Least squares QR, LSMR: Least squares Minimal Residual iteration, LGMRES: Least squares Generalized Minimum RESidual, rSVD: randomized Singular Value Decomposition, and SLQ: Stochastic Lanczos Quadrature.
C.3 Lower precision linear algebra
The accumulation of round-off error is usually the breaking point of several numerical linear algebra (NLA) routines. As such, it is common to use precisions like f1oat64 or higher, especially when running these routines on a CPU. In contrast, in machine learning, lower precisions like f1oat32 or float16 are ubiquitously used because more parameters and data can be fitted into the GPU memory (whose memory is usually much lower than CPUs) and because the MVMs can be done faster (the CUDA kernels are optimized for operations on these precisions). Additionally, the round-off error incurred on MVMs is not as detrimental when training machine learning models (as we are already
running noisy optimization algorithms) as when solving linear algebra problems (where round-off error can lead us to poor solutions). Thus, it is an active area of research in NLA to derive routines which utilize lower precisions than f1oat64 or that mix precisions in order to achieve better runtimes without a complete degradation of the quality of the solution.
In CoLA we take a two prong approach to deal with lower precisions in our NLA routines. First, we incorporate additional variants of well-known algorithms that propagate less round-off error at the expense of requiring more computation, as seen in Figure 7. Second, we integrate novel variants of algorithms that are designed to be used on lower precisions such as the CG modification found in Maddox et al. [32]. We now discuss the first approach.
As discussed in Section C.2, there are two algorithms that are key for eigendecompositions. The first is Arnoldi (applicable to any operator), and the second is Lanczos (for symmetric operators) — where actually Lanczos can be viewed as a simplified version of Arnoldi. Central to these algorithms is the use of an orthogonalization step which is well-known to be a source of numerical instability. One approach to aggressively ameliorate the propagation of round-off error during orthogonalization is to use Householder projectors, which is the strategy that we use in CoLA. Given a unitary vector  $\mathbf{u}$ , a Householder projector (or Householder reflector) is defined as the following operator  $\mathbf{R} = \mathbf{I} - 2\mathbf{u}\mathbf{u}^*$ . When applied to a vector  $\mathbf{x}$  the result  $\mathbf{R}\mathbf{x}$  is basically a reflection of  $\mathbf{x}$  over the  $\mathbf{u}^{\top}$  space. To easily visualize this, suppose that  $\mathbf{x} \in \mathbb{R}^2$  and  $\mathbf{u} = \mathbf{e}_1$ . Hence,
$$
\mathbf {R} \mathbf {x} = \left( \begin{array}{c} x _ {1} \\ x _ {2} \end{array} \right) - 2 \left( \begin{array}{c} x _ {1} \\ 0 \end{array} \right) = \left( \begin{array}{c} - x _ {1} \\ x _ {2} \end{array} \right)
$$
which is exactly the reflection of the vector across the axis generated by  $\mathbf{e}_2$ . Most notably,  $\mathbf{R}$  is unitary  $\mathbf{RR}^* = \mathbf{I}$  which can be easily verified from the definition. Being unitary is crucial as under the usual round-off error model, applying  $\mathbf{R}$  to another matrix  $\mathbf{A}$  does not worsen the already accumulated error  $\mathbf{E}$ . Mathematically,  $\| \mathbf{R}(\mathbf{A} + \mathbf{E}) - \mathbf{RA}\| = \| \mathbf{RE}\| = \| \mathbf{E}\|$ , where the last equality results from basic properties of unitary matrices. We are going to use Arnoldi as an example of how Householder projectors are used during orthogonalization. In Figure 7 we have an example of two different variants of Arnoldi present in CoLA. The implementations are notably different and also it is easy to see how Algorithm 2 is more expensive than Algorithm 1. First, note that for Algorithm 2 we have two for loops (line 6 and line 8) whereas for Algorithm 1 we only have one (line 4-6). Worse, the two for loops in Algorithm 2 require more flops than the only for loop in Algorithm 1. Note that we do not always favor the more expensive but robust implementation of an algorithm as in some cases, like when running GMRES, the round-off error is not as impactful to the quality of the solution, and shorter runtimes are actually more desirable.
D Experimental Details
In this section we expand upon the details of all the experiments ran in the paper. Such details include the datasets that were used, the hyperparameters of different algorithms and the specific choices of algorithms used both for CoLA but also for the alternatives. We run each of the experiments 3 times and compute the mean dropping the first observation (as usually the first run contains some compiling time much is not too large). We do not display the standard deviation as those numbers are imperceptible for each experiment. In terms of hardware, the CPU experiments were run on an Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz and the GPU experiments were run on a NVIDIA GeForce RTX 2080 Ti.
D.1 Datasets
Below we enumerate the datasets that we used in the various applications. Most of the datasets are sourced from the University of California at Irvine's (UCI) Machine Learning Repository that can be found here: https://archive.ics.uci.edu/ml/datasets.php. Also, a community repo hosting these UCI benchmarks can be found here: https://github.com/treforevans/uci_datasets (we have no affiliation).
1. Elevators. This dataset is a modified version of the Ailerons dataset, where the goal is to predict the control action on the ailerons of the aircraft. This UCI dataset consists of  $N = 14\mathrm{K}$  observations and has  $D = 18$  dimensions.