Title: One Reflection at a Time: Provable Compression and Denoising of Orthogonal Matrices
Primary area: learning theory
Keywords: orthogonal matrices, Householder reflections, matrix approximation, compression, denoising, eigenspace methods, recurrent neural networks, vision transformers, optimization

Abstract: Orthogonal matrices have become central to modern machine learning for several reasons, including gradient norm preservation, allowing invertible transformations, and supporting efficient parameterization. While any orthogonal matrix can be written as a product of Householder reflections, existing constructive methods do not address approximation, denoising, or reflector minimality under noise and computational constraints. We propose a simple greedy eigenspace-based algorithm that approximates or recovers an orthogonal matrix $\mathbf{V}$ using a product of Householder reflections. Our method (i) preserves orthogonality at every approximation level $K$, (ii) yields provable per-iteration error bounds, (iii) terminates exactly with a minimality certificate, and (iv) has denoising guarantees under additive random noise (Ginibre/GOE). Empirically, our approach outperforms previous methods across reconstruction error metrics. Experimental ablations on expRNN and ViT show controllable accuracy-compute trade-offs when replacing learned weights with $K$-Householder products, with storage and matvec costs reduced from $\mathcal{O}(n^2)$ to $\mathcal{O}(Kn)$. Our results position greedy Householder products as a theoretically grounded tool for projection, compression, and denoising of orthogonal matrices in learning systems.

TL;DR: We propose a greedy eigenspace algorithm that approximates and denoises orthogonal matrices with minimal Householder reflections, offering error bounds, noise robustness, and efficiency for compressing orthogonal layers in RNNs and Transformers.