Title: ECAttention: Adaptive and Principled Feature Expansion-Compression with Linear Efficiency
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: white-box deep neural networks, projection, compression, expansion

Abstract: The maximal coding rate reduction ($\text{MCR}^2$) objective has been proposed to learn low-dimensional subspace representations by minimizing the compression term for intra-group compactness and maximizing the expansion term for inter-group separation.Several studies have leveraged $\text{MCR}^2$ to design principled, interpretable deep models 
by following or approximating its gradient to derive layer structures.However, these approaches remain limited in achieving fully principled and effective compression and lack self-adaptive control over the strength of expansion and compression across layers.In this work, we introduce \textbf{ECAttention}, a novel attention mechanism that incorporates principled expansion and compression modules inspired by the \textbf{geometric insight} of $\text{MCR}^2$.
Geometrically, gradient-based updates of  $\text{MCR}^2$  move features along directions shaped by the underlying data structure.
Our method efficiently captures this structure using randomization combined with Cholesky decomposition to guide feature updates with \textbf{nearly linear complexity}.By introducing two trainable weights per layer, ECAttention self-adaptively regulates the strengths of compression and expansion.The resulting ECA transformer not only matches or surpasses prior methods, but also exhibits greater interpretability, with different heads focusing on distinct image regions and capturing \textbf{fine-grained structures} under simple supervised training.

TL;DR: A principled attention with better interpretability that achieving feature compression or expansion  based on the  geometric insight of gradient of MCR^2