Title: ORIENT: A Rank-Adaptive, Orthogonality-Preserving Neural Architecture
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Givens Rotation, Singular Value Decomposition

Abstract: In this paper, we propose a novel neural network architecture ORIENT (ORthogonality-Intrinsic Elastic NeTwork), which is promising for various tasks including continual learning, light-weighting and so on.
ORIENT is designed to intrinsically maintain strict orthogonality throughout training by parameterizing Givens rotations.
Combined with a learnable diagonal scaling component inspired by singular value decomposition, the architecture provides a compact and flexible factorization of weight matrices.
It achieves high parallelism by arranging mutually non-interfering rotations at each layer, and supports rank-adaptive model construction by allowing selective activation of Givens rotations.
These properties enable ORIENT to efficiently represent low-rank approximations with a fixed structure. Hence, ORIENT is suitable for scenarios that require dynamic model reconfiguration.
In the experimental evaluation, we show that ORIENT matches the performance of standard fully connected layers in the full-rank setting, while enabling smooth accuracy control through rank adjustment for light-weighting.
In a continual learning task, progressive rank expansion allows effective capacity growth while retaining previously learned knowledge, demonstrating the model's adaptability.