While the orthonormal regularizer in SORSA provides significant benefits in maintaining the orthonormality of singular vectors, there are potential limitations and drawbacks that need to be considered:
Computational Overhead: The orthonormal regularizer introduces additional computational complexity during training, as it requires the calculation of the Frobenius norm of the orthonormality constraints. This can lead to increased training time, especially for large models. To address this, one could explore more efficient implementations of the regularizer, such as approximating the orthonormality constraints or using lower-dimensional representations to reduce computational costs.
Sensitivity to Hyperparameters: The effectiveness of the orthonormal regularizer is sensitive to the choice of hyperparameters, particularly the scaling factor (γ). If γ is too large, it may dominate the training process, leading to suboptimal convergence. Conversely, if it is too small, the regularizer may not effectively maintain orthonormality. To mitigate this issue, adaptive learning rate techniques could be employed, allowing the model to dynamically adjust γ based on training progress and convergence behavior.
Potential Overfitting: While the regularizer aims to stabilize training, there is a risk that it may inadvertently lead to overfitting, especially in scenarios with limited data. To counteract this, one could implement early stopping criteria or incorporate dropout techniques alongside the regularizer to enhance generalization.