Title: Binary Oscillation-Regulated Network (BORN): Approach for Binary Neural Networks Training
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Binary neural network, Adaptive sign approximation, Adaptive learning rate

Abstract: Binary Neural Networks (BNNs) are gaining attention for making energy-intensive deep learning more accessible to resource-constrained edge devices. Traditionally, training methods for such models focus on minimizing quantization error in forward propagation and approximating the $sign$ function for gradient computation. However, these methods overlook the BNN-specific phenomenon of oscillations and do not adapt the learning rate to the training dynamics. To address this, we propose BORN, which adapts learning to oscillatory behavior. The proposal is based on two key innovations: an Oscillations-aware Sign Approximation, which reduces gradient information loss by gradually approximating the $sign$ function in backward propagation, and an adaptive learning rate adjusted according to progress and oscillations. Beyond that, we are the first to combine a static sign approximation for activations with a dynamic one for weights, which stabilizes optimization while maintaining flexibility. Moreover, unlike prior work that chose approximations solely to replicate the sign function as closely as possible, we are the first to motivate the choice of approximating functions through the properties they induce for binary optimization, providing a principled foundation for their design. The novelty of BORN lies not only in its BNN-specific adaptive mechanisms but also in its ease of integration into existing architectures such as convolutional and transformer networks. To validate this, we integrate the method into several state-of-the-art (SoTA) BNN training frameworks across tasks like super-resolution (SR) and large language modeling (LLM). Experimental results show that the proposed method can cover an average of 34.81% of the proximity interval from SoTA methods to full-precision models.