## 5 Ablation Studies and Comparative Analysis
### Activation Functions  
To improve our models even further, we investigate various activation functions and their impact on model accuracy. We employ the ReLU (Rectified Linear Unit) activation function [91], as shown in Equation (17) and Figure 9. However, ReLU activation functions can result in dead neurons, where weights less than zero prevent the model from properly fitting the data.  
\[\text{ReLU(x)}=\begin{cases}x&\text{if }x>0,\\ 0&\text{if }x\leq 0,\end{cases} \tag{17}\]  
To mitigate this, we explore PReLU (Parametric ReLU), which introduces a learnable parameter updated during training, as shown in Equation (18) and Figure 9.  
\[\text{PReLU(x)}=\begin{cases}x&\text{if }x_{i}>0,\\ a_{i}x_{i}&\text{if }x_{i}\leq 0,\end{cases} \tag{18}\]  
Additionally, we utilize the Mish activation function, which is a non-monotonic function that has been shown to improve model expressivity and regularization [92]. The Mish activation function is defined as follows:  
\[\text{Mish(x)}=x\cdot\tanh(\text{softplus}(x)) \tag{19}\]  
where the Tanh function is given by  
\[\text{Tanh(x)}=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \tag{20}\]  
and the Softplus function is  
\[\text{Softplus(x)}=\ln(1+e^{x}) \tag{21}\]  
Figure 9 illustrates the connections for a convolutional layer using different activation functions.  
When these three different activation functions are applied to binary neural network models, the PReLU activation function with orthogonal initialization yields the best results for AlexNet and ResNet-20.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & _Top-1 (\%)_ & _Param Size (MB)_ & _Compression_ \\ \hline [PERSON] & 81.79\% & 2.85 & - \\ [PERSON] & 81.70\% & - & - \\ \hline Orthogonal & 82.01\% & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 8: Impact of initialization on the AlexNet architecture on CIFAR-10.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & _Top-1 (\%)_ & _Param Size (MB)_ & _Compression_ \\ \hline [PERSON] & 81.87\% & 0.047 & - \\ [PERSON] & 81.24\% & - & - \\ Orthogonal & 82.55\% & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 9: Impact of initialization on the ResNet-20 architecture on CIFAR-10.  
**Analysis**: Tables 10 and 11 show that the PReLU activation function provides the highest accuracy across both models, increasing AlexNet's accuracy to 84.01% and ResNet-20's accuracy to 83.93%. This is primarily due to PReLU's ability to avoid dead neurons, allowing the network to learn more effectively compared to the traditional ReLU activation function. The Mish activation function also performs well, particularly for deeper networks like ResNet-20, achieving 82.30% top-1 accuracy.  
When we apply our methods using orthogonal initialization, PReLU activation, and sensitivity-based rank selection, we achieve a highly compressed model with competitive accuracy, as shown in Table 14. For instance, our proposed method reduces the size of the ResNet-20 model from 1.1 MB to 0.034 MB with a compression ratio of approximately 32\(\times\) while preserving a top-1 accuracy of 81.89%. Compared to ResNet20-XNOR, which achieves an accuracy of 83.93% with a parameter size of 0.047 MB, our method offers a smaller model size with only a marginal drop in accuracy. Furthermore, our approach outperforms other lightweight models such as MobileNet and MobileNetV2, which have parameter sizes of 12.4 MB and 9.0 MB, respectively, and higher accuracy but at the cost of significantly larger model sizes. Overall, our method demonstrates a more effective balance between compression and accuracy, making it particularly suitable for deployment on resource-constrained devices like edge and IoT systems, where both memory and computational resources are limited.