## 3 Material and Methods
### The Recommended Method
#### 3.3.3 Log-Leaky ReLU Activation  
Logarithmic approaches and computation-saving strategies like sparsity, pruning, and quantization have been suggested in the literature [31]. As the aggregation itself would become computationally expensive, the log-domain is used instead. The \(a\) constant from the leaky ReLU function has been changed to \(\beta\) in the log-leaky ReLU activation function [31]. As a result, a ReLU activation function in the log domain is obtained. It also has a single hyperparameter like the leaky-ReLU function. Figure 5 presents a mathematical illustration of the log-leaky ReLU activation function. \[g_{logLRelu}=((\mathsf{X},m_{x})|\beta)= \begin{cases}\mathsf{X},&if\ m_{x}=1 \\ \mathsf{X}+\beta,&if\ m_{x}=0\end{cases}  \tag{4}\]  
Here, \(m_{x}\) is a sign parameter that takes binary values 0 or 1 and determines the behavior of the activation function accordingly. When \(m_{x}=1\), the function returns the input \(X\); when \(m_{x}=0\), it returns \(\mathsf{X}+\beta\). This parameter is initialized independently and controls the switching in the log-leaky ReLU formulation. The sign parameter can be initialized independently of the 0.5 value derived from the Bernoulli distribution and is chosen at random. This can be determined using the weights in the distribution used by the activation function's magnitude value of (5). Equation (5) is derived from the recursive definition of the weighting function in the logarithmic activation framework. It defines the scaling behavior of \(f_{\omega}\) at discrete points \(k\) based on its values at \(2^{k}\). This ensures the preservation of logarithmic properties, contributing to numerical stability and smooth gradient propagation during training. \[f_{\omega}(k)=2^{k+1}(log_{e}2)f_{\omega}\left(2^{k}\right) \tag{5}\]  
The Bernoulli distribution (0.5) and log sizes are sampled from the distribution in (6) since we are sampling from a normal distribution with a mean value of 0 and a standard deviation of \(\sigma\). This means that the sampling parameters in the log domain are from a normal distribution [31]. The mathematical formulation in the Softmax layer is expressed as (7) and (8). \[f_{\omega}(k)=\frac{2 log_{e}2}{\sqrt{2\pi\sigma^{2}}}2^{k}e^{\frac{-2^{k}}{2 \sigma^{2}}} \tag{6}\]  
\[P_{ij}=\frac{e^{a_{ij}}}{\sum_{j=1}^{N}e^{a_{ij}}}=\frac{e^{a_{ij}}}{d_{i}} \tag{7}\]  
\[\delta_{ij}=P_{ij}-k_{ij} \tag{8}\]In the (8), \(\delta\) is the back-propagation initiation term and \(k\) is the only hot output label. Probability value calculation in log-domain is provided by (9)-(11). \[\left(D_{i},m_{d_{i}}\right)=\max\left(\sum_{j=1}^{N}\left(a_{ij}log_{2}e \right),1\right) \tag{9}\]  
\[\left(P_{ij},m_{p_{ij}}\right)=\left(\left(a_{ij}log_{2}e\right),1\right))\times \left(-D_{i},m_{d_{i}}\right) \tag{10}\]  
\[\left(log_{2}|\delta_{ij}|,m_{\delta_{i}}\right)=P_{ij}-\left(log_{2}|k_{ij}|,m _{k_{ij}}\right) \tag{11}\]  
To improve interpretability, we provide a visual comparison of ReLU, Leaky ReLU, and the proposed Log-Leaky ReLU in Figure 6. While ReLU outputs zero for negative values and Leaky ReLU allows small negative outputs, Log-Leaky ReLU introduces a logarithmic scaling that smoothly handles both positive and negative inputs, enhancing numerical stability and gradient flow. Log-Leaky ReLU logarithmically scales very small activations in the negative region, reducing issues like vanishing gradients. This enables more meaningful and balanced information flow in the negative domain during training. Additionally, the logarithmic growth for positive values prevents activations from becoming excessively large, further improving numerical stability. Thus, the model can learn more stably and precisely. The pseudocode of Log-DRSN is presented in Algorithm 1. Figure 5: Mathematical illustration of the log-leaky ReLU activation. Figure 6: Visual comparison of ReLU, Leaky ReLU, and proposed Log-Leaky ReLU activation functions.