# Trigonometric Inference Providing Learning in Deep Neural Networks
## 3 Methodology
### Forward and Backward Inference
#### 3.1.2 Backward Inference  
When training a neural network, the error \(E\) calculated from the loss function is backpropagated to each layer (error backpropagation) for weight update. The weights in each layer are then updated for narrowing the error \(E\). This parameter optimization problem can be solved using gradient descent, which requires calculating \(\frac{\partial E}{\partial w}\) for all \(w\) in the model. Here we denoted \(E_{j}\) as the backpropagated error signal from the latter layer to the layer \(j\). According to the chain rule, we can use Equation (6) to calculate the error signal \(E_{i}\) at layer \(i\), which is backpropagated from layer \(j\). For conventional CNNs, the error \(E_{i}\) is calculated by \(E_{j}\frac{\partial u_{i}}{\partial w_{i}}\), which equals the multiplication of \(E_{j}\) and \(x_{i}\) (\(\frac{\partial u_{i}}{\partial w_{i}}=x_{i}\)). \[E_{i}=\frac{\partial E}{\partial w_{i}}=\frac{\partial E}{\partial a_{i}}\frac {\partial a_{i}}{\partial w_{i}}=E_{i}\frac{\partial a_{i}}{\partial w_{i}} \tag{6}\]  
For our trigonometric inference, \(E_{i}\) is calculated differently since we use \(sine(x)\) and \(sine(W)\) to approximate \(x\) and \(W\), respectively, in the forward pass (as Equation (4)). Figure 3: Activation functions. The sine activation behaves similarly to the hyperbolic tangent activation for \(|x|<\pi/2\). By taking the partial derivative of \(a_{i}\) over \(w_{i}\), we get \(\frac{\partial a_{i}}{\partial w_{i}}=\sin(a_{i})\cos(w_{i})\). Therefore, the error \(E_{i}\) in trigonometric inference should be calculated as Equation (7) when the activation is larger than zero (Equation (5)). \[E_{i}=E_{j}\frac{\partial a_{i}}{\partial w_{i}}=E_{j}\sin(a_{i})\cos(w_{i}) \tag{7}\]  
Equation (7) gives the precise error signal \(\frac{\partial E}{\partial w_{i}}\) for weight update in the middle layer \(i\) during gradient descent. However, the direct calculation of Equation (7) will incur multiplication overheads, so here we developed two methods for computing \(E_{i}\). As shown in Table 1, the mean and variance of the weights and errors are very small in a downsized VGG of eight convolutional layers. This property, which makes the approximation possible, is common in modern deep neural networks. Experimental results show the trade-off between these two approaches; here, we outline the computation details:  
(1) Equation (7) is divided into two parts and approximately computed. Equation (8) indicates a recursive approximation computation (\(E_{j}\) is approximated by \(\sin(E_{j})\)). In this way, multipliers are not required, and precise error signals are calculated despite the computation complexity being increased. \[ \begin{array}{l}T_{j}=\sin\bigl{(}E_{j}\bigr{)}\sin(a_{i})\\ E_{i}=\sin\bigl{(}T_{j}\bigr{)}\cos(w_{i})\end{array}  \tag{8}\]  
(2) First, we replace \(E_{j}\) with \(\sin(E_{j})\) in Equation (7). The above approximation strategy enables us to compute gradients using Equation (9). \[\frac{\partial E}{\partial w_{i}}=\sin(E_{i})\sin(a_{i})\cos(w_{i}) \tag{9}\]  
Second, we use a pseudo-error signal for backpropagation. Because the cosine term in Equation (9) adds extra computational complexity, we replace it with shift operations. Since the values of the weights are generally highly concentrated around the value of zero, in an ideal situation we can simply omit the cosine term (\(\cos(x)\approx 1\) if \(x\to 0\)). However, in practice, we found that omitting the cosine term directly sometimes causes an excessive error, which is contributed by those weights of larger magnitude, so we replace the cosine term with \(1/2\). By simply scaling the original error signals to half, the algorithm provided learning in our experiments. The half-scaling strategy is formulated as in Equation (10): A similar concept was found in feedback alignment methods [29; 30] for error backpropagation. [PERSON] et al. [29] discovered that the weights do not have to be symmetric in the forward and backward inferences during the training.