# Orthogonal Neural Network: An Analytical Model for Deep Learning
## 2 Orthogonal Neural Network Model
### Operational Rules of ONN Model  
Application of frequency modulation to the values input into the deep learning model is equivalent to converting the input values into a function, so it is necessary to extend the computational rules of the deep learning model such that it can satisfy both the operation of the function and the computational logic in being consistent with the original numerical computation. In the following discussion, extension of the operation rules is defined for each calculation unit. First, we discuss the case of a single neuron, which consists of an affine function and an activation function. For \(n\) numeric features, \(\mathbf{x}=\left\{x_{1},x_{2},\cdots,x_{n}\right\}\), the weights corresponding to the features are \(\mathbf{w}=\left\{w_{1},w_{2},\cdots,w_{n}\right\}\), and \(b\) is the bias and can be defaulted. Thus, the affine function is \(a=\sum_{i=1}^{n}w_{i}x_{i}+b=\mathbf{w}\mathbf{x}^{T}+b\). Then, over the activation function, it constitutes a single neuron, which can be expressed as (7). As explained at the beginning of the article, the activation function is limited to the ReLU function. \[ReLU(a)=\left\{ \begin{array}{ll}a&if\ a\geq 0\\ 0&if\ a<0\end{array} \right. \tag{7}\]  
We perform frequency modulation for the numerical elements in \(\mathbf{x}\). For the sake of discussion, the frequency corresponding to each element is the subscript of that element. \(\mathbf{x}_{mod}=\left\{x_{i}\sin(it)\mid x_{i}\in\mathbf{x}\right\}\), and the frequency assigned for the bias \(b\) assignment is \(n+1\) such that the affine function becomes \(a_{mod}=\sum_{i=1}^{n}w_{i}x_{i}\sin(it)+b\sin((n+1)t)=\mathbf{w}\mathbf{x}_{ mod}^{T}+b\sin((n+1)t)\). It can be seen that after modulation of the numerical elements, the calculation of the affine function changes from a numerical value to a function containing a variable \(t\), which is only useful in the demodulation session and, therefore, is not additionally written. Since the calculation result of the affine function changes from a numerical value to a function, it is necessary to extend the definition of the operation rules for the activation function, which is in Equation (8). \[ReLU_{mod}(a_{mod})=\left\{ \begin{array}{ll}a_{mod}&if\ a\geq 0\\ 0&if\ a<0\end{array} \right. \tag{8}\]  
The judgment condition is the same as the original activation function. Due to the premise of frequency modulation, we can obtain \(a\) by demodulating the values in \(a_{mod}\) by Equation (9). \[a=\sum_{i=1}^{n+1}\int_{0}^{2\pi}a_{mod}\sin(it)dt \tag{9}\]  
Here, we can figuratively understand the extended definition of the activation function as a switch function, and the affine function calculates \(a_{mod}\) as a signal. When \(a>0\), the switch opens the signal through; when \(a\leq 0\), the switch closes the signal blocking. There are differences in the definition of the activation function for the judgment condition when \(a=0\). In the original activation function, the \(a=0\) condition can be included in either the greater- or less-than conditions. In the extended definition of the activation function, the \(a=0\) condition can only be included in the less-than condition because consistency with the original definition must be guaranteed in the numerical computation logic of the extended definition. In the original definition of the condition \(a=0\), the activation function outputs \(a=0\). In the extended definition, if the equal condition is included in the greater-than condition, the activation function outputs \(a_{mod}\), which is inconsistent with the computational logic of the original activation function. When the neuron is not in the input layer, each output of the previous layer is a linear combination of sine functions, which can be expressed as \(\sum_{i}c_{i}\sin(it)\), and the modulated \(\mathbf{x}_{mod}=\{x_{i}\sin(it)\mid x_{i}\in\mathbf{x}\}\) is not essentially different, but is a linear combination of sine functions, so it still satisfies the above discussion of affine functions and activation functions.