# Orthogonal Neural Network: An Analytical Model for Deep Learning
## 2 Orthogonal Neural Network Model
### Operational Rules of ONN Model
The output of each computational unit is a linear combination of sine functions. With the above discussion on the extension of the activation function operation rules, we initially show the extension method. In order to further provide a normatively rigorous mathematical representation of the arithmetic process, we define a function set \(F\). \[F=\left\{\sum_{i}c_{i}\sin(it),c_{i}\in\mathbb{R},i\in\mathbb{N},i\in[1,n],t \in[0,2\pi]\right\} \tag{10}\]  
The function set \(F\) consists of linear combinations of sine functions with integer frequencies, where \(F\) satisfies \(\forall a,b\in\mathbb{R}\), \(f_{1},f_{2}\in F\), \(af_{1}+af_{2}\in F\). Thus, the function set \(F\) is complete. For the convenience of discussion, we define a size measure for \(f\in F\). Here, \(f\in F\) represents the linear superposition of a set of sine functions, corresponding to the deep learning computation, where the information modulated on the corresponding sine functions needs to be demodulated and summed. Combining the definition of the function set \(F\), we define the sum of the coefficients of the sine functions \(S(f)\) for \(f\in F\) in Equation (11). \[S(f)=\sum_{i=1}^{n}c_{i}=\frac{1}{\pi}\sum_{i=1}^{n}\int_{0}^{2\pi}f\cdot\sin( it)dt \tag{11}\]  
Here, we further provide a more normative definition of the ReLU activation function in Equation (12). The \(x_{i}\sin(it)\) and \(\sum_{i}c_{i}\sin(it)\) in the previous discussion belong to the function set \(F\). \[ReLU(f)=\left\{ \begin{array}{ll}f&if\ S(f)>0\\ 0&if\ S(f)\leq 0\end{array} \right. \tag{12}\]  
For a single-layer neural network containing \(n\) neurons, the input layer contains \(m\) features. The calculation rule is output \(Y=ReLU(X\cdot W)\), where \(X\in F^{1\times m}\) is a function matrix denoting the input layer of the neural network, \(W\in\mathbb{R}^{m\times n}\) is a numerical matrix denoting the parameters of that layer of the neural network, \(Y=F^{1\times n}\) is a function matrix denoting the output layer of the neural network, which is in Equation (13). \[Y_{1,j}=\sum_{i=1}^{m}X_{1,i}\cdot W_{i,j} \tag{13}\]  
Due to \(X_{1,i}\in F\), \(W_{i,j}\in\mathbb{R}\), obviously \(X_{1,i}\cdot W_{i,j}\in F\). For the calculation rules of multilayer neural networks, one layer contains \(n\) neurons. The output of the previous layer has \(m\) neurons, so the input of the layer is the output of the previous layer as \(X\in F^{1\times m}\) and \(W\in\mathbb{R}\) is the parameter matrix of the neural network of the current layer. The output is \(Y\in F^{1\times n}\), and \(Y=ReLU(X\cdot W)\) is consistent with the above. Multilayer neural networks [16] can be constructed according to such rules. Multilayer neurons are logically composed as a linear computational combination of multiple neurons. It should be emphasized here that their composition is characterized as a linear combination but, as a whole, does not satisfy the property of linearity. Since the activation function in a single neuron is nonlinear, the condition of linearity is not satisfied. Therefore, the combination of linear superposition of neurons also does not satisfy the conditions for linearity, but its calculation results in a linear combination of the input features. This phenomenon occurs because the ReLU function is a piecewise linear function and can therefore guarantee the properties of partial linear calculation under piecewise conditions. In typical deep learning models, multiple layers of neurons are generally used as fully connected layers in combination with other calculation units to build models capable of achieving complex learning functions. In convolutional neural networks [17], the basic calculation units are convolution, activation function, pooling, and multilayer neurons. Next, we discuss the convolutional neural network used in computer vision as an example. Convolution is one of the operations for extracting features in a convolutional neural network.