# Orthogonal Neural Network: An Analytical Model for Deep Learning
## 2 Orthogonal Neural Network Model
### Summary of the Model  
In the previous discussion, the features input into the depth learning model are loaded onto different bases by introducing a set of orthogonal bases, and the orthogonal relationship between the bases is used to ensure the input features remain independent of each other. Further, according to the numerical computation relationship of each feature in the original model, the corresponding operation rules are defined in the space composed of this set of orthogonal bases, which is the extension discussed for the deep learning model. From the perspective of numerical calculation, the two are entirely consistent, but the latter can separate the features on each based on the orthogonal relationship between the bases.  
Most of the computational units of the deep learning model are linear, so the computation between features and model parameters is actually a linear operation on the real number field. In the expanded deep learning model, the features exist as vectors in space, and their operation with the model parameters becomes a linear operation between the vectors in space.  
Regarding the two typical nonlinear operation units, activation function and maximum pooling, the operation rules in vector set \(E\) can still be expanded and defined through the analysis of their operation logic and the use of their \"partially linear\" nature; that is, the vectors involved in the operation can be linearly expressed with an orthogonal basis. Therefore, we can extend the deep learning model to operate in the vector set \(E\).  
It should be emphasized again that the operation here maintains the property that the output results of the computational units can all be expressed as linear combinations of orthogonal bases, but is not a linear system and does not satisfy the definition of a linear system. On the other hand, if the system is linear, the deep learning model does not have the problem of insufficient interpretability, and the linear system can directly solve the system function using impulse response methods.  
The most important feature of the discussion in this paper is the introduction of the orthogonal function set and orthogonal vector set to extend the deep learning model with operational rules and introduce the concept of modulation and demodulation, so the extended model is called **orthogonal neural network** (ONN) in this paper. The process of loading features onto an orthogonal basis is expressed as orthogonal modulation of features. The definition of the model is shown in Figure 1, which includes the modulated data, the parameters of the deep learning model, and the algorithmic rules defined in the extension in this section. Then, we can compute the analysis data of the classification results based on the orthogonal neural network model by demodulating values on the orthogonal bases. The ONN model can be applied to almost any deep learning model that uses the ReLU activation function. In the following, we detail the applications of the ONN model on two typical deep learning models: convolutional neural network and graph neural network.  
Figure 1: Orthogonal neural network model.