# Orthogonal Neural Network: An Analytical Model for Deep Learning
## 2 Orthogonal Neural Network Model
### Further Optimization for Orthogonal Bases  
In the previous discussion, inspired by the frequency division multiplexing technology in communication technology, we introduced the concepts of \"frequency\", \"modulation\", and \"demodulation\" and extended the operation rules of the deep learning model. This discussion revolves around the physical meaning of sine function and frequency. From a mathematical point of view, the sine function is introduced to form a function set and the elements in this function set are orthogonal to each other on the interval \([0,2\pi]\). The sine function on the interval \([0,2\pi]\) is taken as a set of orthogonal bases to form a space and further extend the corresponding definition of the operation rules of the deep learning model on this space. However, what needs to be further discussed is that from the perspective of concrete calculation in the computer, the continuous function in an interval is a sampled discrete sequence on the computer. When the sampling interval is \(T\) in the interval \([0,2\pi]\), there are \(K=\frac{2\pi}{T}\) sampling points, so the sequence sampled by \(\sin(nkT)\) is \(\{\sin(nkT),0\leq k\leq K\}\), and the sampled sequence has the same orthogonal property, which is in Equation (17). \[\frac{1}{\pi}\sum_{k=0}^{K}\sin(nkT)\sin(mkT)=\left\{ \begin{array}{ll}1,&if \;\;n=m\geq 0\\ 0,&if\;\;\;\;\;other\end{array} \right. \tag{17}\]  
Therefore, when performing operations on the computer, we actually use a set of orthogonal sequences after sampling the sine function, which is in Equation (18). \[\left\{\left\{\sin(kT)\right\}_{k=0^{*}}^{K},\left\{\sin(2 kT)\right\}_{k=0^{ *}}^{K},\cdots,\left\{\sin(nkT)\right\}_{k=0}^{K}\right\} \tag{18}\]  
When \(n\) features need to be modulated, \(n\) frequencies need to be allocated and generated. In order to ensure that the sine functions of these \(n\) frequencies do not produce frequency aliasing, when the maximum frequency is \(n\), it is necessary to satisfy the Nyquist sampling theorem when discretizing, which in this case means that a sequence of at least \(2n\) points is needed to represent the sine functions on the interval. In the scenario of deep learning, this \(n\) is always enormous. When allocating \(2n\) values of storage space for each frequency, \(2n^{2}\) floating point values need to be allocated to memory, which will cause considerable resource consumption. At the same time, there will also be quantization errors in the process of discrete sampling of continuous functions. After entering the deep learning algorithm, the cumulative errors resulting from these quantization errors will affect its accuracy. Therefore, the space constructed by the sine function on the interval \([0,2\pi]\) as the orthogonal basis has clear physical significance, but it is challenging to realize in practical applications. Further, consider constructing a set of orthogonal bases from a mathematical perspective and the equivalent, realizing the theoretical process discussed in the previous article. The most common orthogonal basis in mathematics is the standard orthogonal basis of Euclidean space, which can be expressed as an orthogonal vector set \(\mathcal{A}=\left\{\mathbf{\alpha}_{1},\mathbf{\alpha}_{2},\cdots,\mathbf{\alpha}_{n}\right\}\) and (19). The \"\(n\)\" is the number of input features which is consistent with Equation (2). \[ \begin{array}{l}\mathbf{\alpha}_{1}:[1,0,0,\cdots,0]\\ \mathbf{\alpha}_{2}:[0,1,0,\cdots,0]\\ \vdots\\ \mathbf{\alpha}_{n}:\left[\underbrace{0,0,0}_{n-1},\cdots\right]\end{array}  \tag{19}\]  
The use of the orthogonal basis is not only simple and elegant in form, but can also allow directly avoiding the quantization error generated by digital sampling of the sine function when the sine function is used as the orthogonal basis. The position index of the nonzero item in the base vector also completely corresponds to the base number. Using sparse data structures for storage can significantly save the storage space and improve the operation efficiency.