# Orthogonal Neural Network: An Analytical Model for Deep Learning
## 2 Orthogonal Neural Network Model
### Modulation and Demodulation on Orthogonal Bases  
To simplify the description of the problem, we introduce here for discussion a function set composed of sine functions on \(t\in[0,2\pi]\) in Equation (2). The \"\(n\)\" in Equation (2) is the number of input features.  
\[\{\sin(t),\sin(2t),\cdots,\sin(nt)\} \tag{2}\]  
The sine functions in this function set are orthogonally, as in Equation (3).  
\[\frac{1}{\pi}\int_{0}^{2\pi}\sin(nt)\sin(mt)dt=\left\{\begin{array}{ll}1,n=m \geq 1\\ 0,\quad\text{other}\end{array}\right. \tag{3}\]  
For a \"modulation\" process, modulating \(x_{1}\) to a sine wave of frequency \(n\) can be expressed as Equation (4).  
\[x_{1}\to x_{1}\sin(nt) \tag{4}\]  
The amplitude modulation of the sine wave with a value \(x_{1}\) causes the value \(x_{1}\) to be loaded onto the sine wave, which is the amplitude information after modulation. \"Demodulation\" is the inverse process of \"modulation\". For a \"demodulation\" process, demodulating \(x_{1}\) from a sine wave of frequency \(n\) can be expressed as (5).  
\[x_{1}\sin(nt)\xrightarrow[\frac{1}{\pi}\int_{0}^{2\pi}x_{1}\sin(nt)\sin(nt)dt] \xrightarrow[]{}x_{1} \tag{5}\]  
Here, modulation and demodulation are entirely defined by computation to separately modulate all values at the input of a deep learning model to different frequencies so that these values can exist independently during the computation. Such a process is implemented in a communication system using a circuit whose corresponding mathematical description differs from this one. However, the computational result is the same; here, it is fully implemented by computation.  
For a value \(x_{i}\in\mathbb{R}\), given as input to the deep learning model, the operations involved are mainly multiplication and addition, i.e., linear operations. In standard models, the computational units that generate nonlinearity are mainly the activation function and maximum pooling, which are characterized by selection of the linear computation results in other links. Although understood as nonlinear in terms of computational properties, the output of their operations is still the result of linear computation of the values of model input. The values input to the deep learning model are denoted as \(\{x_{i}\mid i\in\mathbb{N},1\leq i\leq N\}\), for each computational session, the values in the result can be expressed as \(\sum_{i=1}^{N}w_{i}x_{i}\), \(w_{i}\) is the weight corresponding to \(x_{i}\), and \(w_{i}=0\) if the computational result does not contain the input \(x_{i}\).  
Therefore, by assigning a frequency to each value input in the deep learning model and modulating it according to specific rules, the values generated by each computational link in the deep learning model become a linear superposition of a set of sine functions, which can be expressed as \(\sum_{i=1}^{N}w_{i}x_{i}\sin(it)\). When it is necessary to extract an \(x_{j}\) of interest, \"demodulation\" can be computationally implemented, such as in Equation (6).  
\[\frac{1}{\pi}\int_{0}^{2\pi}\sum_{i=1,j\leq j\leq N}^{N}w_{i}x_{i}\sin(it)\sin( jt)dt=w_{j}x_{j} \tag{6}\]  
The conclusion from calculating the orthogonality of sinusoidal functions is that the summation is nonzero only when \(i=j\), which means that \(x_{j}\) can be demodulated only by using \(\sin\left(jt\right)\). Through the above modulation and demodulation computation process, each input value to the deep learning model is actually assigned a \"frequency\" label, which can be used to extract the value corresponding to the initial input value at any time during model operation, and this label has a clear physical meaning and is easy to compute.