# Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations
## 2 Separable-Gaussian Neural Networks  
**Definition 1**.: _A \(d\)-variate function \(f(x_{1},x_{2},\ldots,x_{d})\) is separable if it can be expressed as a product of multiple uni-variate functions; i.e.,_  
\[f(x_{1},x_{2},\ldots,x_{d})=f_{1}(x_{1})\cdot f_{2}(x_{2})\cdots\cdot f_{d}(x_ {d}). \tag{3}\]  
**Remark 1**.: _Recall that the Gaussian radial-basis function in Equation (2) is separable and can be represented in the form_  
\[G(\mathbf{x})=\prod_{k=1}^{d}\varphi^{(k)}(x_{k}), \tag{4}\]  
_where \(\varphi^{(k)}(x_{k})=\exp(-\frac{1}{2}(x_{k}-\mu_{k})^{2}/\sigma_{k}^{2})\), with \(k=1,2,\ldots,d\)._  
The product chain in Equation (4) can be constructed through the forward propagation of a feedforward network with a single neuron per layer where \(\varphi^{(k)}(x_{k})\) is the neuron of the \(k\)-th layer. This way, the multi-variate Gaussian function \(G(\mathbf{x})\) is reconstructed at the output of the network. By adding more neurons to each layer and assigning weights to all edges, we can eventually construct a network whose output is equivalent to the output of a GRBFNN. Figure 1 shows an example of an SGNN approximating a tri-variate function.  
Next, we use this property to define SGNN.  
Figure 1: The general structure of SGNNs. In this paper, the weights of the output layer are in unity. The distinct feature of the SGNN is that input is divided and fed sequentially to hidden layers. Therefore, the depth (layers) of SGNNs is identical to the number of input dimensions. Each neuron in hidden layers is associated with an uni-variate Gaussian function. Each path in feedforward propagation will lead to a chain of multiplications of uni-variate Gaussian functions, equivalent to a \(d\)-dimensional Gaussian radial-basis function shown in Equation (4). In other words, each SGNN can be converted to an RGBFNN.  
**Definition 2**.: _The separable-Gaussian neural network (SGNN) with d-dimensional input can be constructed in the form_  
\[\mathcal{N}_{i}^{(0)} =x_{i},\ 1\leq i\leq d, \tag{5}\] \[\mathcal{N}_{i}^{(1)} =\varphi_{i}^{(1)}(x_{1},\mu_{i}^{(1)},\sigma_{i}^{(1)}),\ 1\leq i \leq N_{1},\] (6) \[\mathcal{N}_{i}^{(\ell)} =\varphi_{i}^{(\ell)}(x_{\ell},\mu_{i}^{(\ell)},\sigma_{i}^{( \ell)})\sum_{j=1}^{N_{l}}W_{ij}^{(\ell)}\mathcal{N}_{j}^{(\ell-1)},\ \ 2\leq\ell\leq d,1\leq i\leq N_{\ell},\] (7) \[\bar{f}(\mathbf{x}) =\mathcal{N}(\mathbf{x})=\sum_{j=1}^{N_{d}}\mathcal{N}_{j}^{(d)}, \tag{8}\]  
_where \(N_{i}\) (\(i=1,2,\ldots,d\)) represents the number of neurons of the \(l\)-th layer, \(\mathcal{N}_{i}^{(l)}\) represents the output of the \(i\)-th Gaussian neuron (activation function) of the \(l\)-th layer. In addition, \(\mu_{i}^{(l)},\sigma_{i}^{(l)}\) represent the \(i\)-th center and width of the \(l\)-th layer, respectively._  
The weights of the output layer are assumed to be unity, although they can be trainable. Substituting Equations (5)-(7) into Equation (8) yields  
\[\bar{f}(\mathbf{x})=\sum_{i_{d}=1}^{N_{d}}\sum_{i_{d-1}=1}^{N_{d-1}}\cdots\sum _{i_{1}=1}^{N_{1}}\left[W_{i_{d}i_{d-1}}^{(d-1)}W_{i_{d-1}i_{d-2}}^{(d-2)}\ldots W _{i_{2}i_{1}}^{(1)}\right]\prod_{\ell=1}^{d}\varphi_{i_{\ell}}^{(\ell)}(x_{ \ell}), \tag{9}\]  
with  
\[\prod_{\ell=1}^{d}\varphi_{i_{\ell}}^{(\ell)}(x_{\ell})=\varphi_{i_{d}}^{(d) }(x_{d})\varphi_{i_{d-1}}^{(d-1)}(x_{d-1})\ldots\varphi_{i_{1}}^{(1)}(x_{1}), \tag{10}\]  
where \(W_{i_{d}i_{d}}^{(l)}\) (\(l=1,2,\ldots,d\)) represents the weight of the \(i_{(l+1)}\)-th neuron of the \((l+1)\)-th layer and the \(i_{l}\)-th neuron of the \(l\)-th layer. The loss function of the SGNN is defined in the form  
\[J=\|f-\bar{f}\|_{2}=\sqrt{\sum_{i=1}^{d}\left[f(x_{i})-\bar{f}(x_{i})\right]^ {2}}. \tag{11}\]  
The center \(\mu_{i}^{(l)}\) and width \(\sigma_{i}^{(l)}\) in the Gaussian function \(\varphi_{i}^{(l)}\) can also be treated as trainable. They are not included in this discussion for simplicity.