# Fitness Landscape Analysis of Product Unit Neural Networks
## 2 Product Unit Neural Networks  
Higher-order NNs include functional link NNs [4], sigma-pi NNs [3], second-order NNs [5], and PUNNs [2]. PUNNs [2; 6; 8] calculate the net input signal to hidden units as a weighted product of the input signals, i.e.,  
\[net_{y_{j},p}=\prod_{i=1}^{\mathcal{I}}z_{i,p}^{v_{ji}} \tag{1}\]  
instead of using the traditional SU, where the input signal is calculated as a linear weighted sum of the input signals, i.e.,  
\[net_{y_{j},p}=\sum_{i=1}^{\mathcal{I}+1}z_{i,p}v_{ji} \tag{2}\]  
In the above, \(net_{y_{j},p}\) is the net input signal to unit \(y_{j}\) for pattern \(p\), \(z_{i,p}\) is the activation level of unit \(z_{i}\), \(v_{ji}\) is the weight between units \(y_{j}\) and \(z_{i}\), and \(\mathcal{I}\) is the total number of units in the previous layer [14]. The bias is modeled as the (\(\mathcal{I}+1\))-th unit, where \(z_{\mathcal{I}+1,p}=-1\) for all patterns, and \(v_{j,\mathcal{I}+1}\) represents the bias [14]. A SUNN is implemented with bias units for the hidden and output layer; a PUNN is implemented with a bias unit for the output layer only. There are two types of architectures that incorporate PUs [2]: (1) each layer alternates between PUs and SUs, with the output layer always consisting of SUs; (2) a group of dedicated PUs are connected to each SU while also being connected to the input units. This paper makes use of the former architecture, with one hidden layer consisting of PUs and linear activation functions used in all layers. Using this architecture, the activation of a PU for a pattern \(p\) is expressed as  
\[net_{y_{j},p}=\prod_{i=1}^{\mathcal{I}}z_{i,p}^{v_{ji}}=\prod_{i=1}^{\mathcal{ I}}e^{v_{ji}ln(z_{i,p})}=e^{\sum_{i=1}^{\mathcal{I}}v_{ji}ln(z_{i,p})} \tag{3}\]  
for \(z_{i,p}>0\). If \(z_{i,p}<0\), then \(z_{i,p}\) is written as the complex number \(z_{i,p}=i^{2}|z_{i,p}|\), yielding  
\[net_{y_{j},p} = e^{\sum_{i=1}^{\mathcal{I}}v_{ji}ln|z_{i,p}|} \tag{4}\] \[\times (\cos(\pi\sum_{i=1}^{\mathcal{I}}v_{ji}\mathcal{I}_{i})+i\sin( \pi\sum_{i=1}^{\mathcal{I}}v_{ji}\mathcal{I}_{i}))\]  
where  
\[\mathcal{I}_{i}=\begin{cases}0&\text{if }z_{i}>0\\ 1&\text{if }z_{i}<0\end{cases} \tag{5}\]  
The above equations illustrate that the computational costs for gradient-based approaches are higher than when SUs are used.  
[PERSON] and [PERSON] discovered that, apart from the added complexity of working in the complex domain, which results in double the number of equations and weight variables, no substantial improvements in results were gained [2; 14]. Therefore, the complex part of Equation (4) is omitted. Refer to [14] for the PUNN training rules using stochastic gradient descent (SGD).  
Research has shown that the approximation of higher-order functions using PUNNs provides more accurate results, better training time, and simpler network architectures than SUNNs [7]. Training time is less because PUNNs automatically learn the higher-order terms that are required to implement a specific function [6]. PUNNs have increased information capacity compared to SUNNs [2; 6]. The information capacity of a single PU is approximately \(3N\), compared to \(2N\) for a single SU, where \(N\) is the number of inputs to the unit. The increased information capacity results in fewer PUs required to learn complex functions, resulting in smaller network architectures.