# A New Cascade-Correlation Growing Deep Learning Neural Network Algorithm
## 3 New Proposed Cascade-Correlation Growing Deep Learning Neural Network Algorithm (CCG-DLNN)
### Steps of CCG-DLNN Algorithm  
As shown by Algorithm 1 and Figure 4, the CCG-DLNN algorithm begins with minimal architecture (input and output layers) without any hidden layers, and this simple network is trained and the loss function \(L(\varphi_{t})\) is calculated. Let us consider that DE is the desired error [2], \(L(\varphi_{t})\geq\text{DE}\) denotes that more neural hidden layers and more nodes are required to fit the training data set correctly. To do so, we add a new completely linked neuron to the last hidden layer or add a new hidden layer. We let \(C\) be introduced to measure changes in the loss function when a new neuron is added to the current hidden layer and \(\tau\) is the threshold to add a new hidden layer. As shown in Equations (5) and (6), if \(L(\varphi_{t-1})\) is degraded compared to \(L(\hat{\varphi}_{t})\) before adding the new neuron, then increases \(C\) by one. Otherwise, \(C\) will be zero. While \(C\) is less than the threshold \(\tau\), we add one new neuron to the last hidden layer. If \(C\) reaches the threshold value \(\tau\), then we add a new hidden layer that has one neuron as in Figure 5:  
\[ \begin{cases}\begin{aligned} C=0&& ifL(\varphi_{t-1} )-L(\hat{\varphi}_{t})>\epsilon\\ C=C+1&& otherwise\end{aligned}\end{cases}  \tag{5}\]  
\[ \begin{cases}\begin{aligned} l=l+1&& ifC\geq\tau\\ N_{ul}=N_{ul}+1&& otherwise\end{cases} \end{cases} \tag{6}\]  
where \(\epsilon\) is a growing process threshold. The stage of the neuron can be divided into two parts. First, we start with a candidate neuron that accepts input links at the last hidden layer from all pre-existing hidden units. The candidate unit's output is not yet connected to the active network. We run several passes over the training set examples, adjusting the candidate unit's input weights for each pass. Second, the candidate is connected to the (activated) output neurons, and then all output connections are learned. The entire cycle is repeated until the network's required accuracy is achieved. Increasing the number of hidden layers may or may not increase the accuracy, as it really depends on the difficulty of the solved problem. Increasing the number of hidden layers much more than a sufficient number of layers would reduce the accuracy of the test set. It will make the network over-fit the training set,  
Figure 3: Candidate neuron at neural network. that is, it will know the training data, but it will not be able to generalize to new unknown data [29]. The large weights in the neural network are a sign of over-fitting. The research in [30] defined weights in a qualified network in a more coherent manner, by preventing them from taking broad values using Equation (7):  
\[min(E(\varphi)),where(l\leq\varphi\leq u) \tag{7}\]  
where \(\varphi\) is all the neural network weight connections, \(E(\varphi)\) is the output error, and \(l\) is a lower bound and \(u\) is an upper bound. ```
Input: \(X:\) training data set \(S:\) the training data set size \(iter\): iteration number for the NN training \(t:\) the index of training step \(l\): current hidden layer index \(N_{ul}:\) the count of neurons at layer \(l\) \(W^{l}\): Matrix of the links from layers \(l\) to \((l-1)\) \(L(\varphi_{t})\): The loss function at the t step DE: The threshold Error \(\tau\): adding a new hidden layer Threshold \(C\): Condition for adding a new hidden layer Output:  Deep Neural Network \(l=0\), \(N_{ul}=0\), \(C\)=0, \(t=0\) \(feedforward(X)\) Calculate \(L(\varphi_{t})\) \(backpropagation(X,L(\varphi_{t}))\) update weights \(W^{l}\) of the initial architecture that has input and output layers only. \(t=t+1\) while\(L(\varphi_{t})\geq DE\) and \(t\leq iter\)do if\(L(\varphi_{l-1})-L(\hat{\varphi}_{t})>\epsilon\)then break else if\(C\geq\tau\)then \(l=l+1\) \(C\)= 0 \(N_{ul}=1\) else \(C\)= \(C\)+1 \(N_{ul}=N_{ul}+1\)
```  
**Algorithm 1**Cascade-Correlation Growing Deep Learning Neural Network  
The large weights in the neural network are a sign of over-fitting.