# A New Cascade-Correlation Growing Deep Learning Neural Network Algorithm
## 2 State of the Art
### Constructive Neural Networks  
Constructive neural networks (CoNNs) are a set of algorithms that change the design of the network, automatically generating a network with an acceptable dimension. The algorithms used for the CoNN learning method are called constructive algorithms. A constructive algorithm begins with a small network design and adding layers, nodes, and connections as appropriate during training. The architecture adaptation process resumes until the training algorithm obtains an almost best model that provides a sufficient problem solution [4].  
Constructive algorithms have many huge benefits over pruning algorithms, such as, for constructive algorithms, it is fairly simple to define the initial network architecture, whereas in pruning algorithms, the initial network does not have an accurate number of layers [5]. In constructive algorithms, at the initial step of the training phase a smaller number of parameters (weights) are to be modified, thus requiring less training data for good generalization; however, we need large amounts of training data in pruning algorithms [4]. One popular feature of constructive algorithms is to suggest that the hidden nodes that are already installed in the network are useful for modeling part of the underlying task. In such situations, the weights fed to such implemented nodes may be frozen in order to prevent the problem of moving the target. The number of optimized weights is reduced over time to reduce the time and requirements of the memory [4].  
For all these previous reasons, we used a constructive algorithm rather than the pruning algorithm for the proposed algorithm.  
CoNN algorithms are different from each other, in terms of [6]:  
1. Number of nodes that have been added per layer at each iteration;
2. The network grows forward or backward;
3. Whether all neurons make the same function;
4. Stopping criteria;
5. Newly added neuron connectivity pattern;
6. Kind of input patterns that deal with binary valued or categorical attributes;
7. Type of problems including classification and regression problems or clustering.  
We preferred to make our proposed constructive algorithm according to the following characteristics:  
1. The network grows forward;
2. Stopping criteria are based on when the desired accuracy of the network is achieved;
3. Kind of input patterns is binary valued;
4. This algorithm can deal with classification problems.  
Within neural network literature, several CoNN algorithms that are only suitable for classification problems have been proposed. The well-known two-class CoNN algorithms are the tower and pyramid [7], the tiling [8], the upstart [9] and the cascade of perceptron [10]. The MTower, MPyramid, Mtiling, MUpstart and MPerceptron cascade are the multi-class versions of these algorithms. One can see these algorithms in [11; 12]. The major constructive algorithms for regression problems, as mentioned in [13], are dynamic node creation (DNC) [14; 15; 16; 17; 18; 19; 20; 21; 22], cascade-correlation algorithm (CCA) [23], projection pursuit regression based on the statistical technique [24], resource-allocating network [25], group method of data handling [26] and miscellaneous [27].  
Among these algorithms, the CCA was chosen as the proposed algorithm and the next subsection will show how CCA works and its advantages over the other algorithms.