# Constructing Features Using a Hybrid Genetic Algorithm
## 1 Introduction  
Artificial neural networks (ANNs) are programming tools [1; 2], based on a series of parameters that are commonly called weights or processing units. They have been used in a variety of problems from different scientific areas such as physics [3; 4; 5], chemistry [6; 7; 8], economics [9; 10; 11] and medicine [12; 13]. A common way to express a neural network is a function \(N(\overrightarrow{x},\overrightarrow{u})\), with \(\overrightarrow{x}\) as the input vector (commonly called pattern) and \(\overrightarrow{u}\) as the weight vector. A method that trains a neural network should be used to estimate the vector \(\overrightarrow{u}\) for a certain problem. The training procedure can be also formulated as an optimization problem, wherein the objective is to minimize the so-called error function:  
\[E\left(N\left(\overrightarrow{x},\overrightarrow{u}\right)\right)=\sum_{i=1} ^{M}\left(N\left(\overrightarrow{x}_{i},\overrightarrow{u}\right)-y_{i} \right)^{2} \tag{1}\]  
In Equation (1), the set \(\left(\overrightarrow{x}_{i},y_{i}\right),\ i=1,\ldots,M\) is the dataset used to train the neural network, with \(y_{i}\) being the actual output for the point \(\overrightarrow{x_{i}}\). The neural network form used here was also considered in [14]. Suppose we have a neural network with a processing level that uses the sigmoid function as an output function. Every output of the network is defined as  
\[o_{i}(x)=\sigma\left(p_{i}^{T}x+\theta_{i}\right), \tag{2}\]  
where \(p_{i}\) is the weight vector and \(\theta_{i}\) is the bias for the output \(i\). For a neural network with \(H\) hidden nodes, the final output function can be written as  
\[N(x)=\sum_{i=1}^{H}v_{i}o_{i}(x), \tag{3}\]  
where \(v_{i}\) is the output weight for the processing unit \(i\). Hence, by using one vector for all parameters (weights and biases), the neural network can be written in the following form:  
\[N\left(\overrightarrow{x},\overrightarrow{u}\right)=\sum_{i=1}^{H}w_{(d+2)i-( d+1)}\sigma\left(\sum_{j=1}^{d}x_{j}w_{(d+2)i-(d+1)+j}+w_{(d+2)i}\right) \tag{4}\]where \(H\) is the number of processing units of the neural network and \(d\) is the dimension of vector \(\overrightarrow{x}\). The function \(\sigma(x)\) is the sigmoid function defined as  
\[\sigma(x)=\frac{1}{1+\exp(-x)} \tag{5}\]  
From Equation (4), one can obtain that the dimension of the weight vector \(w\) is computed as: \(w=(d+2)H\). The function of Equation (1) has been minimized with a variety of optimization methods during the past years, such as the back propagation method [15; 16], the RPROP method [17; 18; 19], Quasi Newton methods [20; 21] and particle swarm optimization [22; 23]. All the previously mentioned methods have to overcome two major problems:  
* Excessive computational times, because they require a processing time proportional to the dimension of the objective problem and the number of processing units as well. For example, a neural network of \(H=10\) processing units applied to a test data with \(d=3\) is considered an optimization problem with dimension \(w=(d+2)H=50\). This means that the total number of network parameters is growing extremely quickly, which results in a longer computation time than the corresponding universal optimization method. An extensive discussion of the problems caused by the dimensionality of neural networks was presented in [24]. A common approach to overcome this problem is to use the PCA technique to reduce the dimensionality of the objective problem [25; 26; 27], i.e., the parameter \(d\). * The overfitting problem, which is quite common for these methods to produce poor results when they are applied to data (test data) not previously used in the training procedure. This problem was discussed in detail in the article by [PERSON] et al. [28] as well as in the article of [PERSON] [29]. A variety of methods have been proposed to overcome this problem, such as weight sharing [30], pruning [31; 32; 33], the dropout technique [34], early stopping [35; 36], and weight decaying [37; 38].