Each time the weights change, the network is taking a step on a multidimensional surface, representing the overall error space. During training the network is traveling along the surface, trying to find the lowest point or minimum error. Other training parameters that can be adjusted during the training process include temperature, gain, and noise. With various training parameters, preprocessing methods, and architectural configurations that can be explored, an automated training and testing regime that integrates testing with training is needed. Tools such as genetic algorithms and simulated annealing can be used to expedite searching these parameter spaces.
Genetic algorithms are effective for many parameter optimization tasks.
Simulated annealing automates learning rate adjustments during training by including a variable temperature term that affects the learning rate. When the temperature is high, learning is rapid. When the temperature drops, learning slows down as the network settles upon a solution. Over training Over training, analogous to curve fitting of rule-based trading systems, is one of the major pitfalls that must be avoided when developing neural networks. Over training occurs when a network memorizes the subtleties and idiosyncrasies specific to the training set, lessening its ability to generalize to new data.