Thus, higher learning rates can be used to obtain solutions similar to those found with lower learning rates without increasing training time. Certain tools allow additional parameters such as temperature, gain and noise to be modified during the training process. The sheer number of decisions to be made when developing a neural network necessitates training testing automation. This is especially true for setting training parameters, selecting preprocessing and choosing the number of hidden layers and neurons. To expedite parameter space searches, we use in-house development tools.
Genetic algorithms are an example of such a tool.
These algorithms are effective for many parameter optimization tasks. They use simple mechanisms analogous to those used in genetics to breed populations of superior solutions to optimization problems. Certain forms of simulated annealing have also been found to be useful for automating learning rate adjustments during training. This method of training simulates annealing by including a temperature term that directly affects the learning rate. In simulated annealing, temperature refers to the energy of a neural network.