If the model complexity is too high the model will memorize the training data, hence , have a very low training error, but the test error will be high. The test error is defined as the expected error on an hithereto unseen test example. The two most widely used techniques for implementing this compromise are regularization and pruning. Both schemes have been shown to improve test performance significantly. It is however possible to estimate test performance, either using statistical theory or test sets.
Test sets are data from the database that are hold out during training.
For a number of the standard neural net models it has shown possible to use the trained neural net to guide the expert in providing more examples. When the specifics of the problem has been resolved, and a given family of network algorithms has been chosen one faces an optimization problem with many feasible or near-feasible solutions. Since neural networks involve non-linear adaptation, the training process often provides a wide variety of solutions, e.g., due to random initializations or random sequencing of examples. Rather than dismissing solutions it is recommended to form collective decisions, i.e., by forming consensus, among the ensemble of network solutions. One may think of this tool as a way of using network that are specialised in certain regions of input space.