When too many examples are used for validation, not enough examples are left for training to generalize, i.e. overfitting occurs. When too few examples are used for validation, the validation error does not accurately reflect the test error. On the ugcs machines, the model with one hidden unit is selected by cross-validation, and the model with two hidden units has the lowest test error. Weight Decay Shrink multiplier 0.999 works best, allowing the model enough flexibility to train, but not enough to overfit the training data. When the shrink multiplier is too small, the gradient descent step is overwhelmed by weight shrinking.
All weights shrink exponentially to zero.
When the shrink multiplier is too large, weights are allowed to grow out of hand. There is no regularization, and overfitting occurs.