There are other important considerations for getting good results from weight decay. You must either standardize the inputs and targets, or adjust the penalty term for the standard deviations of all the inputs and targets. It is usually a good idea to omit the biases from the penalty term. A fundamental problem with weight decay is that different types of weights in the network will usually require different decay constants for good generalization. Adjusting all these decay constants to produce the best estimated generalization error often requires vast amounts of computation.
Bayesian estimation makes it possible to estimate efficiently numerous decay constants.
Generalization by weight-elimination with application to forecasting. I haven't really come to grips with error criteria in general. MacKay and Neal have done the most work on Bayesian methods for neural nets. Whereas ridge regression tends to shrink the large coefficients more than the small ones, weight elimination tends to shrink the small coefficients more, and is therefore more useful for suggesting subset models. How well this really works, no one seems to know.