The parameter estimates are rarely interpretable. Even then, it is common for the parameters to be underidentified. I've added weight decay to my NN software. I'm not even sure I've done it right. Here's what I've got so far.
Weight decay adds a penalty term to the error function.
The usual penalty is the sum of squared weights times a decay constant. In a linear model, this form of weight decay is equivalent to ridge regression. The penalty term causes the weights to converge to smaller absolute values than they otherwise would. Large weights can hurt generalization in two different ways. Excessively large weights leading to hidden units can cause the output function to be too rough, possibly with near discontinuities.