The usual penalty is the sum of squared weights times a decay constant. In a linear model, this form of weight decay is equivalent to ridge regression. Weight decay is a subset of regularization methods. The penalty term in weight decay, by definition, penalizes large weights. The weight decay penalty term causes the weights to converge to smaller absolute values than they otherwise would.
Large weights can hurt generalization in two different ways.
Excessively large weights leading to hidden units can cause the output function to be too rough, possibly with near discontinuities. Excessively large weights leading to output units can cause wild outputs far beyond the range of the data if the output activation function is not bounded to the same range as the data. Other penalty terms besides the sum of squared weights are sometimes used. The generalization ability of the network can depend crucially on the decay constant, especially with small training sets. There are other important considerations for getting good results from weight decay.