Weight Decay is a regularization technique that penalizes large weights in the model. It helps prevent overfitting and improves generalization performance.