Add penalty terms to other layers.