Clean up mailer settings [Fixes #72911316]