Error notification if send test mail returns 500

closes #3728
- If the mail config is broken, the ajax call will fail with 500 and
not return an error.
- This change catches 500 and wraps the error in a notification.
Well, the more you know I guess. I'll update the PR.