Mailer: Honor port setting also when using SSL

Fixes #110 and #57