Repository: privacyidea/privacyidea

Issue:
Sending mails fails if SMTP server advertises CRAM-MD5 authentication
Hello,

I've installed privacyidea 3.0 on a debian stretch in an virtualenv with python 2.7. There were a problem so the smtp module did not works as exprected.

If you try to use a smtp server privacyidea throws the exception: "character mapping must return integer, None or unicode"

Here's the stack:
```
[2019-05-21 09:59:31,017] ERROR in app: Exception on /smtpserver/send_test_email [POST]
 127.0.0.1:34212] Traceback (most recent call last):
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
 127.0.0.1:34212]     response = self.full_dispatch_request()
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
 127.0.0.1:34212]     rv = self.handle_user_exception(e)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
 127.0.0.1:34212]     reraise(exc_type, exc_value, tb)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
 127.0.0.1:34212]     rv = self.dispatch_request()
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
 127.0.0.1:34212]     return self.view_functions[rule.endpoint](**req.view_args)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/api/lib/prepolicy.py", line 129, in policy_wrapper
 127.0.0.1:34212]     return wrapped_function(*args, **kwds)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/log.py", line 154, in log_wrapper
 127.0.0.1:34212]     return func(*args, **kwds)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/api/smtpserver.py", line 157, in test
 127.0.0.1:34212]     u"The configuration {} is working.".format(identifier))
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/smtpserver.py", line 145, in send_or_enqueue_email
 127.0.0.1:34212]     return send(config, recipient, subject, body, sender, reply_to, mimetype)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/smtpserver.py", line 116, in test_email
 127.0.0.1:34212]     mail.login(config['username'], password)
 127.0.0.1:34212]   File "/usr/lib/python2.7/smtplib.py", line 607, in login
 127.0.0.1:34212]     (code, resp) = self.docmd(encode_cram_md5(resp, user, password))
 127.0.0.1:34212]   File "/usr/lib/python2.7/smtplib.py", line 571, in encode_cram_md5
 127.0.0.1:34212]     response = user + " " + hmac.HMAC(password, challenge).hexdigest()
 127.0.0.1:34212]   File "/usr/lib/python2.7/hmac.py", line 75, in __init__
 127.0.0.1:34212]     self.outer.update(key.translate(trans_5C))
 127.0.0.1:34212] TypeError: character mapping must return integer, None or unicode
 127.0.0.1:34212] /opt/privacyidea/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:536: Warning: Data truncated for column 'info' at row 1
 127.0.0.1:34212]   cursor.execute(statement, parameters)
```


Discussion:
Thank you for your bug report. What SMTP server are you using? Judging from the traceback, I think we attempt to authenticate to the SMTP server using CRAM-MD5. The error is thrown because the type of ``password`` is ``unicode``, which  [breaks](https://stackoverflow.com/a/20862445) the ``hmac`` module.
I've tried two servers both with tls encryption. The one fron selfhost.de and gmx.net.
Thanks for the update! Indeed, it seems like sending mails is broken under Python 2 + privacyIDEA 3.0 if the SMTP server advertises CRAM-MD5.

I checked the behavior under privacyIDEA 2.23.5 to see if we had this bug prior to 3.0:
* under 2.23.5, **testing** a SMTP server configuration with a CRAM-MD5 server fails with the same error ...
* but if we just save a SMTP server configuration without testing, we can actually use it e.g. for email tokens, because the password is passed in a bytestring in this case.

So there might be 2.23.5 installations which currently work and would break with an update to 3.0 or 3.0.1. 

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -17,7 +17,7 @@
 # built documents.
 #
 # The short X.Y version.
-version = '3.0.1dev2'
+version = '3.0.1'
 # The full version, including alpha/beta/rc tags.
 #release = '2.16dev5'
 release = version
diff --git a/privacyidea/lib/smtpserver.py b/privacyidea/lib/smtpserver.py
--- a/privacyidea/lib/smtpserver.py
+++ b/privacyidea/lib/smtpserver.py
@@ -17,11 +17,14 @@
 # License along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 #
+import six
+
+from privacyidea.lib.framework import get_app_config_value
 from privacyidea.lib.queue import job, wrap_job, has_job_queue
 from privacyidea.models import SMTPServer as SMTPServerDB
 from privacyidea.lib.crypto import (decryptPassword, encryptPassword,
                                     FAILED_TO_DECRYPT_PASSWORD)
-from privacyidea.lib.utils import fetch_one_resource
+from privacyidea.lib.utils import fetch_one_resource, to_bytes
 import logging
 from privacyidea.lib.log import log_with
 from time import gmtime, strftime
@@ -113,6 +116,11 @@ def test_email(config, recipient, subject, body, sender=None,
             password = decryptPassword(config['password'])
             if password == FAILED_TO_DECRYPT_PASSWORD:
                 password = config['password']
+            # Under Python 2, we pass passwords as bytestrings to get CRAM-MD5 to work.
+            # We add a safeguard config option to disable the conversion.
+            # Under Python 3, we pass passwords as unicode.
+            if six.PY2 and get_app_config_value("PI_SMTP_PASSWORD_AS_BYTES", True):
+                password = to_bytes(password)
             mail.login(config['username'], password)
         r = mail.sendmail(mail_from, recipient, msg.as_string())
         log.info("Mail sent: {0!s}".format(r))
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,7 @@
 import sys
 
 #VERSION="2.1dev4"
-VERSION="3.0.1dev2"
+VERSION="3.0.1"
 
 # Taken from kennethreitz/requests/setup.py
 package_directory = os.path.realpath(os.path.dirname(__file__))


Test patch: