Repository: privacyidea/privacyidea

Issue:
Audit log signature fails for users with non-ASCII characters
With #748 applied, LDAP users with non-ASCII usernames can be authenticated, but signing the audit log entry fails:
```
[2017-07-21 10:54:04,701][5822][139711469012032][ERROR][privacyidea.lib.auditmodules.sqlaudit:239] exception UnicodeEncodeError('ascii', u'id=125,date=2017-07-21 10:54:04.685018,action=GET /validate/check,succ=1,serial=PISP00000B46,t=None,u=k\xf6lbel,r=defrealm,adm=None,ad=,i=matching 1 tokens,ps=localhost:5000,c=127.0.0.1,l=None,cl=None', 103, 104, 'ordinal not in range(128)')
[2017-07-21 10:54:04,701][5822][139711469012032][ERROR][privacyidea.lib.auditmodules.sqlaudit:240] DATA: {'info': 'matching 1 tokens', 'realm': u'defrealm', 'tokentype': u'spass', 'success': True, 'privacyidea_server': 'localhost:5000', 'client_user_agent': None, 'client': '127.0.0.1', 'user': u'k\xf6lbel', 'resolver': u'ad', 'action_detail': '', 'action': 'GET /validate/check', 'serial': u'PISP00000B46'}
[2017-07-21 10:54:04,702][5822][139711469012032][DEBUG][privacyidea.lib.auditmodules.sqlaudit:241] Traceback (most recent call last):
  File "/home/fred/dev/netknights/privacyidea/privacyidea/privacyidea/lib/auditmodules/sqlaudit.py", line 234, in finalize_log
    sign = self.sign_object.sign(s)
  File "/home/fred/dev/netknights/privacyidea/privacyidea/privacyidea/lib/crypto.py", line 654, in sign
    hashvalue = HashFunc.new(s).digest()
  File "/home/fred/dev/netknights/privacyidea/privacyidea/venv/lib/python2.7/site-packages/Crypto/Hash/SHA256.py", line 88, in new
    return SHA256Hash().new(data)
  File "/home/fred/dev/netknights/privacyidea/privacyidea/venv/lib/python2.7/site-packages/Crypto/Hash/SHA256.py", line 75, in new
    return SHA256Hash(data)
  File "/home/fred/dev/netknights/privacyidea/privacyidea/venv/lib/python2.7/site-packages/Crypto/Hash/SHA256.py", line 72, in __init__
    HashAlgo.__init__(self, hashFactory, data)
  File "/home/fred/dev/netknights/privacyidea/privacyidea/venv/lib/python2.7/site-packages/Crypto/Hash/hashalgo.py", line 51, in __init__
    self.update(data)
  File "/home/fred/dev/netknights/privacyidea/privacyidea/venv/lib/python2.7/site-packages/Crypto/Hash/hashalgo.py", line 69, in update
    return self._hash.update(data)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 103: ordinal not in range(128)
```
Here, ``s`` is:
```
u'id=125,date=2017-07-21 10:54:04.685018,action=GET /validate/check,succ=1,serial=PISP00000B46,t=None,u=k\xf6lbel,r=defrealm,adm=None,ad=,i=matching 1 tokens,ps=localhost:5000,c=127.0.0.1,l=None,cl=None'
```
i.e. a unicode string. This is because ``_log_to_string`` returns a unicode string, which is no problem as long as it only contains ASCII characters.


Discussion:
We could use utf8 encoding for the resulting string ``s`` in ``_log_to_string``.

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -319,6 +319,8 @@ def _log_to_string(le):
                                                   le.client,
                                                   le.loglevel,
                                                   le.clearance_level)
+        if type(s) == unicode:
+            s = s.encode("utf-8")
         return s
 
     @staticmethod


Test patch:
diff --git a/tests/test_lib_audit.py b/tests/test_lib_audit.py
--- a/tests/test_lib_audit.py
+++ b/tests/test_lib_audit.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 """
 This tests the files
   lib/audit.py and
@@ -193,3 +194,15 @@ def test_06_truncate_data(self):
                          column_length.get("serial"))
         self.assertEqual(len(self.Audit.audit_data.get("token_type")),
                          column_length.get("token_type"))
+
+    def test_07_sign_non_ascii_entry(self):
+        # Log a username as unicode with a non-ascii character
+        self.Audit.log({"serial": "1234",
+                        "action": "token/assign",
+                        "success": True,
+                        "user": u"kölbel"})
+        self.Audit.finalize_log()
+        audit_log = self.Audit.search({"user": u"kölbel"})
+        self.assertEqual(audit_log.total, 1)
+        self.assertEqual(audit_log.auditdata[0].get("user"), u"kölbel")
+        self.assertEqual(audit_log.auditdata[0].get("sig_check"), "OK")