Repository: privacyidea/privacyidea

Issue:
Creating tokens with Umlaut in Username not possible
When there's a german umlaut in the username the enrollment of a token isn't possible.
The error is:
'ascii' codec can't encode character u'\xfc' in position 1: ordinal not in range(128)

Username here was "Müller"

I know this is quite unusual but my customer has one .. 

Version tested was 2.19.

EDIT: Usersource is LDAP with simple bind 



Discussion:
I have no idea about privacyidea.

But Linux only accepts ASCII as username.
Maybe its inherited. :)
Yep, but the username is retrieved via LDAP/AD
@mimugmail Can you please give more details about the LDAP? Is it AD or OpenLDAP? What system/distribution is the LDAP running on? What encoding is used?

Sure! 
LDAP is running against W2K13 Server. This is my setup:
LDAP to IP XXX
Base DN: DC=XXX,DC=lokal
BIND DN: cn=otp user,ou=otp,dc=XXX,dc=lokal
SimpleBind

Searchfilter:
(memberOf=CN=OTPGruppe,OU=OTP,DC=XXX,DC=lokal)
User found:
![image](https://user-images.githubusercontent.com/20438689/27824555-262f7944-60ad-11e7-843a-b6e64b7dde42.png)

The PI system itself is latest 2.19 via PIP / virtualenv on Debian 8
**todo**: install 2K13 on bors
@cornelinux Sorry it's 2012 of course. The customer has also a second DC 2008R2 with the same result
of course! 2013 ;-)
@mimugmail It just came to my mind, that this might not be due to ldap! Obviously the loginname resolves correctly as you screenshot shows.
If this occurs during token enrollment this could be everything:
* faulty formated log entry
* problem with the QR code...

Are you enrolling a smartphone token?

Can you please reproduce and provide the interesting part of the log file?

If you want to, you can upload the log file here: https://lancelot.netknights.it/owncloud/s/3SF2FO5Z19x98S8
@cornelinux It happens when I click on the username, so it's before enrolling. 
Sent to you a full debug log, hope that's ok.
OK, I am just reproducing it. You are in the *user list* and then selecting the user "Müller"

Then you get the error. It comes from fetching tokens for the user "Müller". This call:

    GET https://localhost/pi//token/?page=1&pagesize=5&realm=mimi&user=m%C3%BCller


Unfortunately when in the ``token->enroll`` view and searching for the user

    POST https://localhost/pi//token/init

will also fail for the user "Müller".

My Log on the master branch:

<pre>
[2017-07-18 18:15:33,525][4880][140360892217088][ERROR][privacyidea.app:1423] Exception on /token/init [POST]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1473, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1666, in preprocess_request
    rv = func()
  File "/usr/lib/python2.7/dist-packages/privacyidea/api/auth.py", line 324, in decorated_function
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/privacyidea/api/before_after.py", line 78, in before_user_request
    before_request()
  File "/usr/lib/python2.7/dist-packages/privacyidea/api/before_after.py", line 111, in before_request
    request.User = get_user_from_param(request.all_data)
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/log.py", line 154, in log_wrapper
    return func(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 576, in get_user_from_param
    resolver=param.get("resolver"))
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/log.py", line 154, in log_wrapper
    return func(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 98, in __init__
    self._get_user_from_userstore()
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/usercache.py", line 63, in cache_wrapper
    return wrapped_function(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 106, in _get_user_from_userstore
    self._get_resolvers()
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 207, in _get_resolvers
    if self._locate_user_in_resolver(resolvername):
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 227, in _locate_user_in_resolver
    uid = y.getUserId(self.login)
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/resolvers/LDAPIdResolver.py", line 163, in cache_wrapper
    f_result = func(self, *args, **kwds)
  File "/usr/lib/python2.7/dist-packages/privacyidea/lib/resolvers/LDAPIdResolver.py", line 465, in getUserId
    self._escape_loginname(LoginName))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 1: ordinal not in range(128)

</pre>
@fredreichbier bummer. ``{n!s}`` ?
Plus we have a "{0!s}" in the log.debug in the cache wrapper. 

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -2,6 +2,8 @@
 #  Copyright (C) 2014 Cornelius Kölbel
 #  contact:  corny@cornelinux.de
 #
+#  2017-07-20 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#             Fix unicode usernames
 #  2017-01-23 Cornelius Kölbel <cornelius.koelbel@netknights.it>
 #             Add certificate verification
 #  2017-01-07 Cornelius Kölbel <cornelius.koelbel@netknights.it>
@@ -71,7 +73,7 @@
 import datetime
 
 from privacyidea.lib import _
-from privacyidea.lib.utils import to_utf8
+from privacyidea.lib.utils import to_utf8, to_unicode
 from privacyidea.lib.error import privacyIDEAError
 import uuid
 from ldap3.utils.conv import escape_bytes
@@ -131,7 +133,7 @@ def get_info_configuration(noschemas):
     get_schema_info = ldap3.SCHEMA
     if noschemas:
         get_schema_info = ldap3.NONE
-    log.debug("Get LDAP schema info: {0!s}".format(get_schema_info))
+    log.debug("Get LDAP schema info: {0!r}".format(get_schema_info))
     return get_schema_info
 
 
@@ -156,7 +158,7 @@ def cache_wrapper(self, *args, **kwds):
                         datetime.datetime.now() < r_cache[args[0]][
                     "timestamp"] + \
                         datetime.timedelta(seconds=self.cache_timeout):
-            log.debug("Reading {0!s} from cache for {1!s}".format(args[0],
+            log.debug("Reading {0!r} from cache for {1!r}".format(args[0],
                                                               func.func_name))
             return r_cache[args[0]]["value"]
 
@@ -193,7 +195,7 @@ def __init__(self):
         self.timeout = 5.0  # seconds!
         self.sizelimit = 500
         self.loginname_attribute = ""
-        self.searchfilter = ""
+        self.searchfilter = u""
         self.userinfo = {}
         self.uidtype = ""
         self.noreferrals = False
@@ -340,8 +342,9 @@ def _getDN(self, userId):
             # get the DN for the Object
             self._bind()
             search_userId = self._trim_user_id(userId)
-            filter = "(&{0!s}({1!s}={2!s}))".format(self.searchfilter,
-                                                    self.uidtype, search_userId)
+            filter = u"(&{0!s}({1!s}={2!s}))".format(self.searchfilter,
+                                                     self.uidtype,
+                                                     search_userId)
             self.l.search(search_base=self.basedn,
                           search_scope=self.scope,
                           search_filter=filter,
@@ -353,7 +356,7 @@ def _getDN(self, userId):
             elif len(r) == 1:
                 dn = r[0].get("dn")
             else:
-                log.info("The filter {0!s} returned no DN.".format(filter))
+                log.info("The filter {0!r} returned no DN.".format(filter))
 
         return dn
 
@@ -392,12 +395,13 @@ def getUserInfo(self, userId):
             # encode utf8, so that also german ulauts work in the DN
             self.l.search(search_base=to_utf8(userId),
                           search_scope=self.scope,
-                          search_filter="(&" + self.searchfilter + ")",
+                          search_filter=u"(&" + self.searchfilter + u")",
                           attributes=self.userinfo.values())
         else:
-            search_userId = self._trim_user_id(userId)
-            filter = "(&{0!s}({1!s}={2!s}))".format(self.searchfilter,
-                                                    self.uidtype, search_userId)
+            search_userId = to_unicode(self._trim_user_id(userId))
+            filter = u"(&{0!s}({1!s}={2!s}))".format(self.searchfilter,
+                                                     self.uidtype,
+                                                     search_userId)
             self.l.search(search_base=self.basedn,
                               search_scope=self.scope,
                               search_filter=filter,
@@ -460,9 +464,11 @@ def getUserId(self, LoginName):
         :return: UserId as found for the LoginName
         """
         userid = ""
+        LoginName = to_unicode(LoginName)
         self._bind()
-        filter = "(&{0!s}({1!s}={2!s}))".format(self.searchfilter, self.loginname_attribute,
-             self._escape_loginname(LoginName))
+        filter = u"(&{0!s}({1!s}={2!s}))".format(self.searchfilter,
+                                                 self.loginname_attribute,
+                                                 self._escape_loginname(LoginName))
 
         # create search attributes
         attributes = self.userinfo.values()
@@ -502,14 +508,15 @@ def getUserList(self, searchDict):
         # do the filter depending on the searchDict
         filter = u"(&" + self.searchfilter
         for search_key in searchDict.keys():
+            # convert to unicode
+            searchDict[search_key] = to_unicode(searchDict[search_key])
             if search_key == "accountExpires":
                 comperator = ">="
                 if searchDict[search_key] in ["1", 1]:
                     comperator = "<="
-                filter += "(&({0!s}{1!s}{2!s})(!({3!s}=0)))".format(self.userinfo[search_key],
-                                                  comperator,
-                                                  get_ad_timestamp_now(),
-                                                  self.userinfo[search_key])
+                filter += u"(&({0!s}{1!s}{2!s})(!({3!s}=0)))".format(
+                    self.userinfo[search_key], comperator,
+                    get_ad_timestamp_now(), self.userinfo[search_key])
             else:
                 filter += u"({0!s}={1!s})".format(self.userinfo[search_key],
                                                   searchDict[search_key])
@@ -544,9 +551,9 @@ def getResolverId(self):
         This should be an Identifier of the resolver, preferable the type
         and the name of the resolver.
         """
-        s = "{0!s}{1!s}{2!s}{3!s}".format(self.uri, self.basedn,
+        s = u"{0!s}{1!s}{2!s}{3!s}".format(self.uri, self.basedn,
                                           self.searchfilter, self.userinfo)
-        r = binascii.hexlify(hashlib.sha1(s).digest())
+        r = binascii.hexlify(hashlib.sha1(s.encode("utf-8")).digest())
         return r
 
     @staticmethod
@@ -783,7 +790,7 @@ def testconnection(cls, param):
             # search for users...
             g = l.extend.standard.paged_search(
                 search_base=param["LDAPBASE"],
-                search_filter="(&" + param["LDAPSEARCHFILTER"] + ")",
+                search_filter=u"(&" + param["LDAPSEARCHFILTER"] + ")",
                 search_scope=param.get("SCOPE") or ldap3.SUBTREE,
                 attributes=attributes,
                 paged_size=100,
@@ -848,12 +855,13 @@ def add_user(self, attributes=None):
             self.l.add(dn, self.object_classes, params)
 
         except Exception as e:
-            log.error("Error accessing LDAP server: {0}".format(e))
+            log.error("Error accessing LDAP server: {0!r}".format(e))
             log.debug("{0}".format(traceback.format_exc()))
             raise privacyIDEAError(e)
 
         if self.l.result.get('result') != 0:
-            log.error("Error during adding of user {0}: {1}".format(dn, self.l.result.get('message')))
+            log.error("Error during adding of user {0!r}: "
+                      "{1!r}".format(dn, self.l.result.get('message')))
             raise privacyIDEAError(self.l.result.get('message'))
 
         return self.getUserId(attributes.get("username"))
@@ -874,7 +882,7 @@ def delete_user(self, uid):
 
             self.l.delete(self._getDN(uid))
         except Exception as exx:
-            log.error("Error deleting user: {0}".format(exx))
+            log.error("Error deleting user: {0!r}".format(exx))
             res = False
         return res
 
@@ -976,12 +984,13 @@ def update_user(self, uid, attributes=None):
             params = self._attributes_to_ldap_attributes(mapped)
             self.l.modify(self._getDN(uid), params)
         except Exception as e:
-            log.error("Error accessing LDAP server: {0!s}".format(e))
+            log.error("Error accessing LDAP server: {0!r}".format(e))
             log.debug("{0!s}".format(traceback.format_exc()))
             return False
 
         if self.l.result.get('result') != 0:
-            log.error("Error during update of user {0!s}: {1!s}".format(uid, self.l.result.get("message")))
+            log.error("Error during update of user {0!r}: "
+                      "{1!r}".format(uid, self.l.result.get("message")))
             return False
 
         return True
diff --git a/privacyidea/lib/utils.py b/privacyidea/lib/utils.py
--- a/privacyidea/lib/utils.py
+++ b/privacyidea/lib/utils.py
@@ -129,6 +129,18 @@ def to_utf8(password):
     return password
 
 
+def to_unicode(s, encoding="utf-8"):
+    """
+    converts a value to unicode if it is of type str.
+    
+    :param s: The utf-8 encoded str 
+    :return: unicode string
+    """
+    if type(s) == str:
+        s = s.decode(encoding)
+    return s
+
+
 def generate_otpkey(key_size=20):
     """
     generates the HMAC key of keysize. Should be 20 or 32


Test patch:
diff --git a/tests/ldap3mock.py b/tests/ldap3mock.py
--- a/tests/ldap3mock.py
+++ b/tests/ldap3mock.py
@@ -383,7 +383,14 @@ def _match_equal_to(search_base, attribute, value, candidates):
                         entry["type"] = "searchResEntry"
                         matches.append(entry)
                 else:
-                    if str(value) == str(values_from_directory):
+                    # The value, which we compare is unicode, so we convert
+                    # the values_from_directory to unicode rather than str.
+                    if type(values_from_directory) == str:
+                        values_from_directory = values_from_directory.decode(
+                            "utf-8")
+                    elif type(values_from_directory) == int:
+                        values_from_directory = u"{0!s}".format(values_from_directory)
+                    if value == values_from_directory:
                         entry["type"] = "searchResEntry"
                         matches.append(entry)
 
@@ -451,7 +458,7 @@ def _parse_filter():
         # Value to contain:
         #   numbers, upper/lower case letters, astrisk, at symbol, minus, full
         #   stop, backslash or a space
-        v = pyparsing.Word(pyparsing.alphanums + "-*@.\\ ")
+        v = pyparsing.Word(pyparsing.alphanums + "-*@.\\ äöü")
         rel = pyparsing.oneOf("= ~= >= <=")
 
         expr = pyparsing.Forward()
@@ -648,10 +655,15 @@ def search(self, search_base=None, search_scope=None,
         self.result = dict()
 
         try:
+            if type(search_filter) == str:
+                # We need to convert to unicode otherwise pyparsing will not
+                # find the u"ö"
+                search_filter = search_filter.decode("utf-8")
             expr = Connection._parse_filter()
             s_filter = expr.parseString(search_filter).asList()[0]
-        except pyparsing.ParseBaseException:
-            pass
+        except pyparsing.ParseBaseException as exx:
+            # Just for debugging purposes
+            s = "{!s}".format(exx)
 
         for item in s_filter:
             if item[0] in self.operation:
diff --git a/tests/test_lib_resolver.py b/tests/test_lib_resolver.py
--- a/tests/test_lib_resolver.py
+++ b/tests/test_lib_resolver.py
@@ -30,7 +30,8 @@
 objectGUIDs = [
     '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d31',
     '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d77',
-    '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d54'
+    '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d54',
+    '7dd0533c-afe3-4c6f-b49e-af82eaed045c'
 ]
 
 
@@ -65,7 +66,17 @@
                                 'userPassword': 'ldaptest',
                                 "accountExpires": 9223372036854775807,
                                 "objectGUID": objectGUIDs[2],
-                                'oid': "1"}}]
+                                'oid': "1"}},
+                 {"dn": 'cn=kölbel,ou=example,o=test',
+                  "attributes": {'cn': "kölbel",
+                                 "givenName": "Cornelius",
+                                 "sn": "Kölbel",
+                                 "email": "cko@o",
+                                 "mobile": "123456",
+                                 "userPassword": "mySecret",
+                                 "accoutnExpires": 9223372036854775807,
+                                 "objectGUID": objectGUIDs[3],
+                                 "oid": "4"}}]
 
 LDAPDirectory_small = [{"dn": 'cn=bob,ou=example,o=test',
                  "attributes": {'cn': 'bob',
@@ -512,7 +523,7 @@ def test_01_LDAP_DN(self):
         })
 
         result = y.getUserList({'username': '*'})
-        self.assertEqual(len(result), 3)
+        self.assertEqual(len(result), len(LDAPDirectory))
 
         user = "bob"
         user_id = y.getUserId(user)
@@ -564,7 +575,7 @@ def test_01_LDAP_double_mapping(self):
         })
 
         result = y.getUserList({'username': '*'})
-        self.assertEqual(len(result), 3)
+        self.assertEqual(len(result), len(LDAPDirectory))
 
         user = "bob"
         user_id = y.getUserId(user)
@@ -635,7 +646,7 @@ def test_01_broken_uidtype(self):
         })
 
         result = y.getUserList({'username': '*'})
-        self.assertEqual(len(result), 3)
+        self.assertEqual(len(result), len(LDAPDirectory))
 
         rid = y.getResolverId()
         self.assertTrue(rid == "d6ce19abbc3c23e24e1cefa41cbe6f9f118613b9", rid)
@@ -673,7 +684,7 @@ def test_02_LDAP_OID(self):
         })
 
         result = y.getUserList({'username': '*'})
-        self.assertEqual(len(result), 3)
+        self.assertEqual(len(result), len(LDAPDirectory))
 
         user = "bob"
         user_id = y.getUserId(user)
@@ -726,7 +737,7 @@ def test_03_testconnection(self):
         })
 
         self.assertTrue(res[0], res)
-        self.assertTrue("3" in res[1])
+        self.assertTrue("{!s}".format(len(LDAPDirectory)) in res[1], res[1])
         # 'Your LDAP config seems to be OK, 3 user objects found.'
 
     @ldap3mock.activate
@@ -749,7 +760,7 @@ def test_03_testconnection_anonymous(self):
         })
 
         self.assertTrue(res[0], res)
-        self.assertTrue("3" in res[1])
+        self.assertTrue("{!s}".format(len(LDAPDirectory)) in res[1])
         #'Your LDAP config seems to be OK, 3 user objects found.'
 
     @ldap3mock.activate
@@ -1207,7 +1218,7 @@ def test_23_start_tls(self):
         start_tls_resolver = LDAPResolver()
         start_tls_resolver.loadConfig(config)
         result = start_tls_resolver.getUserList({'username': '*'})
-        self.assertEqual(len(result), 3)
+        self.assertEqual(len(result), len(LDAPDirectory))
         # We check two things:
         # 1) start_tls has actually been called!
         self.assertTrue(start_tls_resolver.l.start_tls_called)
@@ -1240,12 +1251,121 @@ def test_24_ldaps(self):
         start_tls_resolver = LDAPResolver()
         start_tls_resolver.loadConfig(config)
         result = start_tls_resolver.getUserList({'username': '*'})
-        self.assertEqual(len(result), 3)
+        self.assertEqual(len(result), len(LDAPDirectory))
         # We check that all Server objects were constructed with a non-None TLS context and use_ssl=True
         for _, kwargs in ldap3mock.get_server_mock().call_args_list:
             self.assertIsNotNone(kwargs['tls'])
             self.assertTrue(kwargs['use_ssl'])
 
+    @ldap3mock.activate
+    def test_25_LDAP_DN_with_utf8(self):
+        # This tests usernames are entered in the LDAPresolver as utf-8 encoded
+        ldap3mock.setLDAPDirectory(LDAPDirectory)
+        y = LDAPResolver()
+        y.loadConfig({'LDAPURI': 'ldap://localhost',
+                      'LDAPBASE': 'o=test',
+                      'BINDDN': 'cn=manager,ou=example,o=test',
+                      'BINDPW': 'ldaptest',
+                      'LOGINNAMEATTRIBUTE': 'cn',
+                      'LDAPSEARCHFILTER': '(cn=*)',
+                      'USERINFO': '{ "username": "cn",'
+                                  '"phone" : "telephoneNumber", '
+                                  '"mobile" : "mobile"'
+                                  ', "email" : "mail", '
+                                  '"surname" : "sn", '
+                                  '"givenname" : "givenName" }',
+                      'UIDTYPE': 'DN',
+                      })
+
+        result = y.getUserList({'username': '*'})
+        self.assertEqual(len(result), len(LDAPDirectory))
+
+        user = u"kölbel".encode("utf-8")
+        user_id = y.getUserId(user)
+        self.assertEqual(user_id, "cn=kölbel,ou=example,o=test")
+
+        rid = y.getResolverId()
+        self.assertTrue(rid == "d6ce19abbc3c23e24e1cefa41cbe6f9f118613b9", rid)
+
+        rtype = y.getResolverType()
+        self.assertTrue(rtype == "ldapresolver", rtype)
+
+        rdesc = y.getResolverClassDescriptor()
+        rdesc = y.getResolverDescriptor()
+        self.assertTrue("ldapresolver" in rdesc, rdesc)
+        self.assertTrue("config" in rdesc.get("ldapresolver"), rdesc)
+        self.assertTrue("clazz" in rdesc.get("ldapresolver"), rdesc)
+
+        uinfo = y.getUserInfo(user_id)
+        self.assertEqual(uinfo.get("username"), user)
+
+        ret = y.getUserList({"username": user})
+        self.assertTrue(len(ret) == 1, ret)
+
+        username = y.getUsername(user_id)
+        self.assertTrue(username == "kölbel", username)
+
+        res = y.checkPass(user_id, "mySecret")
+        self.assertTrue(res)
+
+        res = y.checkPass(user_id, "wrong pw")
+        self.assertFalse(res)
+
+    @ldap3mock.activate
+    def test_26_LDAP_DN_with_unicode(self):
+        # This tests usernames are entered in the LDAPresolver as unicode.
+        ldap3mock.setLDAPDirectory(LDAPDirectory)
+        y = LDAPResolver()
+        y.loadConfig({'LDAPURI': 'ldap://localhost',
+                      'LDAPBASE': 'o=test',
+                      'BINDDN': 'cn=manager,ou=example,o=test',
+                      'BINDPW': 'ldaptest',
+                      'LOGINNAMEATTRIBUTE': 'cn',
+                      'LDAPSEARCHFILTER': '(cn=*)',
+                      'USERINFO': '{ "username": "cn",'
+                                  '"phone" : "telephoneNumber", '
+                                  '"mobile" : "mobile"'
+                                  ', "email" : "mail", '
+                                  '"surname" : "sn", '
+                                  '"givenname" : "givenName" }',
+                      'UIDTYPE': 'DN',
+                      })
+
+        result = y.getUserList({'username': '*'})
+        self.assertEqual(len(result), len(LDAPDirectory))
+
+        user = u"kölbel"
+        user_id = y.getUserId(user)
+        self.assertEqual(user_id, "cn=kölbel,ou=example,o=test")
+
+        rid = y.getResolverId()
+        self.assertTrue(rid == "d6ce19abbc3c23e24e1cefa41cbe6f9f118613b9", rid)
+
+        rtype = y.getResolverType()
+        self.assertTrue(rtype == "ldapresolver", rtype)
+
+        rdesc = y.getResolverClassDescriptor()
+        rdesc = y.getResolverDescriptor()
+        self.assertTrue("ldapresolver" in rdesc, rdesc)
+        self.assertTrue("config" in rdesc.get("ldapresolver"), rdesc)
+        self.assertTrue("clazz" in rdesc.get("ldapresolver"), rdesc)
+
+        uinfo = y.getUserInfo(user_id)
+        self.assertEqual(uinfo.get("username"), user.encode("utf-8"))
+
+        ret = y.getUserList({"username": user})
+        self.assertTrue(len(ret) == 1, ret)
+
+        username = y.getUsername(user_id)
+        self.assertTrue(username == user.encode("utf-8"), username)
+
+        res = y.checkPass(user_id, "mySecret")
+        self.assertTrue(res)
+
+        res = y.checkPass(user_id, "wrong pw")
+        self.assertFalse(res)
+
+
 
 class BaseResolverTestCase(MyTestCase):
 
diff --git a/tests/test_lib_utils.py b/tests/test_lib_utils.py
--- a/tests/test_lib_utils.py
+++ b/tests/test_lib_utils.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 """
 This tests the file lib.utils
 """
@@ -9,7 +10,8 @@
                                    parse_date, compare_condition,
                                    get_data_from_params, parse_legacy_time,
                                    int_to_hex, compare_value_value,
-                                   parse_time_offset_from_now, parse_time_delta)
+                                   parse_time_offset_from_now,
+                                   parse_time_delta, to_unicode)
 from datetime import timedelta, datetime
 from netaddr import IPAddress, IPNetwork, AddrFormatError
 from dateutil.tz import tzlocal, tzoffset
@@ -349,3 +351,12 @@ def test_13_parse_time_offset_from_now(self):
         s, td = parse_time_offset_from_now("Hello {current_time}-3habc")
         self.assertEqual(s, "Hello {current_time}abc")
         self.assertEqual(td, timedelta(hours=-3))
+
+    def test_14_to_unicode(self):
+        s = "kölbel"
+        su = to_unicode(s)
+        self.assertEqual(su, u"kölbel")
+
+        s = u"kölbel"
+        su = to_unicode(s)
+        self.assertEqual(su, u"kölbel")
diff --git a/tests/testdata/tmp_directory b/tests/testdata/tmp_directory
--- a/tests/testdata/tmp_directory
+++ b/tests/testdata/tmp_directory
@@ -1 +1 @@
-[{'dn': 'cn=alice,ou=example,o=test', 'attributes': {'cn': 'alice', 'mobile': ['1234', '45678'], 'userPassword': 'alicepw', 'oid': '2', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d31', 'accountExpires': 9223372036854775805, 'homeDirectory': '/home/alice', 'givenName': 'Alice', 'email': 'alice@test.com', 'sn': 'Cooper'}}, {'dn': 'cn=mini,ou=example,o=test', 'attributes': {'cn': 'mini', 'mobile': ['1234', '45678'], 'userPassword': 'minipw', 'oid': '2', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d77', 'accountExpires': 0, 'homeDirectory': '/home/mini', 'givenName': 'Mini', 'email': 'mini@test.com', 'sn': 'Cooper'}}, {'dn': 'cn=bob,ou=example,o=test', 'attributes': {'cn': 'bob', 'description': 'Bobs Account', 'mobile': '123456', 'userPassword': 'bobpw\xc3\xa9\xc3\xa9\xc3\xa9', 'oid': '3', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d54', 'accountExpires': 9223372036854775807, 'homeDirectory': '/home/bob', 'givenName': 'Robert', 'email': 'bob@example.com', 'sn': 'Marley'}}, {'dn': 'cn=manager,ou=example,o=test', 'attributes': {'cn': 'manager', 'mobile': '123354', 'userPassword': 'ldaptest', 'oid': '1', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d88', 'accountExpires': 9223372036854775808L, 'givenName': 'Corny', 'email': 'ck@o', 'sn': 'keule'}}, {u'dn': 'cn=John Smith,ou=example,o=test', u'attributes': {u'objectClass': ['top', 'inetOrgPerson'], 'userPassword': 'S3cr3t', 'sn': 'Smith', 'cn': 'John Smith'}}]
\ No newline at end of file
+[{'dn': 'cn=alice,ou=example,o=test', 'attributes': {'oid': '2', 'cn': 'alice', 'accountExpires': 131024988000000000, 'homeDirectory': '/home/alice', 'mobile': ['1234', '45678'], 'userPassword': 'alicepw', 'givenName': 'Alice', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d31', 'email': 'alice@test.com', 'sn': 'Cooper'}}, {'dn': 'cn=bob,ou=example,o=test', 'attributes': {'oid': '3', 'cn': 'bob', 'accountExpires': 9223372036854775807, 'homeDirectory': '/home/bob', 'mobile': '123456', 'userPassword': 'bobpw\xc3\xa9\xc3\xa9\xc3\xa9', 'givenName': 'Robert', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d77', 'email': 'bob@example.com', 'sn': 'Marley'}}, {'dn': 'cn=manager,ou=example,o=test', 'attributes': {'oid': '1', 'cn': 'manager', 'accountExpires': 9223372036854775807, 'mobile': '123354', 'userPassword': 'ldaptest', 'givenName': 'Corny', 'objectGUID': '039b36ef-e7c0-42f3-9bf9-ca6a6c0d4d54', 'email': 'ck@o', 'sn': 'keule'}}, {'dn': 'cn=k\xc3\xb6lbel,ou=example,o=test', 'attributes': {'accoutnExpires': 9223372036854775807, 'oid': '4', 'cn': 'k\xc3\xb6lbel', 'mobile': '123456', 'userPassword': 'mySecret', 'givenName': 'Cornelius', 'objectGUID': '7dd0533c-afe3-4c6f-b49e-af82eaed045c', 'email': 'cko@o', 'sn': 'K\xc3\xb6lbel'}}]
\ No newline at end of file