internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
2step: allow lower case
I think the user should be allowed to enter the client part as lower case.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/utils.py b/privacyidea/lib/utils.py
--- a/privacyidea/lib/utils.py
+++ b/privacyidea/lib/utils.py
@@ -273,16 +273,23 @@ def checksum(msg):
                 crc = crc ^ 0x8408
     return crc
 
-def decode_base32check(encoded_data):
+
+def decode_base32check(encoded_data, always_upper=True):
     """
     Decode arbitrary data which is given in the following format::
 
         strip_padding(base32(sha1(payload)[:4] + payload))
 
     Raise a ParameterError if the encoded payload is malformed.
+    :param encoded_data: The base32 encoded data.
+    :type encoded_data: basestring
+    :param always_upper: If we should convert lowercase to uppercase
+    :type always_upper: bool
     :return: hex-encoded payload
     """
     # First, add the padding to have a multiple of 8 bytes
+    if always_upper:
+        encoded_data = encoded_data.upper()
     encoded_length = len(encoded_data)
     if encoded_length % 8 != 0:
         encoded_data += "=" * (8 - (encoded_length % 8))
@@ -301,6 +308,7 @@ def decode_base32check(encoded_data):
         raise ParameterError("Malformed base32check data: Incorrect checksum")
     return binascii.hexlify(payload)
 
+
 def sanity_name_check(name, name_exp="^[A-Za-z0-9_\-\.]+$"):
     """
     This function can be used to check the sanity of a name like a resolver,


Test patch:
diff --git a/tests/test_lib_token.py b/tests/test_lib_token.py
--- a/tests/test_lib_token.py
+++ b/tests/test_lib_token.py
@@ -1190,6 +1190,21 @@ def test_50_otpkeyformat(self):
         self.assertEqual(tokenobject.token.get_otpkey().getKey(),
                          binascii.hexlify(otpkey))
         remove_token("NEW002")
+
+        # successful base32check encoding, but lower case
+        base32check_encoding = base64.b32encode(checksum + otpkey).strip("=")
+        base32check_encoding = base32check_encoding.lower()
+        tokenobject = init_token({"serial": "NEW002", "type": "hotp",
+                                  "otpkey": base32check_encoding,
+                                  "otpkeyformat": "base32check"},
+                                 user=User(login="cornelius",
+                                           realm=self.realm1))
+        self.assertTrue(tokenobject.token.tokentype == "hotp",
+                        tokenobject.token)
+        self.assertEqual(tokenobject.token.get_otpkey().getKey(),
+                         binascii.hexlify(otpkey))
+        remove_token("NEW002")
+
         # base32check encoding with padding
         base32check_encoding = base64.b32encode(checksum + otpkey)
         tokenobject = init_token({"serial": "NEW003", "type": "hotp",
Repository: privacyidea/privacyidea

Issue:
TypeError in HTTPSmsProvider
The HTTPSmsProvider will throw a TypeError exception if a timeout is specified in the HTTP SMS Gateway.

This is due to these settings:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/smsprovider/HttpSMSProvider.py#L91
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/smsprovider/HttpSMSProvider.py#L75

The timeout is saved as a string in the database and thus will be passed as string to ``requests.get`` or ``request.post``.

We need cast the timeout to an integer or float.

# Temporary workaround

You can delete the timeout in the HTTP Gateway config. A default value of ``3`` will be used.



Discussion:
The code seemed to be added on Nov 1st. 2017
https://github.com/privacyidea/privacyidea/commit/14140faf63c9ffad774c3f0a86be6e74d623abd4#diff-f534f3bf7b9ffc8acb661fd353622a86

So this is indeed only relevant for version 2.21 and 2.21.1

Resolving patch:
diff --git a/privacyidea/lib/smsprovider/HttpSMSProvider.py b/privacyidea/lib/smsprovider/HttpSMSProvider.py
--- a/privacyidea/lib/smsprovider/HttpSMSProvider.py
+++ b/privacyidea/lib/smsprovider/HttpSMSProvider.py
@@ -3,6 +3,8 @@
 #    E-mail: info@privacyidea.org
 #    Contact: www.privacyidea.org
 #
+#    2018-01-10 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#               Fix type cast for timeout
 #    2016-06-14 Cornelius Kölbel <cornelius@privacyidea.org>
 #               Add properties for new SMS provider model
 #    2016-04-08 Cornelius Kölbel <cornelius@privacyidea.org>
@@ -132,7 +134,7 @@ def submit_message(self, phone, message):
                       data=data,
                       verify=ssl_verify,
                       auth=basic_auth,
-                      timeout = timeout,
+                      timeout=float(timeout),
                       proxies=proxies)
         log.debug("queued SMS on the HTTP gateway. status code returned: {0!s}".format(
                   r.status_code))
diff --git a/privacyidea/lib/tokens/smstoken.py b/privacyidea/lib/tokens/smstoken.py
--- a/privacyidea/lib/tokens/smstoken.py
+++ b/privacyidea/lib/tokens/smstoken.py
@@ -293,6 +293,7 @@ def create_challenge(self, transactionid=None, options=None):
                 info = ("The PIN was correct, but the "
                         "SMS could not be sent: %r" % e)
                 log.warning(info)
+                log.debug("{0!s}".format(traceback.format_exc()))
                 return_message = info
 
         validity = self._get_sms_timeout()


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add VASCO token type to web UI
As we cannot generate new secrets by ourselves, we just add a possibility to enter a hex-encoded blob directly.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/vascotoken.py b/privacyidea/lib/tokens/vascotoken.py
--- a/privacyidea/lib/tokens/vascotoken.py
+++ b/privacyidea/lib/tokens/vascotoken.py
@@ -98,7 +98,10 @@ def get_class_info(key=None, ret='all'):
         res = {'type': 'vasco',
                'title': 'VASCO Token',
                'description': _('VASCO Token: Authentication using VASCO tokens'),
-               'user': ["enroll"],
+               # If this was set, the user could enroll a Vasco token via the API
+               #'user': ["enroll"],
+               # only administrators can enroll the token in the UI
+               'ui_enroll': ["admin"],
                'policy': {},
                }
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
Editing SMTP Server definition clears the password
Steps to reproduce:
* Have a SMTP Server definition with password
* Edit the SMTP Server, but do not change anything
* Click save
* SMTP Password field is now empty


Discussion:

Resolving patch:
diff --git a/privacyidea/api/smtpserver.py b/privacyidea/api/smtpserver.py
--- a/privacyidea/api/smtpserver.py
+++ b/privacyidea/api/smtpserver.py
@@ -34,6 +34,7 @@
                         required,
                         send_result)
 from ..lib.log import log_with
+from ..lib.crypto import decryptPassword, FAILED_TO_DECRYPT_PASSWORD
 from ..lib.policy import ACTION
 from ..api.lib.prepolicy import prepolicy, check_base_action
 from flask import g
@@ -91,9 +92,14 @@ def list_smtpservers():
     res = {}
     server_list = get_smtpservers()
     for server in server_list:
+        decrypted_password = decryptPassword(server.config.password)
+        # If the database contains garbage, use the empty password as fallback
+        if decrypted_password == FAILED_TO_DECRYPT_PASSWORD:
+            decrypted_password = ""
         res[server.config.identifier] = {"server": server.config.server,
                                          "tls": server.config.tls,
                                          "username": server.config.username,
+                                         "password": decrypted_password,
                                          "port": server.config.port,
                                          "description":
                                              server.config.description,


Test patch:
diff --git a/tests/test_api_smtpserver.py b/tests/test_api_smtpserver.py
--- a/tests/test_api_smtpserver.py
+++ b/tests/test_api_smtpserver.py
@@ -50,7 +50,7 @@ def test_01_create_server(self):
             self.assertEqual(server1.get("server"), "1.2.3.4")
             self.assertEqual(server1.get("sender"), "privacyidea@local")
             self.assertEqual(server1.get("username"), "cornelius")
-            self.assertTrue("password" not in server1)
+            self.assertEqual(server1.get("password"), "secret")
 
         # delete server
         with self.app.test_request_context('/smtpserver/server1',
Repository: privacyidea/privacyidea

Issue:
Fix SMS Validity
There is a mixup in the "SMS timeout".

On the token config tab, there has been the "Timeout", which was used as the validity time for the challenge (how long the SMS would be valid.), NOT for an actual network timeout. The naming was a bit confusing.
entering this ``sms.providerTimeout`` has been disabled during the redesign of the SMS gateways. 

The naming is simply very bad. 

We should take a look at the email token, which uses ``email.validtime``.
Anyway: we should activate the field in the token-config UI again but rename the UI label, so that it is more clear, what this value acutally does.

Rename this label
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/components/config/views/config.token.sms.html#L34

Make it matching the email config:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/components/config/views/config.token.email.html#L19



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/smstoken.py b/privacyidea/lib/tokens/smstoken.py
--- a/privacyidea/lib/tokens/smstoken.py
+++ b/privacyidea/lib/tokens/smstoken.py
@@ -296,7 +296,6 @@ def create_challenge(self, transactionid=None, options=None):
                 log.debug("{0!s}".format(traceback.format_exc()))
                 return_message = info
 
-        validity = self._get_sms_timeout()
         expiry_date = datetime.datetime.now() + \
                                     datetime.timedelta(seconds=validity)
         attributes['valid_until'] = "{0!s}".format(expiry_date)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Allow token janitor to export PW token
The tokenjanitor should be able to export PW token to the PSKC file.

This can be problematic, since there is no corresponding scheme.
We need to use the hotp scheme.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/importotp.py b/privacyidea/lib/importotp.py
--- a/privacyidea/lib/importotp.py
+++ b/privacyidea/lib/importotp.py
@@ -476,7 +476,7 @@ def parsePSKCdata(xml_data,
         except Exception as exx:
             log.debug("Can not get serial string from device info {0!s}".format(exx))
         algo = key["algorithm"]
-        token["type"] = algo[-4:].lower()
+        token["type"] = algo.split(":")[-1].lower()
         parameters = key.algorithmparameters
         token["otplen"] = parameters.responseformat["length"] or 6
         try:
@@ -603,7 +603,7 @@ def export_pskc(tokenobj_list, psk=None):
 """.format(encrypted_mackey=encrypted_mackey), "html.parser")
 
     for tokenobj in tokenobj_list:
-        if tokenobj.type.lower() not in ["totp", "hotp"]:
+        if tokenobj.type.lower() not in ["totp", "hotp", "pw"]:
             continue
         type = tokenobj.type.lower()
         issuer = "privacyIDEA"


Test patch:
diff --git a/tests/test_lib_importotp.py b/tests/test_lib_importotp.py
--- a/tests/test_lib_importotp.py
+++ b/tests/test_lib_importotp.py
@@ -245,6 +245,22 @@
       </Policy>
     </Key>
   </KeyPackage>
+  <KeyPackage>
+    <DeviceInfo>
+      <Manufacturer>privacyIDEA rocks</Manufacturer>
+      <SerialNo>PW001</SerialNo>
+    </DeviceInfo>
+    <Key Id="PW001" Algorithm="urn:ietf:params:xml:ns:keyprov:pskc:pw">
+      <AlgorithmParameters>
+        <ResponseFormat Length="12" Encoding="DECIMAL"/>
+      </AlgorithmParameters>
+      <Data>
+        <Secret>
+          <PlainValue>MTIzNDU2Nzg5MDEy</PlainValue>
+        </Secret>
+      </Data>
+    </Key>
+  </KeyPackage>
 </KeyContainer>
 '''
 
@@ -524,13 +540,18 @@ def test_02_import_yubikey(self):
 
     def test_03_import_pskc(self):
         tokens = parsePSKCdata(XML_PSKC)
-        self.assertEqual(len(tokens), 6)
+        self.assertEqual(len(tokens), 7)
         self.assertEqual(tokens["1000133508267"].get("type"), "hotp")
         self.assertEqual(tokens["2600135004013"].get("type"), "totp")
         # Check the TOTP counter...
         self.assertEqual(tokens["2600135004013"].get("counter"), "121212")
         self.assertEqual(tokens["2600135004013"].get("timeShift"), "-122")
         self.assertEqual(tokens["2600135004013"].get("timeStep"), "60")
+        # check the PW token
+        self.assertEqual(tokens["PW001"].get("type"), "pw")
+        self.assertEqual(tokens["PW001"].get("otplen"), "12")
+        # The secret (password) of the pw token is "123456789012"
+        self.assertEqual(tokens["PW001"].get("otpkey"), binascii.hexlify("123456789012"))
 
     def test_04_import_pskc_aes(self):
         encryption_key_hex = "12345678901234567890123456789012"
Repository: privacyidea/privacyidea

Issue:
LDAP Resolver + AD: Listing users writes misguiding errors in log
* Use LDAP Resolver with AD
* Search for users, works well
* Have a look into the log:
```
[2018-02-24 13:26:44,329][10630][139652256793792][ERROR][privacyidea.lib.resolvers.LDAPIdResolver:564] Error during fetching LDAP objects: AttributeError("'NoneType' object has no attribute 'items'",)
[2018-02-24 13:26:44,329][10630][139652256793792][ERROR][privacyidea.lib.resolvers.LDAPIdResolver:564] Error during fetching LDAP objects: AttributeError("'NoneType' object has no attribute 'items'",)
[2018-02-24 13:26:44,329][10630][139652256793792][ERROR][privacyidea.lib.resolvers.LDAPIdResolver:564] Error during fetching LDAP objects: AttributeError("'NoneType' object has no attribute 'items'",)
```
... which looks bad, but is totally harmless.

The error is thrown when the LDAP Resolver iterates over the list of user entries returned by the LDAP server:
```python
        # returns a generator of dictionaries
        for entry in g:
            # Simple fix for ignored sizelimit with Active Directory
            if len(ret) >= self.sizelimit:
                break
            try:
                attributes = entry.get("attributes")
                user = self._ldap_attributes_to_user_object(attributes)
                user['userid'] = self._get_uid(entry, self.uidtype)
                ret.append(user)
            except Exception as exx:  # pragma: no cover
                log.error("Error during fetching LDAP objects: {0!r}".format(exx))
                log.debug("{0!s}".format(traceback.format_exc()))
```

In the case of AD, a list of user entries may contain ``searchResRef`` entries, where ``entry`` looks like the following:
```python
{'type': 'searchResRef', 'uri': [u'ldap://testfoo.intranet/CN=Configuration,DC=testfoo,DC=intranet']}
```
So then, ``attributes`` is None and passed to ``_ldap_attributes_to_user_object``, which throws the above error.

As the error is harmless, we should probably avoid writing it to the log. A simple fix would be to ignore the ``searchResRef`` entries.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -555,6 +555,9 @@ def getUserList(self, searchDict):
             # Simple fix for ignored sizelimit with Active Directory
             if len(ret) >= self.sizelimit:
                 break
+            # Fix for searchResRef entries which have no attributes
+            if entry.get('type') == 'searchResRef':
+                continue
             try:
                 attributes = entry.get("attributes")
                 user = self._ldap_attributes_to_user_object(attributes)


Test patch:
Repository: privacyidea/privacyidea

Issue:
SQLAlchemy and Oracle send either Bool or String
There is a different behaviour in different SQL Alchemy versions with Oracle.

## SQLAlchemy Version 1.1.11:

**A boolean "False" is sent as string "False" to the database.**
So saving an LDAP resolver works out fine. A paramter is saved as string "False" to the resolverconfig table.

## SQLAlchemy Version 1.2.1

**A boolean "False" is sent as bool "False" to the database.**
With Oracle this results in an ``ORA-00932: Inconsistent datatype``.

We should take care, that privacyIDEA works with SQLA 1.2.1 and Oracle. We should **not** pin this to 1.1.11.

* check for SQLA switches
* check if we need to convert the data.


Discussion:

Resolving patch:
diff --git a/privacyidea/models.py b/privacyidea/models.py
--- a/privacyidea/models.py
+++ b/privacyidea/models.py
@@ -946,10 +946,10 @@ def __init__(self, resolver_id=None,
                                        .filter_by(name=resolver)\
                                        .first()\
                                        .id
-        self.Key = Key
-        self.Value = Value
-        self.Type = Type
-        self.Description = Description
+        self.Key = unicode(Key)
+        self.Value = unicode(Value)
+        self.Type = unicode(Type)
+        self.Description = unicode(Description)
 
     def save(self):
         c = ResolverConfig.query.filter_by(resolver_id=self.resolver_id,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Problematic ldap3 / pyasn1 version combinations
This is an issue to keep track of problematic combinations of ldap3 / pyasn1 versions. See #887, #911.

Functionality I check for each combination:
* STARTTLS
* Complete user info on /validate/check
* Size Limit < number of users

Everything tested on a62fd9b3d0555b43cf31be3c4092a549fd73f724

<table>
<tr>
<th></th><th>pyasn 0.1.9</th><th>pyasn 0.4.2</th>
<tr><td>ldap3 2.1.1</td><td>ok</td><td><ul><li>STARTTLS fails (#885)</li></ul></td></tr>
<tr><td>ldap3 2.3</td><td colspan="2">
<ul><li>Empty user list (#887)</li></ul></td></tr>
<tr><td>ldap3 2.4</td><td colspan="2">
<ul><li>Empty user list (#887)</li><li>Empty user info (#911)</li></ul></td></tr>
<tr><td>ldap3 2.4.1</td><td colspan=2">
<ul><li>Empty user list (#887)</li>
<li><b>Error on /validate/check and resolver error</b> in token list (#911)</li>
<li>cannot search for unicode users (#980)</li></td></tr>
</table>

Combinations in current 2.21.4 packages:
* ``pip install privacyidea``:
```
ldap3==2.1.1
privacyIDEA==2.21.4
pyasn1==0.4.2
```
* CentOS:
```
ldap3==2.3
privacyIDEA==2.21.4
pyasn1==0.4.2
```
* Ubuntu (Launchpad):
```
ldap3==2.1.1
privacyIDEA==2.21.4
pyasn1==0.1.9
```


Discussion:
According to the table above, ldap==2.1.1+pyasn1==0.1.9 is the only combination which currently works well. However, according to the [pyasn1 changelog](https://github.com/etingof/pyasn1/blob/6910ce9/CHANGES.rst), there were at least 3 security fixes since 0.1.9, so we should definitely try to integrate the latest pyasn1. I'll look into this.
ldap3 2.5 [was released yesterday](http://ldap3.readthedocs.io/changelog.html) and it seems like ldap3 2.5+pyasn 0.4.2 indeed fixes #885, #887, #911 and #980! :)

... so we might be able to remove the pinned ldap3 version in setup.py in the near future?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -51,10 +51,10 @@ def get_file_list(file_path):
                     "docutils>=0.12",
                     "funcparserlib>=0.3.6",
                     "itsdangerous>=0.24",
-                    "ldap3==2.1.1",
+                    "ldap3>=2.5",
                     "netaddr>=0.7.12",
                     "passlib>=1.6.2",
-                    "pyasn1>=0.1.7",
+                    "pyasn1>=0.4.2",
                     "pyOpenSSL>=0.15.1",
                     "pycparser>=2.10",
                     "pycrypto>=2.6.1",


Test patch:
Repository: privacyidea/privacyidea

Issue:
make the backend honor the browser language settings
Currently, if the server runs in a different locale environment (i.e. LANG=de_DE.UTF-8) than the browsers language setting, you'll get mixed language output, depending if the text comes from the frontend (javascript) oder the backend (python).
This can be seen when creating a new policy. The action description is retrieved from the backend and thus delivered in the current backend language.
it would be more consistent if the backend could switch the language on each request based on the browser settings.


Discussion:
The browser passes the accepted languages in the header:

    Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7

we could catch this in the ``before_after.py`` and pass this down to the libraries.
We could add the paramter ``languages`` here: https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/__init__.py#L22

The question is, if a language change in the browser (since another user issues a request with another browser langauge) has the desired effect or if we run into caching issues.
 

Resolving patch:
diff --git a/privacyidea/app.py b/privacyidea/app.py
--- a/privacyidea/app.py
+++ b/privacyidea/app.py
@@ -205,6 +205,10 @@ def create_app(config_name="development",
 
     @babel.localeselector
     def get_locale():
+        # if we are not in the request context, return None to use the default
+        # locale
+        if not request:
+            return None
         # otherwise try to guess the language from the user accept
         # header the browser transmits.  We support de/fr/en in this
         # example.  The best match wins.
diff --git a/privacyidea/lib/__init__.py b/privacyidea/lib/__init__.py
--- a/privacyidea/lib/__init__.py
+++ b/privacyidea/lib/__init__.py
@@ -1,4 +1,4 @@
-'''
+"""
 Based on the database models, which are tested in tests/test_db_model.py,
 there are different modules.
 
@@ -14,11 +14,6 @@
 in a realm. And of course the user object itself can be found in a resolver.
 But you need to have working resolver.py and realm.py to be able to
 work with user.py
-'''
+"""
 
-import gettext
-import os
-curdir = os.path.dirname(__file__)
-_ = gettext.translation('messages',
-                        '{0!s}/../translations/'.format(curdir),
-                        fallback=True).ugettext
+from flask_babel import gettext as _


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add command to dump audit data
for #829 we should have a command, maybe integrated into pi-manage, to dump the audit log to a csv file.

We can integrate it into pi-manage. See the subcommand "rotate_audit".
MAybe we can add a subcommand "audit" with "rotate" and "dump".
"rotate_audit" needs to stay for compat.

    pi-manage audit rotate
    pi-manage audit dump --days 7




Discussion:

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -353,8 +353,8 @@ def _get_logentry_attribute(key):
     def csv_generator(self, param=None, user=None, timelimit=None):
         """
         Returns the audit log as csv file.
-        :param config: The current flask app configuration
-        :type config: dict
+        :param timelimit: Limit the number of dumped entries by time
+        :type timelimit: datetime.timedelta
         :param param: The request parameters
         :type param: dict
         :param user: The user, who issued the request


Test patch: