internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
Missing request headers in /auth requests
Up to version 3.5.

It is not possible to use request headers in policies for all policies that apply to the /auth endpoint.

This is due to the fact, the the request_headers are not added in before_after in ``api/auth.py``.
While it is done in [api/validate.py](https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/validate.py#L144) and in generic [before_after](https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/before_after.py#L176).

This we simply need to add 

    g.request_headers = request.headers


Discussion:

Resolving patch:
diff --git a/privacyidea/api/auth.py b/privacyidea/api/auth.py
--- a/privacyidea/api/auth.py
+++ b/privacyidea/api/auth.py
@@ -97,6 +97,8 @@ def before_request():
     # access_route contains the ip adresses of all clients, hops and proxies.
     g.client_ip = get_client_ip(request,
                                 get_from_config(SYSCONF.OVERRIDECLIENT))
+    # Save the HTTP header in the localproxy object
+    g.request_headers = request.headers
     g.serial = getParam(request.all_data, "serial", default=None)
     g.audit_object.log({"success": False,
                         "client": g.client_ip,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Remove getOTP from documentation
Previously there was a function  ``getOTP`` (endpoint) that would return OTP values.
It looks like it has been removed a while ago.

Check that it does not exist anymore and adapt the [documentation](https://privacyidea.readthedocs.io/en/latest/webui/token_details.html?highlight=getotp#get-otp)



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/sshkeytoken.py b/privacyidea/lib/tokens/sshkeytoken.py
--- a/privacyidea/lib/tokens/sshkeytoken.py
+++ b/privacyidea/lib/tokens/sshkeytoken.py
@@ -19,8 +19,7 @@
 # License along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 __doc__="""The SSHKeyTokenClass provides a TokenClass that stores the public
-SSH key and can give the public SSH key via the getotp function.
-This can be used to manage SSH keys and retrieve the public ssh key
+SSH key. This can be used to manage SSH keys and retrieve the public ssh key
 to import it to authorized keys files.
 
 The code is tested in tests/test_lib_tokens_ssh
@@ -48,8 +47,7 @@
 class SSHkeyTokenClass(TokenClass):
     """
     The SSHKeyTokenClass provides a TokenClass that stores the public
-    SSH key and can give the public SSH key via the getotp function.
-    This can be used to manage SSH keys and retrieve the public ssh key
+    SSH key. This can be used to manage SSH keys and retrieve the public ssh key
     to import it to authorized keys files.
     """
     mode = ['authenticate']


Test patch:
Repository: privacyidea/privacyidea

Issue:
Enrollment wizard menu entry is shown even if user already has a token assigned
If the webui policy `tokenwizard_2nd_token` is unset (users should not be able to use the token wizard if they already have a token enrolled), the `token enrollment wizard` menu entry is still shown in the UI.
The `token_wizard_2nd` variable is send with an empty list from the server but the UI checks for non-existence here:
https://github.com/privacyidea/privacyidea/blob/cd311f443352a049833557f2a87f17a939862092/privacyidea/static/components/token/views/token.html#L30


Discussion:
Interesting:

     ("user" == "user" and [])

avaluate to ``[]``  and not to ``False``.

Resolving patch:
diff --git a/privacyidea/api/lib/postpolicy.py b/privacyidea/api/lib/postpolicy.py
--- a/privacyidea/api/lib/postpolicy.py
+++ b/privacyidea/api/lib/postpolicy.py
@@ -497,9 +497,9 @@ def get_webui_settings(request, response):
                                             user=loginname, realm=realm).action_values(unique=True)
         user_page_size_pol = Match.generic(g, scope=SCOPE.WEBUI, action=ACTION.USERPAGESIZE,
                                            user=loginname, realm=realm).action_values(unique=True)
-        token_wizard_2nd = (role == ROLE.USER
-                            and Match.generic(g, scope=SCOPE.WEBUI, action=ACTION.TOKENWIZARD2ND,
-                                          user=loginname, realm=realm).policies())
+        token_wizard_2nd = bool(role == ROLE.USER
+                                and Match.generic(g, scope=SCOPE.WEBUI, action=ACTION.TOKENWIZARD2ND,
+                                                  user=loginname, realm=realm).policies())
         admin_dashboard = (role == ROLE.ADMIN
                            and Match.generic(g, scope=SCOPE.WEBUI, action=ACTION.ADMIN_DASHBOARD,
                                          user=loginname, realm=realm).any())


Test patch:
Repository: privacyidea/privacyidea

Issue:
Database migration fails if the URI contains '%' signs
If the `SQLALCHEMY_DATABASE_URI` contains query parameters like `ssl_ca=/path/to/cert` the path separators will be url-encoded with `%` signs.
This fails when passing the URI to the alembic configuration (https://alembic.sqlalchemy.org/en/latest/api/config.html#alembic.config.Config.set_main_option).
The `%` signs should be escaped in the URI string before passing it to alembic.


Discussion:

Resolving patch:
diff --git a/migrations/env.py b/migrations/env.py
--- a/migrations/env.py
+++ b/migrations/env.py
@@ -3,6 +3,7 @@
 from sqlalchemy import engine_from_config, pool
 from sqlalchemy.engine.url import make_url
 from logging.config import fileConfig
+from six.moves.urllib.parse import quote
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
@@ -28,10 +29,13 @@ def set_database_url(config):
         if url.startswith("mysql"):
             parsed_url = make_url(url)
             parsed_url.query.setdefault("charset", "utf8")
+            # We need to quote the password in case it contains special chars
+            parsed_url.password = quote(parsed_url.password)
             url = str(parsed_url)
     except Exception as exx:
         print(u"Attempted to set charset=utf8 on connection, but failed: {}".format(exx))
-    config.set_main_option('sqlalchemy.url', url)
+    # set_main_option() requires escaped "%" signs in the string
+    config.set_main_option('sqlalchemy.url', url.replace('%', '%%'))
 
 
 set_database_url(config)
@@ -98,10 +102,10 @@ def run_migrations_online():
     finally:
         connection.close()
 
+
 if context.is_offline_mode():
     print("Running offline")
     run_migrations_offline()
 else:
     print("Running online")
     run_migrations_online()
-


Test patch:
Repository: privacyidea/privacyidea

Issue:
reduce the time ldap3 waits between each round
The ldap3 pooling has a ``POOLING_LOOP_TIMEOUT``. (see 
https://ldap3.readthedocs.io/en/latest/server.html?highlight=serverpool#server-pool)

This is the time ldap3 waits between each round. But it looks like, as if this time is also waited, even if there is only one round.
The POOLING_LOOP_TIMEOUT defaults to 10.
We should either reduce this time or make it configurable.

An easy approach is to set it directly in the LDAPResolver.py:

~~~~python
import ldap3
from ldap3 import MODIFY_REPLACE, MODIFY_ADD, MODIFY_DELETE
from ldap3 import Tls
from ldap3.core.exceptions import LDAPOperationResult
from ldap3.core.results import RESULT_SIZE_LIMIT_EXCEEDED
import ssl

ldap3.set_config_parameter("POOLING_LOOP_TIMEOUT", 0)
~~~~


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -78,7 +78,7 @@
 import hashlib
 import binascii
 from privacyidea.lib.utils import is_true
-from privacyidea.lib.framework import get_app_local_store
+from privacyidea.lib.framework import get_app_local_store, get_app_config_value
 import datetime
 
 from privacyidea.lib import _
@@ -97,6 +97,12 @@
 SERVERPOOL_ROUNDS = 2
 # The number of seconds a non-responding server is removed from the server pool
 SERVERPOOL_SKIP = 30
+# The number of seconds that ldap3 waits if no server is left in the pool, before
+# starting the next round
+pooling_loop_timeout = get_app_config_value("PI_LDAP_POOLING_LOOP_TIMEOUT", 10)
+log.info("Setting system wide POOLING_LOOP_TIMEOUT to {0!s}.".format(pooling_loop_timeout))
+ldap3.set_config_parameter("POOLING_LOOP_TIMEOUT", pooling_loop_timeout)
+
 # 1 sec == 10^9 nano secs == 10^7 * (100 nano secs)
 MS_AD_MULTIPLYER = 10 ** 7
 MS_AD_START = datetime.datetime(1601, 1, 1)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Allow auth_cache to cache only for a limited number of authentications
The ``auth_cache`` is policy in scope=auth.
http://privacyidea.readthedocs.io/en/latest/policies/authentication.html#auth-cache

Currently you can define it like 4h/4m, which would mean cache for 4 hours if every 4 minutes there is a request. 

We need to enhance this login to "cache for 5 seconds, but only allow 2 authentications within this 5 seconds".

This could be specified like "5s/2".

To achieve this, we need to

1. allow seconds in the auth_cache
2. count the authentications performed by the auth_cache. Therefore we need to add a column for counting the auth's https://github.com/privacyidea/privacyidea/blob/master/privacyidea/models.py#L2431


Discussion:
This was e.g. needed for the ldap proxy, if you know, that the same OTP would be used within the next 5 seconds, but only two times.

Currently the auth_cache allows to specify  2h/5m, which means, that the the user may authenticate for 2h (checked in the column ``first_auth``, if the last authentication happened less then 5 minutes ago (checkd in the column ``last_auth``).

Thus we need the additional column ``auth_count``. That simply records, how often this auth_cache entry was used.
The the setting 10s/2 will mean, check ``first_auth`` to be less than 10s ago and theck ``auth_count`` to be less than 2.
(If it exceeds the auth_cache entry can be deleted again.)
Forgot to add the migration script for the database...

Resolving patch:
diff --git a/migrations/versions/d415d490eb05_.py b/migrations/versions/d415d490eb05_.py
new file mode 100644
--- /dev/null
+++ b/migrations/versions/d415d490eb05_.py
@@ -0,0 +1,33 @@
+"""v3.6: Add auth_count column to authcache table
+
+Revision ID: d415d490eb05
+Revises: 9155f0d3d028
+Create Date: 2021-04-06 21:19:25.931603
+privacyIDEA Version: 3.6
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'd415d490eb05'
+down_revision = '9155f0d3d028'
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.exc import OperationalError, ProgrammingError, InternalError
+
+
+def upgrade():
+    try:
+        op.add_column('authcache', sa.Column('auth_count', sa.Integer(), nullable=True))
+    except (OperationalError, ProgrammingError, InternalError) as exx:
+        if "duplicate column name" in str(exx.orig).lower():
+            print("Good. Column adminrealm already exists.")
+        else:
+            print(exx)
+    except Exception as exx:
+        print("Could not add the column 'auth_count' to table 'authcache'")
+        print(exx)
+
+
+def downgrade():
+    op.drop_column('authcache', 'auth_count')


Test patch:
Repository: privacyidea/privacyidea

Issue:
Reduce to one head in migration scripts
In the current development branch we ended up with two heads in the alembic history. Check with:

    pi-manage db history

We need to reorganize the migration scripts to only have one head in v3.6.


Discussion:

Resolving patch:
diff --git a/migrations/versions/449903fb6e35_.py b/migrations/versions/449903fb6e35_.py
--- a/migrations/versions/449903fb6e35_.py
+++ b/migrations/versions/449903fb6e35_.py
@@ -1,5 +1,4 @@
-"""Add table 'radiusserver' to configure the settings of
-remote RADIUS servers.
+"""Add table 'radiusserver' to configure the settings of remote RADIUS servers.
 
 Revision ID: 449903fb6e35
 Revises: 4023571658f8
diff --git a/migrations/versions/888b56ed5dcb_.py b/migrations/versions/888b56ed5dcb_.py
--- a/migrations/versions/888b56ed5dcb_.py
+++ b/migrations/versions/888b56ed5dcb_.py
@@ -1,14 +1,14 @@
 """v3.6: Add table for custom user attributes
 
 Revision ID: 888b56ed5dcb
-Revises: d5870fd2f2a4
+Revises: d415d490eb05
 Create Date: 2021-02-10 12:17:40.880224
 
 """
 
 # revision identifiers, used by Alembic.
 revision = '888b56ed5dcb'
-down_revision = 'd5870fd2f2a4'
+down_revision = 'd415d490eb05'
 
 from alembic import op
 import sqlalchemy as sa


Test patch:
Repository: privacyidea/privacyidea

Issue:
Stamp database version when creating the db with pi-manage
Running `pi-manage createdb` does not stamp the database version thus we always need to call `pi-manage db stamp -d <path to migrations> head` afterwards.
We can call the `stamp()` function from [flask-migrate](https://flask-migrate.readthedocs.io/en/latest/) directly after creating the database.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/utils/__init__.py b/privacyidea/lib/utils/__init__.py
--- a/privacyidea/lib/utils/__init__.py
+++ b/privacyidea/lib/utils/__init__.py
@@ -33,8 +33,7 @@
 import base64
 import qrcode
 import sqlalchemy
-from six.moves.urllib.parse import urlunparse, urlparse, urlencode
-from six.moves import zip_longest
+from six.moves.urllib.parse import urlencode
 from io import BytesIO
 import string
 import re
@@ -46,7 +45,7 @@
 import hashlib
 import traceback
 import threading
-import pkg_resources
+import importlib_metadata
 import time
 import html
 
@@ -1252,7 +1251,7 @@ def get_version_number():
     """
     version = "unknown"
     try:
-        version = pkg_resources.get_distribution("privacyidea").version
+        version = importlib_metadata.version("privacyidea")
     except:
         log.info("We are not able to determine the privacyidea version number.")
     return version
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -45,6 +45,7 @@ def get_file_list(file_path):
                     "future>=0.18.2;python_version<'3.0'",
                     "google-auth>=1.24.0",
                     "huey[redis]>=1.11.0",
+                    "importlib_metadata>=2.1.1",
                     "ldap3>=2.6",
                     "netaddr>=0.7.12",
                     "passlib[bcrypt]>=1.7.0",


Test patch:
Repository: privacyidea/privacyidea

Issue:
Testing LDAP connections does not honor the TLS setting
Testing an LDAP connection from the UI does not take the set TLS version into account. Thus testing the connection works but the resolver itself failed with a TLS error (in case the LDAP server doesn't speak TLSv1.3).


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -505,7 +505,7 @@ def _get_tls_context(ldap_uri=None, start_tls=False, tls_version=None, tls_verif
                 tls_version = int(DEFAULT_TLS_PROTOCOL)
             # If TLS_VERSION is 2, set tls_options to use TLS v1.3
             if not tls_options:
-                tls_options = TLS_OPTIONS_1_3 if tls_version == int(ssl.PROTOCOL_TLS) else None
+                tls_options = TLS_OPTIONS_1_3 if int(tls_version) == int(ssl.PROTOCOL_TLS) else None
             if tls_verify:
                 tls_ca_file = tls_ca_file or DEFAULT_CA_FILE
             else:


Test patch:
Repository: privacyidea/privacyidea

Issue:
LDAPResolver fails with "AttributeError('module' object has no attribute 'PROTOCOL_TLS')"
On CentOS 7 with Python 2.7.5 instantiating the LDAPResolver fails with the message above.
This is due to the `ssl.PROTOCOL_TLS` constant being only available since Python 2.7.13.
Python versions before 2.7.13 can use the [`PROTOCOL_SSLv23`](https://docs.python.org/release/2.7.13/library/ssl.html#ssl.PROTOCOL_SSLv23) constant.

PS: Please switch to Python 3 ASAP


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -111,10 +111,17 @@
 else:
     DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
 
-DEFAULT_TLS_PROTOCOL = ssl.PROTOCOL_TLS
+try:
+    TLS_NEGOTIATE_PROTOCOL = ssl.PROTOCOL_TLS
+except AttributeError as _e:
+    # this is Python < 2.7.13, it does not provide ssl.PROTOCOL_TLS
+    TLS_NEGOTIATE_PROTOCOL = ssl.PROTOCOL_SSLv23
+
+DEFAULT_TLS_PROTOCOL = TLS_NEGOTIATE_PROTOCOL
 
 TLS_OPTIONS_1_3 = (ssl.OP_NO_TLSv1_2, ssl.OP_NO_TLSv1_1, ssl.OP_NO_TLSv1, ssl.OP_NO_SSLv3)
 
+
 class LockingServerPool(ldap3.ServerPool):
     """
     A ``ServerPool`` subclass that uses a RLock to synchronize invocations of
@@ -516,7 +523,7 @@ def _get_tls_context(ldap_uri=None, start_tls=False, tls_version=None, tls_verif
                 tls_version = int(DEFAULT_TLS_PROTOCOL)
             # If TLS_VERSION is 2, set tls_options to use TLS v1.3
             if not tls_options:
-                tls_options = TLS_OPTIONS_1_3 if int(tls_version) == int(ssl.PROTOCOL_TLS) else None
+                tls_options = TLS_OPTIONS_1_3 if int(tls_version) == int(TLS_NEGOTIATE_PROTOCOL) else None
             if tls_verify:
                 tls_ca_file = tls_ca_file or DEFAULT_CA_FILE
             else:


Test patch:
Repository: privacyidea/privacyidea

Issue:
upper case realm break login to webui
version 3.6 and above,

When logging in the webui with an upper case REALM name, we get this error.

~~~~
ERR905: Cannot pass user_object (<Administrator.netknights-lab@netknights-lab>) as well as user (Administrator), resolver (None), realm (NETKNIGHTS-LAB)in policy (None, 'user', 'enrollYUBIKEY')"}
~~~~

This is due to a user comparison in the policy module.

mitigation: Avoid this by using lower case realm.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/policy.py b/privacyidea/lib/policy.py
--- a/privacyidea/lib/policy.py
+++ b/privacyidea/lib/policy.py
@@ -718,7 +718,7 @@ def match_policies(self, name=None, scope=None, realm=None, active=None,
             # if a user_object is passed, we check, if it differs from potentially passed user, resolver, realm:
             if (user and user.lower() != user_object.login.lower()) \
                     or (resolver and resolver.lower() != user_object.resolver.lower()) \
-                    or (realm and realm != user_object.realm):
+                    or (realm and realm.lower() != user_object.realm):
                 tb_str = ''.join(traceback.format_stack())
                 log.warning("Cannot pass user_object as well as user, resolver, realm "
                             "in policy {0!s}. "


Test patch:
diff --git a/tests/test_lib_policy.py b/tests/test_lib_policy.py
--- a/tests/test_lib_policy.py
+++ b/tests/test_lib_policy.py
@@ -1107,6 +1107,20 @@ def test_26_match_policies_user_object(self):
         self.assertEqual(P.get_action_values(action=ACTION.OTPPIN, scope=SCOPE.AUTH, user_object=None),
                          {"userstore": ["act1", "act2"], "none": ["act3", "act4", "act5"]})
 
+        # if we pass user_obj ornelius, we only get 4 policies
+        self.assertEqual(set(p['name'] for p in P.match_policies(user_object=cornelius)),
+                         {"act1", "act2", "act3", "act5"})
+        # Provide a user object and parameters
+        self.assertEqual(set(p['name'] for p in P.match_policies(user_object=cornelius,
+                                                                 user="cornelius",
+                                                                 realm="realm1")),
+                         {"act1", "act2", "act3", "act5"})
+        # For some reason pass the same user_obj and parameters, but realm in upper case
+        self.assertEqual(set(p['name'] for p in P.match_policies(user_object=cornelius,
+                                                                 user="cornelius",
+                                                                 realm="REALM1")),
+                         {"act1", "act2", "act3", "act5"})
+
         delete_policy("act1")
         delete_policy("act2")
         delete_policy("act3")
Repository: privacyidea/privacyidea

Issue:
Update requirements for v3.7
As described in the [Wiki](https://github.com/privacyidea/privacyidea/wiki/Development-workflow), we try to update all dependencies at the beginning of a new release cycle.
This time there are several major upgrades which break things and we need to investigate if we want to include these (and the corresponding changes) in the upcoming v3.7 version.


**Non-Flask packages:**
- ~~`pydash`: `4.9.2` -> `5.0.1`~~
- `PyJWT`: `1.7.1` -> `2.1.0`
- `idna`: `2.10` -> `3.2`
- `google-auth`: `1.32.1` -> `2.0.1`



Discussion:
Let's do the upgrade of SQL Alchemy with removing soup in 3.8.

Resolving patch:
diff --git a/privacyidea/api/lib/utils.py b/privacyidea/api/lib/utils.py
--- a/privacyidea/api/lib/utils.py
+++ b/privacyidea/api/lib/utils.py
@@ -333,7 +333,7 @@ def verify_auth_token(auth_token, required_role=None):
                     log.warning(u"Unsupported JWT algorithm in PI_TRUSTED_JWT.")
             except jwt.DecodeError as err:
                 log.info(u"A given JWT definition does not match.")
-            except jwt.ExpiredSignature as err:
+            except jwt.ExpiredSignatureError as err:
                 # We have the correct token. It expired, so we raise an error
                 raise AuthError(_("Authentication failure. Your token has expired: {0!s}").format(err),
                                 id=ERROR.AUTHENTICATE_TOKEN_EXPIRED)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add PyKCS11 to packaging
We should add PyKCS11 to the packaging or debian and rpm packages. This can be either achieved by adding it as a dependency in `setup.py` (which would install it on all installations) or by explicitly only installing it in RPM and DEB packages.

I think we also have additional python modules like pgsql that are only packed in the DEB and RPM.

Also see https://github.com/NetKnights-GmbH/centos7/issues/25


Discussion:
The most straightforward way might be to add an `extras_require` to the `setup.py`.

See https://github.com/privacyidea/privacyidea/blob/master/setup.py#L121

Then we can add

~~~~python
  'hsm' : ['PyKCS11']
~~~~

And the build process could build packages like

    pip install privacyidea[hsm]



Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -118,7 +118,8 @@ def get_scripts(dir):
                  "pytest-cov>=2.5.1",
                  "responses>=0.9.0",
                  "testfixtures>=6.14.2"],
-        'postgres': ['psycopg2>=2.8.3']
+        'postgres': ['psycopg2>=2.8.3'],
+        'hsm': ['PyKCS11>=1.5.10']
     },
     install_requires=install_requires,
     include_package_data=True,


Test patch: