internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
passthru policy in audit log
The passthru policy is only written to the audit log, if the password was correct. This can be misleading.
It should also be written to the audit log, if the passthru was actually checked.

See here: https://community.privacyidea.org/t/release-privacyidea-3-7-2/2560/7

We would simply have to move the `add_policy` calls to an upper location.
https://github.com/privacyidea/privacyidea/blob/e668d96b20c5d4480ddda30649dee5259471d7a4/privacyidea/lib/policydecorators.py#L296


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/policydecorators.py b/privacyidea/lib/policydecorators.py
--- a/privacyidea/lib/policydecorators.py
+++ b/privacyidea/lib/policydecorators.py
@@ -290,10 +290,10 @@ def auth_user_passthru(wrapped_function, user_object, passw, options=None):
             policy_object.check_for_conflicts(pass_thru, "passthru")
             pass_thru_action = pass_thru[0].get("action").get("passthru")
             policy_name = pass_thru[0].get("name")
+            g.audit_object.add_policy([p.get("name") for p in pass_thru])
             if pass_thru_action in ["userstore", True]:
                 # Now we need to check the userstore password
                 if user_object.check_password(passw):
-                    g.audit_object.add_policy([p.get("name") for p in pass_thru])
                     return True, {"message": u"against userstore due to '{!s}'".format(
                                       policy_name)}
             else:
@@ -303,7 +303,6 @@ def auth_user_passthru(wrapped_function, user_object, passw, options=None):
                 radius = get_radius(pass_thru_action)
                 r = radius.request(radius.config, user_object.login, passw)
                 if r:
-                    g.audit_object.add_policy([p.get("name") for p in pass_thru])
                     # TODO: here we can check, if the token should be assigned.
                     passthru_assign = Match.user(g, scope=SCOPE.AUTH, action=ACTION.PASSTHRU_ASSIGN,
                                                  user_object=user_object).action_values(unique=True)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Remove Support for Python 3.5
As announced we remove support for Python 3.5 in v3.8



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -104,7 +104,7 @@ def get_scripts(dir):
     author_email='cornelius@privacyidea.org',
     url='http://www.privacyidea.org',
     keywords='OTP, two factor authentication, management, security',
-    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5',
     packages=find_packages(),
     scripts=["pi-manage"] + get_scripts("tools"),
     extras_require={
@@ -150,7 +150,6 @@ def get_scripts(dir):
                  'Programming Language :: Python :: 2',
                  'Programming Language :: Python :: 2.7',
                  'Programming Language :: Python :: 3',
-                 'Programming Language :: Python :: 3.5',
                  'Programming Language :: Python :: 3.6',
                  'Programming Language :: Python :: 3.7',
                  'Programming Language :: Python :: 3.8',


Test patch:
diff --git a/tests/test_lib_token.py b/tests/test_lib_token.py
--- a/tests/test_lib_token.py
+++ b/tests/test_lib_token.py
@@ -1633,7 +1633,7 @@ def test_01_failcounter_no_increase(self):
         self.assertEqual(r, 0)
 
         # Now set the challenge to be answered and recheck:
-        Challenge.query.filter(Challenge.transaction_id == transaction_id).update({"otp_valid": 1})
+        Challenge.query.filter(Challenge.transaction_id == transaction_id).update({"otp_valid": True})
         r, r_dict = check_token_list([token1], "", user=user, options={"transaction_id": transaction_id})
         self.assertTrue(r, r_dict)
         self.assertEqual(r_dict.get("message"), "Found matching challenge", r_dict)
Repository: privacyidea/privacyidea

Issue:
Improve the logging of SSH tokens
If a user has many different SSH keys assigned on one machine for different ssh-users on this machine, this line gets logged for each SSH key, that is not used:

https://github.com/privacyidea/privacyidea/blob/745a829d89fb2824d253e27b510027316c856245/privacyidea/lib/applications/ssh.py#L84

We should change this to "debug".

On the other hand we should add a log here 
https://github.com/privacyidea/privacyidea/blob/745a829d89fb2824d253e27b510027316c856245/privacyidea/lib/applications/ssh.py#L73
like

~~~~python
log.info(u"Using SSH key {0!s} for user {1!s}".format(tokclass.token.serial, options.get("user")))
~~~~




Discussion:

Resolving patch:
diff --git a/privacyidea/lib/applications/ssh.py b/privacyidea/lib/applications/ssh.py
--- a/privacyidea/lib/applications/ssh.py
+++ b/privacyidea/lib/applications/ssh.py
@@ -70,6 +70,8 @@ def get_authentication_item(token_type,
                     tokclass = toks[0]
                     # We just return the ssh public key, so that
                     # it can be included into authorized keys.
+                    log.info(u"Using SSH key {0!s} for SSH user {1!s}".format(tokclass.token.serial,
+                                                                              options.get("user")))
                     ret["sshkey"] = tokclass.get_sshkey()
                     # We return the username if the token is assigned to a
                     # user, so that this username could be used to save
@@ -79,14 +81,13 @@ def get_authentication_item(token_type,
                         uInfo = user_object.info
                         if "username" in uInfo:
                             ret["username"] = uInfo.get("username")
-                    # ret["info"] = uInfo
                 else:
-                    log.info("The requested user %s does not match the user "
-                             "option (%s) of the SSH application." % (
+                    log.debug(u"The requested user {0!s} does not match the user "
+                              "option ({0!s}) of the SSH application.".format(
                         user_filter, options.get("user")))
         else:
-            log.info("Token %r, type %r is not supported by "
-                     "SSH application module" % (serial, token_type))
+            log.info(u"Token {0!r}, type {0!r} is not supported by "
+                     "SSH application module".format(serial, token_type))
 
         return ret
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
The `user_fields` parameter in `GET /token` is not implemented
The documentation of the `GET /token` API describes a `user_fields` parameter:
https://privacyidea.readthedocs.io/en/master/modules/api/token.html#get--token-
This parameter is currently not used:
https://github.com/privacyidea/privacyidea/blob/756116fbef9f7790138f54ddb6ff858e76249fa0/privacyidea/api/token.py#L444

We should either update the API docs or implement this feature (if this means the `userinfo` we would need to query every user for all matching tokens).



Discussion:
This is actually from January 2015. Very originial. It could be that I simply remove this functionality right from the start.
Also I am currently not sure, what to do with that.

User information could be retrieved via GET /user, so technical this is possible, only in two logical steps.
@jona-samuel recommends to remove the dead code ;-) +1

Resolving patch:
diff --git a/privacyidea/api/token.py b/privacyidea/api/token.py
--- a/privacyidea/api/token.py
+++ b/privacyidea/api/token.py
@@ -405,8 +405,6 @@ def list_api():
     :query assigned: Only return assigned (True) or not assigned (False) tokens
     :query active: Only return active (True) or inactive (False) tokens
     :query pagesize: limit the number of returned tokens
-    :query user_fields: additional user fields from the userid resolver of
-        the owner (user)
     :query outform: if set to "csv", than the token list will be given in CSV
     :query rollout_state: only list tokens with the given rollout_state
     :query infokey: only list tokens, where the infokey has the given infovalue
@@ -430,7 +428,6 @@ def list_api():
     realm = getParam(param, "tokenrealm", optional)
     userid = getParam(param, "userid", optional)
     resolver = getParam(param, "resolver", optional)
-    ufields = getParam(param, "user_fields", optional)
     output_format = getParam(param, "outform", optional)
     assigned = getParam(param, "assigned", optional)
     active = getParam(param, "active", optional)
@@ -444,10 +441,6 @@ def list_api():
         assigned = assigned.lower() == "true"
     if active:
         active = active.lower() == "true"
-    
-    user_fields = []
-    if ufields:
-        user_fields = [u.strip() for u in ufields.split(",")]
 
     # allowed_realms determines, which realms the admin would be allowed to see
     # In certain cases like for users, we do not have allowed_realms


Test patch:
Repository: privacyidea/privacyidea

Issue:
2.7 and 3.6 not supported on ubuntu-latest in github tests
Github seems to have changed "ubuntu-latest" to 22.04.
See: https://github.com/privacyidea/privacyidea/blob/master/.github/workflows/python.yml#L19

Thus our tests for 2.7 adn 3.6 fail, since these versions are not available for 22.04.
So we either need to revert to 20.04 or we need to run 3.6/2.7 explicitly on 20.04 and all other on 22.04.



Discussion:
We can just switch to 20.04 completely until we remove support for Python 2.7 and 3.6.
Python 3.11 and 3.12 will also run on 20.04

Resolving patch:
diff --git a/privacyidea/lib/policy.py b/privacyidea/lib/policy.py
--- a/privacyidea/lib/policy.py
+++ b/privacyidea/lib/policy.py
@@ -611,7 +611,7 @@ def list_policies(self, name=None, scope=None, realm=None, active=None,
                         if value_found and not value_excluded:
                             new_policies.append(policy)
                 reduced_policies = new_policies
-                log.debug("Policies after matching {1!s}={2!s}: {0!s}".format(
+                log.debug(u"Policies after matching {1!s}={2!s}: {0!s}".format(
                     reduced_policies, searchkey, searchvalue))
 
         # We need to act individually on the resolver key word


Test patch:
Repository: privacyidea/privacyidea

Issue:
logging documentation
Check the log configuration.
The yaml config seems to be wrong. 
Is the indentation of `root:` correct?

https://privacyidea.readthedocs.io/en/master/installation/system/logging.html#debug-log


Discussion:

Resolving patch:
diff --git a/privacyidea/app.py b/privacyidea/app.py
--- a/privacyidea/app.py
+++ b/privacyidea/app.py
@@ -228,7 +228,8 @@ def get_locale():
         with app.app_context():
             init_hsm()
 
-    logging.debug(u"Reading application from the static folder {0!s} and "
-                  u"the template folder {1!s}".format(app.static_folder, app.template_folder))
+    logging.getLogger(__name__).debug(u"Reading application from the static "
+                                      u"folder {0!s} and the template folder "
+                                      u"{1!s}".format(app.static_folder, app.template_folder))
 
     return app


Test patch:
Repository: privacyidea/privacyidea

Issue:
Wrong QR-Code sent during enroll_via_validate
If `enroll_via_multichallenge` is active, the QR-code sent after the first request only contains the token seed, not the complete `otpauth://` uri in case of enrolling HOTP/TOTP token.
When calling [`get_init_detail()`](https://github.com/privacyidea/privacyidea/blob/b1e6c1d073190d68e5171bd5870cec99edda1ab8/privacyidea/lib/tokens/hotptoken.py#L835) and passing the user object, the complete QR-code is available in the dictionary as `googleurl` and `oathurl` (See [`POST /token/init`](https://privacyidea.readthedocs.io/en/latest/modules/api/token.html?highlight=google#post--token-init)).



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/hotptoken.py b/privacyidea/lib/tokens/hotptoken.py
--- a/privacyidea/lib/tokens/hotptoken.py
+++ b/privacyidea/lib/tokens/hotptoken.py
@@ -832,10 +832,10 @@ def enroll_via_validate(cls, g, content, user_obj):
         # Create a challenge!
         c = token_obj.create_challenge()
         # get details of token
-        init_details = token_obj.get_init_detail()
+        init_details = token_obj.get_init_detail(user=user_obj)
         detail["transaction_ids"] = [c[2]]
         chal = {"transaction_id": c[2],
-                "image": init_details.get("otpkey").get("img"),
+                "image": init_details.get("googleurl").get("img"),
                 "client_mode": CLIENTMODE.INTERACTIVE,
                 "serial": token_obj.token.serial,
                 "type": token_obj.type,


Test patch:
diff --git a/tests/test_api_validate.py b/tests/test_api_validate.py
--- a/tests/test_api_validate.py
+++ b/tests/test_api_validate.py
@@ -5377,10 +5377,11 @@ def test_01_enroll_HOTP(self):
             transaction_id = detail.get("transaction_id")
             self.assertTrue("Please scan the QR code!" in detail.get("message"), detail.get("message"))
             # Get image and client_mode
-            self.assertEqual(CLIENTMODE.INTERACTIVE, detail.get("client_mode"))
+            self.assertEqual(CLIENTMODE.INTERACTIVE, detail.get("client_mode"), detail)
             # Check, that multi_challenge is also contained.
-            self.assertEqual(CLIENTMODE.INTERACTIVE, detail.get("multi_challenge")[0].get("client_mode"))
-            self.assertIn("image", detail)
+            chal = detail.get("multi_challenge")[0]
+            self.assertEqual(CLIENTMODE.INTERACTIVE, chal.get("client_mode"), detail)
+            self.assertIn("image", detail, detail)
             serial = detail.get("serial")
 
         # 3. scan the qrcode / Get the OTP value
Repository: privacyidea/privacyidea

Issue:
WebHook handler not available.
The WebHook Handler is not available in the UI.


Discussion:

Resolving patch:
diff --git a/privacyidea/api/event.py b/privacyidea/api/event.py
--- a/privacyidea/api/event.py
+++ b/privacyidea/api/event.py
@@ -67,7 +67,7 @@ def get_eventhandling(eventid=None):
     elif eventid == "handlermodules":
         # TODO: We need to provide a dynamic list of event handlers
         res = ["UserNotification", "Token", "Federation", "Script", "Counter",
-               "RequestMangler", "ResponseMangler", "Logging", "CustomUserAttributes"]
+               "RequestMangler", "ResponseMangler", "Logging", "CustomUserAttributes", "WebHook"]
     else:
         res = g.event_config.get_event(eventid)
     g.audit_object.log({"success": True})
diff --git a/privacyidea/lib/event.py b/privacyidea/lib/event.py
--- a/privacyidea/lib/event.py
+++ b/privacyidea/lib/event.py
@@ -164,6 +164,7 @@ def get_handler_object(handlername):
     from privacyidea.lib.eventhandler.responsemangler import ResponseManglerEventHandler
     from privacyidea.lib.eventhandler.logginghandler import LoggingEventHandler
     from privacyidea.lib.eventhandler.customuserattributeshandler import CustomUserAttributesHandler
+    from privacyidea.lib.eventhandler.webhookeventhandler import WebHookHandler
     h_obj = None
     if handlername == "UserNotification":
         h_obj = UserNotificationEventHandler()
@@ -183,6 +184,8 @@ def get_handler_object(handlername):
         h_obj = LoggingEventHandler()
     elif handlername == "CustomUserAttributes":
         h_obj = CustomUserAttributesHandler()
+    elif handlername == "WebHook":
+        h_obj = WebHookHandler()
     return h_obj
 
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
certifcate_ca_connector Policy does not work
It looks like the policy "certificate_ca_connector" policy does not overwrite the "ca" parameter sent in the `/token/init` request.

The problem is, that the policy actions "ca_connector" are fetched here:

https://github.com/privacyidea/privacyidea/blob/66d31d5080f75060463c266667261effa9e106b2/privacyidea/api/lib/prepolicy.py#L622-L625



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/caconnector.py b/privacyidea/lib/caconnector.py
--- a/privacyidea/lib/caconnector.py
+++ b/privacyidea/lib/caconnector.py
@@ -292,6 +292,9 @@ def get_caconnector_object(connector_name):
                     connector_config[conf.Key] = value
                 c_obj.set_config(connector_config)
 
+    if not c_obj:
+        log.warning("A CA connector with the name {0!s} could not be found!".format(connector_name))
+
     return c_obj
 
 
diff --git a/privacyidea/lib/tokens/certificatetoken.py b/privacyidea/lib/tokens/certificatetoken.py
--- a/privacyidea/lib/tokens/certificatetoken.py
+++ b/privacyidea/lib/tokens/certificatetoken.py
@@ -67,7 +67,7 @@ class ACTION(BASE_ACTION):
     __doc__ = """This is the list of special certificate actions."""
     TRUSTED_CA_PATH = "certificate_trusted_Attestation_CA_path"
     REQUIRE_ATTESTATION = "certificate_require_attestation"
-    CA_CONNECTOR = "ca_connector"
+    CA_CONNECTOR = "certificate_ca_connector"
     CERTIFICATE_TEMPLATE = "certificate_template"
     CERTIFICATE_REQUEST_SUBJECT_COMPONENT = "certificate_request_subject_component"
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
LDAPResolver: NTLM fails with "a bytes-like object is required, not a str".
The LDAP Resolver in 3.8 returns a 

    TypeError("a bytes-like object is required, not 'str'")

when using an NTLM connection.

It looks like it originates from here, where the password is converted:

https://github.com/privacyidea/privacyidea/blob/f707149f42805ad34df56beb1e02168db6577758/privacyidea/lib/resolvers/LDAPIdResolver.py#L1325

Also see:
https://community.privacyidea.org/t/error-in-ldap-resolver-after-update-to-3-8/2785


Discussion:
Well, who would have thought that [`to_utf8()`](https://github.com/privacyidea/privacyidea/blob/58c6a856f133562c52522a2441de09207c78a219/privacyidea/lib/utils/__init__.py#L135) returns bytes...

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -78,12 +78,12 @@
 from passlib.hash import ldap_salted_sha1
 import hashlib
 import binascii
-from privacyidea.lib.utils import is_true, to_bytes
 from privacyidea.lib.framework import get_app_local_store, get_app_config_value
 import datetime
 
 from privacyidea.lib import _
-from privacyidea.lib.utils import to_utf8, to_unicode, convert_column_to_unicode
+from privacyidea.lib.utils import (is_true, to_bytes, to_unicode,
+                                   convert_column_to_unicode)
 from privacyidea.lib.error import privacyIDEAError
 import uuid
 from ldap3.utils.conv import escape_bytes
@@ -1316,18 +1316,18 @@ def create_connection(authtype=None, server=None, user=None,
             conn_opts.update({'authentication': ldap3.ANONYMOUS})
         elif authtype == AUTHTYPE.SIMPLE:
             # SIMPLE works with passwords as UTF8 and unicode
-            password = to_utf8(password)
+            password = to_unicode(password)
             conn_opts.update({'user': user,
                               'password': password,
                               'authentication': ldap3.SIMPLE})
         elif authtype == AUTHTYPE.NTLM:  # pragma: no cover
             # NTLM requires the password to be unicode
-            password = to_utf8(password)
+            password = to_unicode(password)
             conn_opts.update({'user': user,
                               'password': password,
                               'authentication': ldap3.NTLM})
         elif authtype == AUTHTYPE.SASL_DIGEST_MD5:  # pragma: no cover
-            password = to_utf8(password)
+            password = to_unicode(password)
             sasl_credentials = (str(user), str(password))
             conn_opts.update({'sasl_mechanism': ldap3.DIGEST_MD5,
                               'sasl_credentials': sasl_credentials,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Node from policy can not be removed
A policy has a condition with a node.
If I remove all nodes (select none) and save the policy the node is still contained in the policy.

Reopen the policy and check: The node is still there.


Discussion:

Resolving patch:
diff --git a/privacyidea/api/policy.py b/privacyidea/api/policy.py
--- a/privacyidea/api/policy.py
+++ b/privacyidea/api/policy.py
@@ -120,6 +120,7 @@ def set_policy_api(name=None):
     :jsonparam user: The policy is valid for these users.
         string with wild cards or list of strings
     :jsonparam time: on which time does this policy hold
+    :jsonparam pinode: The privacyIDEA node (or list of nodes) for which this policy is valid
     :jsonparam client: for which requesting client this should be
     :jsontype client: IP address with subnet
     :jsonparam active: bool, whether this policy is active or not


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add italian translation



Discussion:

Resolving patch:
diff --git a/privacyidea/webui/login.py b/privacyidea/webui/login.py
--- a/privacyidea/webui/login.py
+++ b/privacyidea/webui/login.py
@@ -45,7 +45,7 @@
 from privacyidea.lib.queue import has_job_queue
 
 DEFAULT_THEME = "/static/contrib/css/bootstrap-theme.css"
-DEFAULT_LANGUAGE_LIST = ['en', 'de', 'nl', 'zh_Hant', 'fr', 'es', 'tr']
+DEFAULT_LANGUAGE_LIST = ['en', 'de', 'nl', 'zh_Hant', 'fr', 'es', 'tr', 'cs', 'it']
 
 login_blueprint = Blueprint('login_blueprint', __name__)
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add event-handler status to audit-log
Currently when an event handler returns, the `success` entry in the audit log is always `true`:
https://github.com/privacyidea/privacyidea/blob/cab88df939e0027ad6f3e34f6df54773ee5fb048/privacyidea/lib/event.py#L139
and
https://github.com/privacyidea/privacyidea/blob/cab88df939e0027ad6f3e34f6df54773ee5fb048/privacyidea/lib/event.py#L97
The event handler code already returns a status boolean which we could use as the `success` value of the audit log entry:
https://github.com/privacyidea/privacyidea/blob/8cfd93fdd90abafe3b864f9aa9a3f3a182f1da21/privacyidea/lib/eventhandler/base.py#L618


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/event.py b/privacyidea/lib/event.py
--- a/privacyidea/lib/event.py
+++ b/privacyidea/lib/event.py
@@ -91,10 +91,10 @@ def event_wrapper(*args, **kwds):
                     event_audit_data["info"] = e_handler_def.get("name")
                     event_audit.log(event_audit_data)
 
-                    event_handler.do(e_handler_def.get("action"),
-                                     options=options)
+                    result = event_handler.do(e_handler_def.get("action"),
+                                               options=options)
                     # set audit object to success
-                    event_audit.log({"success": True})
+                    event_audit.log({"success": result})
                     event_audit.finalize_log()
 
             f_result = func(*args, **kwds)
@@ -131,12 +131,12 @@ def event_wrapper(*args, **kwds):
                     event_audit_data["info"] = e_handler_def.get("name")
                     event_audit.log(event_audit_data)
 
-                    event_handler.do(e_handler_def.get("action"),
-                                     options=options)
+                    result = event_handler.do(e_handler_def.get("action"),
+                                               options=options)
                     # In case the handler has modified the response
                     f_result = options.get("response")
                     # set audit object to success
-                    event_audit.log({"success": True})
+                    event_audit.log({"success": result})
                     event_audit.finalize_log()
 
             return f_result


Test patch: