internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
Email token without user assignment does not work
In case you have an email token without a user assignment you will run into an error like "None Type object has not attribute 'info'".

This is caused by this code https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/tokens/emailtoken.py#L437
where we assume, that the token has a tokenowner to prepare possible tags.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/emailtoken.py b/privacyidea/lib/tokens/emailtoken.py
--- a/privacyidea/lib/tokens/emailtoken.py
+++ b/privacyidea/lib/tokens/emailtoken.py
@@ -434,8 +434,8 @@ def _compose_email(self, message="<otp>", subject="Your OTP", mimetype="plain"):
         tags = create_tag_dict(serial=serial,
                                tokenowner=self.user,
                                tokentype=self.get_tokentype(),
-                               recipient={"givenname": self.user.info.get("givenname"),
-                                          "surname": self.user.info.get("surname")},
+                               recipient={"givenname": self.user.info.get("givenname") if self.user else "",
+                                          "surname": self.user.info.get("surname") if self.user else ""},
                                escape_html=mimetype.lower() == "html")
 
         message = message.format(otp=otp, **tags)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Uncaught exception with Yubikey in "yubikey" mode
It looks like in yubikey mode we do not handle the exception correctly if the user input is wrong. 
Then the modhex decode will fail:

~~~~
  File "/opt/privacyidea/lib/python2.7/site-packages/privacyidea/lib/tokens/yubikeytoken.py", line 296, in check_otp
    log.debug("prefix: {0!r}".format(binascii.hexlify(modhex_decode(yubi_prefix))))
  File "/opt/privacyidea/lib/python2.7/site-packages/privacyidea/lib/utils/__init__.py", line 333, in modhex_decode
    return binascii.unhexlify(''.join([mod2HexDict[c] for c in m]))
TypeError: Odd-length string
~~~~

We need to check this, write a test and catch the exception.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/yubikeytoken.py b/privacyidea/lib/tokens/yubikeytoken.py
--- a/privacyidea/lib/tokens/yubikeytoken.py
+++ b/privacyidea/lib/tokens/yubikeytoken.py
@@ -298,7 +298,7 @@ def check_otp(self, anOtpVal, counter=None, window=None, options=None):
 
         uid = msg_hex[0:12]
         log.debug("uid: {0!r}".format(uid))
-        log.debug("prefix: {0!r}".format(binascii.hexlify(modhex_decode(yubi_prefix))))
+        log.debug("prefix: {0!r}".format(yubi_prefix))
         # usage_counter can go from 1 – 0x7fff
         usage_counter = msg_hex[12:16]
         timestamp = msg_hex[16:22]


Test patch:
diff --git a/tests/test_lib_utils.py b/tests/test_lib_utils.py
--- a/tests/test_lib_utils.py
+++ b/tests/test_lib_utils.py
@@ -822,6 +822,8 @@ def test_28_yubikey_utils(self):
         self.assertEqual(modhex_decode('cbdefghijklnrtuv'),
                           binascii.unhexlify('0123456789abcdef'))
         self.assertEqual(modhex_decode('fjhbhrhrhv'), b'Hallo')
+        # fail with invalid modhex
+        self.assertRaises((binascii.Error, TypeError), modhex_decode, 'nlltvcc')
 
         # now test the crc function
         self.assertEqual(checksum(b'\x01\x02\x03\x04'), 0xc66e)
Repository: privacyidea/privacyidea

Issue:
privacyIDEA login with push-token doesn't work
Since 04285e30 the login to privacyIDEA with push token (and possibly other out-of-band tokens) does not work anymore.
First it fails here:
https://github.com/privacyidea/privacyidea/blob/8bca25e053bf763737ff578a3d8500a32265731d/privacyidea/static/components/login/controllers/loginControllers.js#L254
because the `attributes` variable does not exist anymore.
Second it won't poll the server for answered challenges since the `poll` variable isn't set:
https://github.com/privacyidea/privacyidea/blob/8bca25e053bf763737ff578a3d8500a32265731d/privacyidea/static/components/login/controllers/loginControllers.js#L272


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/pushtoken.py b/privacyidea/lib/tokens/pushtoken.py
--- a/privacyidea/lib/tokens/pushtoken.py
+++ b/privacyidea/lib/tokens/pushtoken.py
@@ -821,7 +821,6 @@ def create_challenge(self, transactionid=None, options=None):
                                                  ACTION.CHALLENGETEXT,
                                                  options) or DEFAULT_CHALLENGE_TEXT
 
-        reply_dict = {}
         data = None
         # Initially we assume there is no error from Firebase
         res = True
@@ -879,6 +878,7 @@ def create_challenge(self, transactionid=None, options=None):
             if is_true(options.get("exception")):
                 raise ValidateError("The token has no tokeninfo. Can not send via Firebase service.")
 
+        reply_dict = {"attributes": {"hideResponseInput": self.client_mode != CLIENTMODE.INTERACTIVE}}
         return True, message, transactionid, reply_dict
 
     @check_token_locked


Test patch:
Repository: privacyidea/privacyidea

Issue:
csv_generator() of SQLAudit does not sort entries
The [`csv_generator()`](https://github.com/privacyidea/privacyidea/blob/059e6b3ef10518595cfb5dfbf3918cc205099d79/privacyidea/lib/auditmodules/sqlaudit.py#L352) function does not sort the audit entries. We should at least sort by timestamp/entry id by default.
Also each entry itself is unsorted due to the fact that python dictionaries are unsorted by default.
We should check if we just output a header with the appropriate column name or sort each line according to the audit view output (or both...).


Discussion:
Ordering of the audit entry dictionary is fixed in f55b1bd2
@plettich I've now looked through several logs and everything is sorted perfectly by date and entry ID. Can you possibly test again yourself whether the problem still exists or it may have already been fixed with your change in f55b1bd2?
The sorting of the columns is fixed.
Probably the rows are still unsorted.

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -419,7 +419,7 @@ def csv_generator(self, param=None, user=None, timelimit=None):
         """
         filter_condition = self._create_filter(param,
                                                timelimit=timelimit)
-        logentries = self.session.query(LogEntry).filter(filter_condition).all()
+        logentries = self.session.query(LogEntry).filter(filter_condition).order_by(LogEntry.date).all()
 
         for le in logentries:
             audit_dict = self.audit_entry_to_dict(le)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Return tokenowner as default user for offline-otp
The TokenApplication `offline` claims `user` to be an optional parameter.
But it looks like as if, if the user is not given, that offline otp fails to be usable.

We should return the token owner as the user, if the option `user` is not set.


Discussion:
We should probably remove the option "user" completely and pass back the tokenowner as user name to the credential provider.

Resolving patch:
diff --git a/privacyidea/lib/applications/offline.py b/privacyidea/lib/applications/offline.py
--- a/privacyidea/lib/applications/offline.py
+++ b/privacyidea/lib/applications/offline.py
@@ -171,7 +171,7 @@ def get_authentication_item(token_type,
                 if user_object:
                     uInfo = user_object.info
                     if "username" in uInfo:
-                        ret["username"] = uInfo.get("username")
+                        ret["user"] = ret["username"] = uInfo.get("username")
 
         else:
             log.info("Token %r, type %r is not supported by "
@@ -185,4 +185,4 @@ def get_options():
         returns a dictionary with a list of required and optional options
         """
         return {'required': [],
-                'optional': ['user', 'count', 'rounds']}
+                'optional': ['count', 'rounds']}


Test patch:
diff --git a/tests/test_api_machines.py b/tests/test_api_machines.py
--- a/tests/test_api_machines.py
+++ b/tests/test_api_machines.py
@@ -447,8 +447,7 @@ def test_12_auth_items_offline(self):
 
         # Attach the token to the machine "gandalf" with the application offline
         r = attach_token(hostname="gandalf", serial=self.serial4,
-                         application="offline", options={"user": "cornelius",
-                                                         "count": 17})
+                         application="offline", options={"count": 17})
 
         self.assertEqual(r.machine_id, "192.168.0.1")
 
diff --git a/tests/test_lib_applications.py b/tests/test_lib_applications.py
--- a/tests/test_lib_applications.py
+++ b/tests/test_lib_applications.py
@@ -105,7 +105,7 @@ def test_01_get_options(self):
         # Can run as class
         options = OfflineApplication.get_options()
         self.assertEqual(options["required"], [])
-        self.assertEqual(options["optional"], ['user', 'count', 'rounds'])
+        self.assertEqual(options["optional"], ['count', 'rounds'])
 
     def test_02_get_auth_item(self):
         serial = "OATH1"
Repository: privacyidea/privacyidea

Issue:
Bad error handling in /ttype/<type> endpoint
We observed a bad error handling when requesting the `/ttype/<type>` endpoint.

In specific, we faced the following error:
```
[ERROR][privacyidea.app:1892] Exception on /ttype/push"}. [GET]
...
AttributeError: 'NoneType' object has no attribute 'api_endpoint'
```
Actually, we could fix the problem but it would be nice to fix this upstream right away.

### Top-level intent

Access the `/ttype/<type>` endpoint.

### Steps to reproduce

1. Query `/ttype/test` endpoint
2. There will be a NoneType error in the logs.

### Expected outcome

Proper error handling

### Actual outcome

NoneType exception.

### Configuration

* **privacyIDEA version**: v3.6.3
* **Installation method**: (from Ubuntu packages, github, PyPI, ...)
* **Python version**: 3
* **Operating system**: linux
* **Webserver**: apache


### Log file

**Set PI_LOGLEVEL = logging.DEBUG in pi.cfg and take a look at the privacyidea.log!**
**If appropriate, attach the log file or paste relevant portions.**



Discussion:
@cornelinux 
I found a bug and propose a possible fix for it.

Resolving patch:
diff --git a/privacyidea/api/ttype.py b/privacyidea/api/ttype.py
--- a/privacyidea/api/ttype.py
+++ b/privacyidea/api/ttype.py
@@ -38,6 +38,7 @@
 from flask import g, jsonify, current_app
 import logging
 from privacyidea.api.lib.utils import get_all_params
+from privacyidea.lib.error import ParameterError
 from privacyidea.lib.policy import PolicyClass
 from privacyidea.lib.audit import getAudit
 from privacyidea.lib.config import (get_token_class, get_from_config,
@@ -90,6 +91,9 @@ def token(ttype=None):
     :return: Token Type dependent
     """
     tokenc = get_token_class(ttype)
+    if tokenc is None:
+        log.error(u"Invalid tokentype provided. ttype: {}".format(ttype.lower()))
+        raise ParameterError(u"Invalid tokentype provided. ttype: {}".format(ttype.lower()))
     res = tokenc.api_endpoint(request, g)
     serial = getParam(request.all_data, "serial")
     user = get_user_from_param(request.all_data)


Test patch:
diff --git a/tests/test_api_ttype.py b/tests/test_api_ttype.py
--- a/tests/test_api_ttype.py
+++ b/tests/test_api_ttype.py
@@ -85,6 +85,17 @@ def test_02_u2f(self):
             self.assertTrue(len(auditdata) > 0)
             self.assertEqual(auditdata[0].get("token_type"), "u2f")
 
+    def test_03_wrong(self):
+        # Test the ttype endpoint for wrong ttype, here /ttype/wrong
+        init_token({"serial": "TIQR1",
+                    "type": "tiqr"}, User("cornelius", self.realm1))
+        with self.app.test_request_context('/ttype/wrong',
+                                           method='POST',
+                                           data={"action": "metadata",
+                                                 "serial": "TIQR1",
+                                                 "session": "12345"}):
+            res = self.app.full_dispatch_request()
+            self.assertTrue(res.status_code == 400, res.status_code)
 
 class TtypePushAPITestCase(MyApiTestCase):
     """
Repository: privacyidea/privacyidea

Issue:
Change data message to notification message
Currently the push notification is sent as data message.
Check, if it is sensible to send the message as notification type.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/smsprovider/FirebaseProvider.py b/privacyidea/lib/smsprovider/FirebaseProvider.py
--- a/privacyidea/lib/smsprovider/FirebaseProvider.py
+++ b/privacyidea/lib/smsprovider/FirebaseProvider.py
@@ -124,6 +124,10 @@ def submit_message(self, firebase_token, data):
             "message": {
                         "data": data,
                         "token": firebase_token,
+                        "notification": {
+                            "title": data.get("title"),
+                            "body": data.get("question")
+                        },
                         "android": {
                                     "priority": "HIGH",
                                     "ttl": "120s",


Test patch:
Repository: privacyidea/privacyidea

Issue:
cannot register Apple devices for WebAuthn (Touch ID or Face ID)
### Top-level intent

What did you try to achieve?

Register iPhone as a WebAuthn token with Touch ID.

### Steps to reproduce

1. Add WebAuthn token
2. use Touch ID

### Expected outcome

Token should be registered.

### Actual outcome

Server error happens, because apple attestation format is not listed in privacyIDEA source code.

### Configuration

* **privacyIDEA version**: 3.7
* **Installation method**: pip
* **Python version**: 3.9
* **Operating system**: does not matter
* **Webserver**: does not matter
* **Token database**: does not matter

webauthn attestation policy set to "direct" (or "indirect")

### Log file

```
Exception on /token/init [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/privacyidea/lib/tokens/webauthn.py", line 1303, in verify
    raise RegistrationRejectedException('Unable to verify attestation statement format.')
privacyidea.lib.tokens.webauthn.RegistrationRejectedException: Unable to verify attestation statement format.
```


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/webauthn.py b/privacyidea/lib/tokens/webauthn.py
--- a/privacyidea/lib/tokens/webauthn.py
+++ b/privacyidea/lib/tokens/webauthn.py
@@ -134,6 +134,7 @@ class ATTESTATION_FORMAT(object):
     TPM = 'tpm'
     ANDROID_KEY = 'android-key'
     ANDROID_SAFETYNET = 'android-safetynet'
+    APPLE = 'apple'
     FIDO_U2F = 'fido-u2f'
     NONE = 'none'
 
@@ -146,12 +147,12 @@ class ATTESTATION_FORMAT(object):
 )
 
 
-# Still waiting for Apple Anonymous Attestation.
 REGISTERED_ATTESTATION_FORMATS = (
     ATTESTATION_FORMAT.PACKED,
     ATTESTATION_FORMAT.TPM,
     ATTESTATION_FORMAT.ANDROID_KEY,
     ATTESTATION_FORMAT.ANDROID_SAFETYNET,
+    ATTESTATION_FORMAT.APPLE,
     ATTESTATION_FORMAT.FIDO_U2F,
     ATTESTATION_FORMAT.NONE
 )


Test patch:
Repository: privacyidea/privacyidea

Issue:
Change policy description and documentation for "appimageurl".
With one of the next versions of the app, it will also be possible to display an image next to the otp using the [privacyidea authenticator](https://github.com/privacyidea/pi-authenticator/issues/249).

In the course of this, the [documentation](https://github.com/privacyidea/privacyidea/blob/master/doc/policies/enrollment.rst#appimageurl) and also the policy [description]( https://github.com/privacyidea/privacyidea/blob/e55a1814cff5c306b0fa8b69ae0b3cd2dde67d0c/privacyidea/lib/policy.py#L2140) should be adjusted.

Supported file formats: png, jpg and gif.

This issue serves as a reminder!


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/policy.py b/privacyidea/lib/policy.py
--- a/privacyidea/lib/policy.py
+++ b/privacyidea/lib/policy.py
@@ -2139,8 +2139,8 @@ def get_static_policy_definitions(scope=None):
             },
             ACTION.APPIMAGEURL: {
                 'type': 'str',
-                'desc': _("This is the URL to the token image for smartphone apps "
-                          "like FreeOTP."),
+                'desc': _("This is the URL to the token image for the privacyIDEA Authenticator "
+                          "and some other apps like FreeOTP (supported file formats: PNG, JPG and GIF)."),
                 'group': GROUP.TOKEN
             },
             ACTION.AUTOASSIGN: {


Test patch:
Repository: privacyidea/privacyidea

Issue:
Update matrix tests to use Python 3.10



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -153,7 +153,8 @@ def get_scripts(dir):
                  'Programming Language :: Python :: 3.6',
                  'Programming Language :: Python :: 3.7',
                  'Programming Language :: Python :: 3.8',
-                 'Programming Language :: Python :: 3.9'
+                 'Programming Language :: Python :: 3.9',
+                 'Programming Language :: Python :: 3.10'
                  ],
     zip_safe=False,
     long_description=get_file_contents('README.rst')


Test patch:
diff --git a/tests/ldap3mock.py b/tests/ldap3mock.py
--- a/tests/ldap3mock.py
+++ b/tests/ldap3mock.py
@@ -52,7 +52,13 @@
 
 from .smtpmock import get_wrapped
 
-from collections import namedtuple, Sequence, Sized
+from collections import namedtuple
+
+try:
+    from collections import Sequence, Sized
+except ImportError:
+    from collections.abc import Sequence, Sized
+
 from privacyidea.lib.utils import to_bytes, to_unicode
 
 DIRECTORY = "tests/testdata/tmp_directory"
diff --git a/tests/radiusmock.py b/tests/radiusmock.py
--- a/tests/radiusmock.py
+++ b/tests/radiusmock.py
@@ -27,7 +27,13 @@
     absolute_import, print_function, division, unicode_literals
 )
 
-from collections import namedtuple, Sequence, Sized
+from collections import namedtuple
+
+try:
+    from collections import Sequence, Sized
+except ImportError:
+    from collections.abc import Sequence, Sized
+
 from pyrad import packet
 from pyrad.client import Timeout
 from pyrad.packet import AccessReject, AccessAccept, AccessChallenge
diff --git a/tests/redismock.py b/tests/redismock.py
--- a/tests/redismock.py
+++ b/tests/redismock.py
@@ -23,7 +23,12 @@
     absolute_import, print_function, division, unicode_literals
 )
 
-from collections import namedtuple, Sequence, Sized
+from collections import namedtuple
+
+try:
+    from collections import Sequence, Sized
+except ImportError:
+    from collections.abc import Sequence, Sized
 
 from .smtpmock import get_wrapped
 
diff --git a/tests/smppmock.py b/tests/smppmock.py
--- a/tests/smppmock.py
+++ b/tests/smppmock.py
@@ -24,7 +24,13 @@
     absolute_import, print_function, division, unicode_literals
 )
 
-from collections import namedtuple, Sequence, Sized
+from collections import namedtuple
+
+try:
+    from collections import Sequence, Sized
+except ImportError:
+    from collections.abc import Sequence, Sized
+
 from smpplib.exceptions import ConnectionError
 
 from .smtpmock import get_wrapped
diff --git a/tests/smtpmock.py b/tests/smtpmock.py
--- a/tests/smtpmock.py
+++ b/tests/smtpmock.py
@@ -35,7 +35,12 @@
 except ImportError:
     from inspect import formatargspec, getargspec
 
-from collections import namedtuple, Sequence, Sized
+try:
+    from collections import Sequence, Sized
+except ImportError:
+    from collections.abc import Sequence, Sized
+
+from collections import namedtuple
 from functools import update_wrapper
 from smtplib import SMTPException
Repository: privacyidea/privacyidea

Issue:
Fix recovery link in email
The email that is sent when resetting a password, contains a faulty link.

Currentlly `https://piserver/#/reset/...` is sent. It has to be `https://piserver/#!/reset/..`.

This happens here:

https://github.com/privacyidea/privacyidea/blob/35f0963b1bbacfbbad128c12ba0a0953fc17a339/privacyidea/lib/passwordreset.py#L67

Also see https://community.privacyidea.org/t/q-password-reset-what-policies-needed/2493/4


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/passwordreset.py b/privacyidea/lib/passwordreset.py
--- a/privacyidea/lib/passwordreset.py
+++ b/privacyidea/lib/passwordreset.py
@@ -1,5 +1,7 @@
 # -*- coding: utf-8 -*-
 #
+#  2022-06-06 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#             Fix recovery link
 #  2015-01-06 Cornelius Kölbel <cornelius@privacyidea.org>
 #             The Password reset functions
 #
@@ -64,7 +66,7 @@ def create_recoverycode(user, email=None, expiration_seconds=3600,
     :return: bool
     """
     base_url = base_url.strip("recover")
-    base_url += "#"
+    base_url += "#!"
     recoverycode = recoverycode or generate_password(size=24)
     hash_code = hash_with_pepper(recoverycode)
     # send this recoverycode


Test patch:
Repository: privacyidea/privacyidea

Issue:
Make user available in user_delete
The event handler has no access on the user information in the event of `DELETE /user/`.
This is due to the fact, that the parameters (username, resolver) of this blueprint 
https://github.com/privacyidea/privacyidea/blob/0f085fd2ed59cd4567989c9bbe55ea01c29ced3a/privacyidea/api/user.py#L221
are not used to create the `request.User` object.

We could change the names of the parameters (but this would create a user object in the default realm!) or we could add the new logic to `before_after`.

We should add this to a 3.7 patch release.


Discussion:

Resolving patch:
diff --git a/privacyidea/api/before_after.py b/privacyidea/api/before_after.py
--- a/privacyidea/api/before_after.py
+++ b/privacyidea/api/before_after.py
@@ -102,13 +102,24 @@ def teardown_request(exc):
 
 @token_blueprint.before_request
 @audit_blueprint.before_request
-@user_blueprint.before_request
 @system_blueprint.before_request
 @user_required
 def before_user_request():
     before_request()
 
 
+@user_blueprint.before_request
+@user_required
+def before_userendpoint_request():
+    before_request()
+    # DEL /user/ has no realm parameter and thus we need to create the user object this way.
+    if not request.User and request.method == "DELETE":
+        resolvername = getParam(request.all_data, "resolvername")
+        username = getParam(request.all_data, "username")
+        if resolvername and username:
+            request.User = User(login=username, resolver=resolvername)
+
+
 @resolver_blueprint.before_request
 @machineresolver_blueprint.before_request
 @machine_blueprint.before_request
diff --git a/privacyidea/api/user.py b/privacyidea/api/user.py
--- a/privacyidea/api/user.py
+++ b/privacyidea/api/user.py
@@ -239,7 +239,7 @@ def delete_user(resolvername=None, username=None):
        Accept: application/json
 
     """
-    user_obj = User(login=username, resolver=resolvername)
+    user_obj = request.User
     res = user_obj.delete()
     g.audit_object.log({"success": res,
                         "info": u"{0!s}".format(user_obj)})


Test patch: