internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
Use information, which pyhsical token was used with a remote token
**Is your feature request related to a problem? Please describe.**

With a remote token it would be great to know, what actually was the physical token that led to a successful authentication.
This can help with rollout scenarios and assigning those physical tokens.

This is interesting if the remote token object points to a user on the remote system. The user can have several tokens. But it would be interesting to know, which token was actually used for authentication.

**Describe the solution you'd like**

The remote tokentype could store the information, which physical token was used for authentication. Returning it in the response does not work out well.
It can be stored in the tokeninfo.




Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/remotetoken.py b/privacyidea/lib/tokens/remotetoken.py
--- a/privacyidea/lib/tokens/remotetoken.py
+++ b/privacyidea/lib/tokens/remotetoken.py
@@ -282,6 +282,8 @@ def check_otp(self, otpval, counter=None, window=None, options=None):
                 result = response.get("result")
                 if result.get("value"):
                     otp_count = 1
+                    # Add the serial of the used remote token in the tokeninfo parameters
+                    self.add_tokeninfo("last_matching_remote_serial", response.get("detail", {}).get("serial"))
 
         except Exception as exx:  # pragma: no cover
             log.error("Error getting response from "


Test patch:
diff --git a/tests/test_lib_tokens_remote.py b/tests/test_lib_tokens_remote.py
--- a/tests/test_lib_tokens_remote.py
+++ b/tests/test_lib_tokens_remote.py
@@ -110,6 +110,8 @@ def test_04_do_request_success(self):
         otpcount = token.check_otp("123456")
         self.assertTrue(otpcount > 0, otpcount)
 
+        remote_serial = token.get_tokeninfo("last_matching_remote_serial")
+        self.assertEqual("PISP0000AB00", remote_serial)
 
     @responses.activate
     def test_05_do_request_fail(self):
Repository: privacyidea/privacyidea

Issue:
Allow SQL Resolver to have no password mapping
The SQL resolver alwas requires a password mapping to be defined.
In certain cases it is not necessary to have a password for a user.
But the password is, when adding a user in a sqlresolver, a required field.

It should be possible to define sqlresolvers without a password field.

What should happen, if the policy "otppin=userstore" is defined. The password from the user store can not be verified, so I think in this case authentication should fail.
Allow SQL Resolver to have no password mapping
The SQL resolver alwas requires a password mapping to be defined.
In certain cases it is not necessary to have a password for a user.
But the password is, when adding a user in a sqlresolver, a required field.

It should be possible to define sqlresolvers without a password field.

What should happen, if the policy "otppin=userstore" is defined. The password from the user store can not be verified, so I think in this case authentication should fail.


Discussion:
The problem arises, when the SQL resolver does not contain a password mapping.
But the problem occurs here: https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/user.py#L179

When there is no password mapping, the "attributes" do not contain a password key, and the UI simply displays ``'password'`` which is the remnant of the keyerror.

So we simply only need to delete the password, if it actually exists!
The problem arises, when the SQL resolver does not contain a password mapping.
But the problem occurs here: https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/user.py#L179

When there is no password mapping, the "attributes" do not contain a password key, and the UI simply displays ``'password'`` which is the remnant of the keyerror.

So we simply only need to delete the password, if it actually exists!

Resolving patch:
diff --git a/privacyidea/api/user.py b/privacyidea/api/user.py
--- a/privacyidea/api/user.py
+++ b/privacyidea/api/user.py
@@ -176,7 +176,8 @@ def create_user_api():
     # Remove the password from the attributes, so that we can hide it in the
     # logs
     password = attributes.get("password")
-    del attributes["password"]
+    if "password" in attributes:
+        del attributes["password"]
     r = create_user(resolvername, attributes, password=password)
     g.audit_object.log({"success": True,
                         "info": u"{0!s}: {1!s}/{2!s}".format(r, username,


Test patch:
Repository: privacyidea/privacyidea

Issue:
policy action webauthn_challenge_text in group WebAuthn
In Scope "enrollment" the policy action should also be put in the group WebAuthn.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/webauthntoken.py b/privacyidea/lib/tokens/webauthntoken.py
--- a/privacyidea/lib/tokens/webauthntoken.py
+++ b/privacyidea/lib/tokens/webauthntoken.py
@@ -697,7 +697,8 @@ def get_class_info(key=None, ret='all'):
                     ACTION.CHALLENGETEXT: {
                         'type': 'str',
                         'desc': _("Use an alternate challenge text for telling the "
-                                  "user to confirm with his WebAuthn device.")
+                                  "user to confirm with his WebAuthn device."),
+                        'group': WEBAUTHNGROUP.WEBAUTHN
                     }
                 }
             }


Test patch:
Repository: privacyidea/privacyidea

Issue:
fixing format errors reported by codacy
Hi,

as I explored the helper tools, I stumbled upon some errors in codacy which were easy enough to fix.

Best regards,

Henning


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/monitoringmodules/sqlstats.py b/privacyidea/lib/monitoringmodules/sqlstats.py
--- a/privacyidea/lib/monitoringmodules/sqlstats.py
+++ b/privacyidea/lib/monitoringmodules/sqlstats.py
@@ -91,7 +91,7 @@ def add_value(self, stats_key, stats_value, timestamp, reset_values=False):
                 self.session.commit()
         except Exception as exx:  # pragma: no cover
             log.error(u"exception {0!r}".format(exx))
-            log.error(u"DATA: {0!s} -> {0!s}".format(stats_key, stats_value))
+            log.error(u"DATA: {0!s} -> {1!s}".format(stats_key, stats_value))
             log.debug(u"{0!s}".format(traceback.format_exc()))
             self.session.rollback()
 
diff --git a/privacyidea/lib/policy.py b/privacyidea/lib/policy.py
--- a/privacyidea/lib/policy.py
+++ b/privacyidea/lib/policy.py
@@ -613,7 +613,7 @@ def list_policies(self, name=None, scope=None, realm=None, active=None,
                 if not policy.get("client"):
                     new_policies.append(policy)
             reduced_policies = new_policies
-            log.debug("Policies after matching client".format(
+            log.debug("Policies after matching client: {0!s}".format(
                 reduced_policies))
 
         if sort_by_priority:
diff --git a/privacyidea/lib/privacyideaserver.py b/privacyidea/lib/privacyideaserver.py
--- a/privacyidea/lib/privacyideaserver.py
+++ b/privacyidea/lib/privacyideaserver.py
@@ -75,7 +75,7 @@ def request(config, user, password):
                   "{0!s}".format(response.status_code))
         if response.status_code != 200:
             log.warning("The request to the remote privacyIDEA server {0!s} "
-                        "returned a status code: {0!s}".format(config.url,
+                        "returned a status code: {1!s}".format(config.url,
                                                                response.status_code))
             return False
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
Catch realm "priority" error in WebUI
Hi!

A missing priority value is not caught by the WebUI, when editing an existing realm (e.g. to add another resolver).

Best,

Henning


Discussion:

Resolving patch:
diff --git a/privacyidea/api/lib/utils.py b/privacyidea/api/lib/utils.py
--- a/privacyidea/api/lib/utils.py
+++ b/privacyidea/api/lib/utils.py
@@ -267,7 +267,7 @@ def get_priority_from_param(param):
     """
     priority = {}
     for k, v in param.items():
-        if k.startswith("priority."):
+        if k.startswith("priority.") and isinstance(v, int):
             priority[k[len("priority."):]] = int(v)
     return priority
 


Test patch:
diff --git a/tests/test_api_lib_utils.py b/tests/test_api_lib_utils.py
--- a/tests/test_api_lib_utils.py
+++ b/tests/test_api_lib_utils.py
@@ -6,7 +6,8 @@
 
 from privacyidea.api.lib.utils import (getParam,
                                        check_policy_name,
-                                       verify_auth_token, is_fqdn, attestation_certificate_allowed)
+                                       verify_auth_token, is_fqdn, attestation_certificate_allowed,
+                                       get_priority_from_param)
 from privacyidea.lib.error import ParameterError
 import jwt
 import mock
@@ -193,3 +194,10 @@ def test_06_attestation_certificate_allowed(self):
                 }
             )
         )
+
+    def test_07_get_priority_from_param(self):
+        # check if only keys with given integer values are returned
+        param = {'priority.resolver1': 1, 'priority.resolver2': None,
+                 'resolvers': 'resolver1,resolver2,resolver3'}
+        priority = get_priority_from_param(param)
+        self.assertEqual(priority, {'resolver1': 1})
Repository: privacyidea/privacyidea

Issue:
Error "Could not get the userinformation" in log file
### Top-level intent

What did you try to achieve?
WEBUI is not displaying **user** column information in `Tokens -> All tokens`.
Also, the WEBUI is not displaying the **Username** in `Token details for OATH**** -> Assigned User`.

### Steps to reproduce

1. In the WEBUI go to `Tokens -> All tokens`, no information is displayed at the **user** column.
2. In the WEBUI go to `Token details for OATH**** -> Assigned User`, no information is displayed in **Username**.

### Expected outcome

What do you think, how the system should have worked?
WEBUI should display the user information.

### Actual outcome

What did actually happen?
An error is logged in the log file:
```
[2020-05-25 18:43:31,475][74911][140538455636608][ERROR][privacyidea.lib.resolvers.SQLIdResolver:353] Could not get the userinformation: DatabaseError('(cx_Oracle.DatabaseError) ORA-00906: missing left parenthesis',)
```

### Configuration

* **privacyIDEA version**: 3.3.1
* **Installation method**: github, PyPI, RHEL package dependencies
* **Python version**: 3.6.9
* **Operating system**: RHEL 7.8
* **Webserver**: NGINX Unit 1.17.0
* **Token database**: Oracle 19c (using SQLAlchemy cx_oracle dialect)

### Log file

**Set PI_LOGLEVEL = logging.DEBUG in pi.cfg and take a look at the privacyidea.log!**
**If appropriate, attach the log file or paste relevant portions.**

Log files attached.
[all_tokens.log](https://github.com/privacyidea/privacyidea/files/4679035/all_tokens.log)
[token_details.log](https://github.com/privacyidea/privacyidea/files/4679036/token_details.log)



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/resolvers/SQLIdResolver.py b/privacyidea/lib/resolvers/SQLIdResolver.py
--- a/privacyidea/lib/resolvers/SQLIdResolver.py
+++ b/privacyidea/lib/resolvers/SQLIdResolver.py
@@ -356,12 +356,14 @@ def getUserInfo(self, userId):
     
     def _get_userid_filter(self, userId):
         column = getattr(self.TABLE, self.map.get("userid"))
-        if isinstance(column.type, Integer):
+        if isinstance(column.type, String):
+            return column == str(userId)
+        elif isinstance(column.type, Integer):
             # since our user ID is usually a string we need to cast
             return column == int(userId)
-        else:
-            # otherwise we cast the column to string (in case of postgres UUIDs)
-            return cast(column, String).like(userId)
+
+        # otherwise we cast the column to string (in case of postgres UUIDs)
+        return cast(column, String).like(userId)
 
     def getUsername(self, userId):
         """


Test patch:
Repository: privacyidea/privacyidea

Issue:
Less prominent translation-missing marker
When there are untranslated strings, they are marked with "[MISSING]: ".

The marking should either be less prominent, or we should completely remove it.

This can be done here:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/app.js#L103

~~~javascript
 gettextCatalog.debug = false;
~~~~

or

~~~~javascript
  gettextCatalog.debug = false;
  gettextCatalog.debugPrefix = "(en) ";
~~~~


Discussion:
We could remove it only for release builds, that way we would still see it during development.
OTOH I am primarily using `en_US` so at least I wouldn't see it.
We should update the translation before the release anyway.
In pi.cfg we can have a setting like PI_TRANSLATION_WARNING = prefix.
Only if this is set, the warning displayed in the UI.

Resolving patch:
diff --git a/privacyidea/config.py b/privacyidea/config.py
--- a/privacyidea/config.py
+++ b/privacyidea/config.py
@@ -53,6 +53,7 @@ class DevelopmentConfig(Config):
     SQLALCHEMY_DATABASE_URI = os.environ.get('DEV_DATABASE_URL') or \
         'sqlite:///' + os.path.join(basedir, 'data-dev.sqlite')
     PI_LOGLEVEL = logging.DEBUG
+    PI_TRANSLATION_WARNING = "[Missing]"
 
 
 class TestingConfig(Config):
diff --git a/privacyidea/webui/login.py b/privacyidea/webui/login.py
--- a/privacyidea/webui/login.py
+++ b/privacyidea/webui/login.py
@@ -86,6 +86,8 @@ def single_page_application():
     #    PI_CUSTOMIZATION/views/includes/token.enroll.post.bottom.html
     # Get the hidden external links
     external_links = current_app.config.get("PI_EXTERNAL_LINKS", True)
+    # Read the UI translation warning
+    translation_warning = current_app.config.get("PI_TRANSLATION_WARNING", False)
     # Get the logo file
     logo = current_app.config.get("PI_LOGO", "privacyIDEA1.png")
     browser_lang = request.accept_languages.best_match(["en", "de", "de-DE"], default="en").split("-")[0]
@@ -152,6 +154,7 @@ def single_page_application():
         'browser_lang': browser_lang,
         'remote_user': remote_user,
         'theme': theme,
+        'translation_warning': translation_warning,
         'password_reset': password_reset,
         'hsm_ready': hsm_ready,
         'has_job_queue': str(has_job_queue()),


Test patch:
Repository: privacyidea/privacyidea

Issue:
linotp migration script: failsafe for tokeninfos
The linotp-magration script fails to migrate token infos, if a phone number contains special characters, which can happen if a user entered faulty characters there.

Als long as #1874 is not completely done, we could add some failsafe for migrating tokeninfo fields.


Discussion:

Resolving patch:
diff --git a/tools/privacyidea-migrate-linotp.py b/tools/privacyidea-migrate-linotp.py
--- a/tools/privacyidea-migrate-linotp.py
+++ b/tools/privacyidea-migrate-linotp.py
@@ -1,6 +1,8 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
+#  2020-06-13 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#             Add failsafe for tokeninfo
 #  2019-10-04 Cornelius Kölbel <cornelius.koelbel@netknights.it>
 #             Allow encryption and config file
 #  2018-02-09 Cornelius Kölbel <cornelius.koelbel@netknights.it>
@@ -72,6 +74,7 @@
     "MIGRATE": {
         "tokens": true,
         "tokeninfo": true,
+        "tokeninfo_unicode_keys": ["phone"],
         "assignments": false
     },
     "ASSIGNMENTS": {
@@ -435,6 +438,8 @@ def insert_chunks(conn, table, values, chunk_size=100000):
             if config_obj.MIGRATE.get("tokeninfo") and ti:
                 # Add tokeninfo for this token
                 for (k, v) in ti.items():
+                    if k in config_obj.MIGRATE.get("tokeninfo_unicode_keys", []):
+                        v = v.encode('ascii', 'ignore').decode('utf8')
                     tokeninfo_values.append(dict(
                         serial=r["LinOtpTokenSerialnumber"],
                         Key=k, Value=v,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Not all audit data is passed through the container audit
The [`add_to_log()`](https://github.com/privacyidea/privacyidea/blob/8572e6b2c3049fac9476687fcfdccbe03dec6ef4/privacyidea/lib/auditmodules/base.py#L184) and [`add_policy()`](https://github.com/privacyidea/privacyidea/blob/8572e6b2c3049fac9476687fcfdccbe03dec6ef4/privacyidea/lib/auditmodules/base.py#L200) functions don't pass their data on to a writable container module. Thus they are missing from the audit entry.



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/containeraudit.py b/privacyidea/lib/auditmodules/containeraudit.py
--- a/privacyidea/lib/auditmodules/containeraudit.py
+++ b/privacyidea/lib/auditmodules/containeraudit.py
@@ -68,6 +68,20 @@ def log(self, param):
         for module in self.write_modules:
             module.log(param)
 
+    def add_to_log(self, param, add_with_comma=False):
+        """
+        Call the add_to_log method for all writeable modules
+        """
+        for module in self.write_modules:
+            module.add_to_log(param, add_with_comma)
+
+    def add_policy(self, policyname):
+        """
+        Call the add_policy method for all writeable modules
+        """
+        for module in self.write_modules:
+            module.add_policy(policyname)
+
     def search(self, search_dict, page_size=15, page=1, sortorder="asc",
                timelimit=None):
         """


Test patch:
diff --git a/tests/test_lib_audit.py b/tests/test_lib_audit.py
--- a/tests/test_lib_audit.py
+++ b/tests/test_lib_audit.py
@@ -406,15 +406,21 @@ def test_15_container_audit_check_audit(self):
                             "PI_AUDIT_NO_SIGN": True,
                             "PI_AUDIT_SQL_URI": 'sqlite:///' + os.path.join(basedir, 'data-test.sqlite')})
         a.log({"action": "something_test_35"})
+        a.add_to_log({'action_detail': 'some detail'})
+        a.add_policy('some policy')
         a.finalize_log()
         r = a.search({"action": "*something_test_35*"})
         # The search should go to the sql audit
         self.assertEqual(r.total, 1)
         self.assertEqual(r.auditdata[0].get("action"), u"something_test_35")
+        self.assertEqual(r.auditdata[0].get("action_detail"), u"some detail")
+        self.assertEqual(r.auditdata[0].get("policies"), u"some policy")
         # now check the log file
         with open("audit.log") as file:
             c = file.readlines()
             self.assertIn("something_test_35", c[-1])
+            self.assertIn("some detail", c[-1])
+            self.assertIn("some policy", c[-1])
         os.unlink('audit.log')
 
     def test_20_container_audit_wrong_module(self):
Repository: privacyidea/privacyidea

Issue:
broken response signature in case of error
All error handlers in [`before_after.py`](https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/before_after.py) are decorated with the `sign_response` postrequest decorator. So if an error occurs, the response will be signed.
Unfortunately the `after_request()` handler will also be called and signs the response again including the existing signature and overwriting it. Thus making it impossible to verify the signature.
Only the `after_request()` handler should sign the response since it will be called regardless of a successful or erroneous response.


Discussion:

Resolving patch:
diff --git a/privacyidea/api/before_after.py b/privacyidea/api/before_after.py
--- a/privacyidea/api/before_after.py
+++ b/privacyidea/api/before_after.py
@@ -257,7 +257,6 @@ def after_request(response):
 @eventhandling_blueprint.app_errorhandler(AuthError)
 @subscriptions_blueprint.app_errorhandler(AuthError)
 @monitoring_blueprint.app_errorhandler(AuthError)
-@postrequest(sign_response, request=request)
 def auth_error(error):
     if "audit_object" in g:
         message = ''
@@ -292,7 +291,6 @@ def auth_error(error):
 @recover_blueprint.app_errorhandler(PolicyError)
 @subscriptions_blueprint.app_errorhandler(PolicyError)
 @monitoring_blueprint.app_errorhandler(PolicyError)
-@postrequest(sign_response, request=request)
 def policy_error(error):
     if "audit_object" in g:
         g.audit_object.add_to_log({"info": error.message}, add_with_comma=True)
@@ -314,7 +312,6 @@ def policy_error(error):
 @register_blueprint.app_errorhandler(ResourceNotFoundError)
 @recover_blueprint.app_errorhandler(ResourceNotFoundError)
 @subscriptions_blueprint.app_errorhandler(ResourceNotFoundError)
-@postrequest(sign_response, request=request)
 def resource_not_found_error(error):
     """
     This function is called when an ResourceNotFoundError occurs.
@@ -341,7 +338,6 @@ def resource_not_found_error(error):
 @recover_blueprint.app_errorhandler(privacyIDEAError)
 @subscriptions_blueprint.app_errorhandler(privacyIDEAError)
 @monitoring_blueprint.app_errorhandler(privacyIDEAError)
-@postrequest(sign_response, request=request)
 def privacyidea_error(error):
     """
     This function is called when an privacyIDEAError occurs.
@@ -369,7 +365,6 @@ def privacyidea_error(error):
 @recover_blueprint.app_errorhandler(500)
 @subscriptions_blueprint.app_errorhandler(500)
 @monitoring_blueprint.app_errorhandler(500)
-@postrequest(sign_response, request=request)
 def internal_error(error):
     """
     This function is called when an internal error (500) occurs.


Test patch:
Repository: privacyidea/privacyidea

Issue:
RADIUS and Remote Tokens do not pass the options parameter to check_pin
In some token classes the `options` parameter (which contains the `g` object, policies and audit) doesn't get passed when calling further functions. This leads to broken policy decorators (e.g. [`check_pin()`](https://github.com/privacyidea/privacyidea/blob/09a857a27f1558b2c9e6a9703de33797017c1a5d/privacyidea/lib/tokens/remotetoken.py#L206))


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/radiustoken.py b/privacyidea/lib/tokens/radiustoken.py
--- a/privacyidea/lib/tokens/radiustoken.py
+++ b/privacyidea/lib/tokens/radiustoken.py
@@ -191,7 +191,7 @@ def is_challenge_request(self, passw, user=None, options=None):
         # should we check the pin locally?
         if self.check_pin_local:
             # With a local PIN the challenge response is always a privacyIDEA challenge response!
-            res = self.check_pin(passw)
+            res = self.check_pin(passw, user=user, options=options)
             return res
 
         else:
@@ -406,7 +406,7 @@ def authenticate(self, passw, user=None, options=None):
             (_res, pin, otpval) = self.split_pin_pass(passw, user,
                                                       options=options)
 
-            if not self.check_pin(pin):
+            if not self.check_pin(pin, user=user, options=options):
                 return False, -1, {'message': "Wrong PIN"}
 
         # attempt to retrieve saved state/result
diff --git a/privacyidea/lib/tokens/remotetoken.py b/privacyidea/lib/tokens/remotetoken.py
--- a/privacyidea/lib/tokens/remotetoken.py
+++ b/privacyidea/lib/tokens/remotetoken.py
@@ -203,7 +203,7 @@ def authenticate(self, passw, user=None, options=None):
             (_res, pin, otpval) = self.split_pin_pass(passw, user,
                                                       options=options)
 
-            if not TokenClass.check_pin(self, pin):
+            if not TokenClass.check_pin(self, pin, user=user, options=options):
                 return False, otp_counter, {'message': "Wrong PIN"}
 
         otp_count = self.check_otp(otpval, options=options)
@@ -312,8 +312,7 @@ def is_challenge_request(self, passw, user=None, options=None):
         request_is_valid = False
 
         if self.check_pin_local:
-            pin_match = self.check_pin(passw, user=user,
-                                         options=options)
+            pin_match = self.check_pin(passw, user=user, options=options)
             if pin_match is True:
                 request_is_valid = True
 


Test patch: