internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
Audit Log does not contain token/resolver details for /auth compared to /validate/check
# What did you try to do?

Performed a successful POST to /auth and /validate/check

# What outcome did you expect?

Audit log should list token serial, user, realm, resolver, etc for both.

# What outcome did you experience?

/validate/check has following fields filled in auditlog:
>serial, user, realm, resolver

/auth has following fields filled in auditlog:
>user, realm

# Configuration

* privacyIDEA Version: 3.0
* Installation method: ubuntu packages

## more details: 

* OS: Ubuntu 16.04LTS
* Webserver: Apache
* Tokendatabase: MySQL




Discussion:
Thank you for the report. You're right, we don't log the user resolver in ``get_auth_token``. This is a bit inconsistent.  Of course, we can only log the resolver if the user actually exists in a resolver (which is not the case for admins added via ``pi-manage admin``). 

Resolving patch:
diff --git a/privacyidea/api/auth.py b/privacyidea/api/auth.py
--- a/privacyidea/api/auth.py
+++ b/privacyidea/api/auth.py
@@ -267,10 +267,14 @@ def get_auth_token():
             g.audit_object.log({"user": "",
                                 "administrator": user_obj.login,
                                 "realm": user_obj.realm,
+                                "resolver": user_obj.resolver,
+                                "serial": details.get('serial', None) if details else None,
                                 "info": log_used_user(user_obj)})
         else:
             g.audit_object.log({"user": user_obj.login,
                                 "realm": user_obj.realm,
+                                "resolver": user_obj.resolver,
+                                "serial": details.get('serial', None) if details else None,
                                 "info": log_used_user(user_obj)})
 
     if not admin_auth and not user_auth:


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add export-counter script
We add an existing script, that can export the otp counter into a CSV with (serial, counter).


Discussion:

Resolving patch:
diff --git a/tools/privacyidea-export-privacyidea-counter.py b/tools/privacyidea-export-privacyidea-counter.py
new file mode 100755
--- /dev/null
+++ b/tools/privacyidea-export-privacyidea-counter.py
@@ -0,0 +1,59 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+#  2018-05-27 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#             init
+
+__doc__ = """
+This script exports counter from privacyIDEA to a csv file.
+It exports 
+   serial, counter
+
+"""
+
+
+import argparse
+import sys
+from sqlalchemy import create_engine
+from sqlalchemy import Table, MetaData, Column
+from sqlalchemy import Integer, Unicode, Boolean
+from sqlalchemy.sql import select
+from privacyidea.models import Token
+import argparse
+import sys
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+metadata = MetaData()
+
+
+def get_privacyidea_uri(config_file):
+    with open(config_file) as f:
+        content = f.readlines()
+
+    lines = [l.strip() for l in content]
+    sql_uri = ""
+    for line in lines:
+        if line.startswith("SQLALCHEMY_DATABASE_URI"):
+            sql_uri = line.split("=", 1)[1].strip().strip("'").strip('"')
+    return sql_uri
+
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument("-c", "--config", help="privacyIDEA config file. We only need the SQLALCHEMY_DATABASE_URI.",
+                    required=True)
+args = parser.parse_args()
+
+# Parse data
+
+SQL_URI = get_privacyidea_uri(args.config)
+
+# Start DB stuff
+
+pi_engine = create_engine(SQL_URI)
+conn_pi = pi_engine.connect()
+
+s = select([Token.serial, Token.count])
+result = pi_engine.execute(s)
+
+for r in result:
+    print(u"{0!s}, {1!s}".format(r.serial, r.count))


Test patch:
Repository: privacyidea/privacyidea

Issue:
Rearrange migration scripts
After merging of several issues like #1495, #1715, #1713 we need to check the database migration scripts for branches

    pi-manage db branches

and might need to rearrange the different migration scripts in a linear order.


Discussion:

Resolving patch:
diff --git a/migrations/versions/3d7f8b29cbb1_.py b/migrations/versions/3d7f8b29cbb1_.py
--- a/migrations/versions/3d7f8b29cbb1_.py
+++ b/migrations/versions/3d7f8b29cbb1_.py
@@ -1,14 +1,14 @@
 """Add a generic admin policy for all admin read actions
 
 Revision ID: 3d7f8b29cbb1
-Revises: 849170064430
+Revises: dceb6cd3c41e
 Create Date: 2019-07-04 22:37:53.594962
 
 """
 
 # revision identifiers, used by Alembic.
 revision = '3d7f8b29cbb1'
-down_revision = '849170064430'
+down_revision = 'dceb6cd3c41e'
 
 from alembic import op
 import sqlalchemy as sa


Test patch:
Repository: privacyidea/privacyidea

Issue:
Fix typo in registration token
The example of the registration token contains a typo.
The toketype of course is a "registration" token, not a "register".


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/registrationtoken.py b/privacyidea/lib/tokens/registrationtoken.py
--- a/privacyidea/lib/tokens/registrationtoken.py
+++ b/privacyidea/lib/tokens/registrationtoken.py
@@ -64,7 +64,7 @@ class RegistrationTokenClass(PasswordTokenClass):
            Host: example.com
            Accept: application/json
 
-           type=register
+           type=registration
            user=cornelius
            realm=realm1
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
Make TiQR infoURl configurable
The TiQR token can receice an info url. Currently this is fixed to privacyidea.org.
This should be configuable in the token settings.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/tiqrtoken.py b/privacyidea/lib/tokens/tiqrtoken.py
--- a/privacyidea/lib/tokens/tiqrtoken.py
+++ b/privacyidea/lib/tokens/tiqrtoken.py
@@ -258,11 +258,13 @@ def api_endpoint(cls, request, g):
             reg_server = get_from_config("tiqr.regServer")
             auth_server = get_from_config("tiqr.authServer") or reg_server
             logo_url = get_from_config("tiqr.logoUrl")
+            info_url = get_from_config("tiqr.infoUrl") or \
+                    "https://www.privacyidea.org"
 
             service = {"displayName": service_displayname,
                        "identifier": service_identifier,
                        "logoUrl": logo_url,
-                       "infoUrl": "https://www.privacyidea.org",
+                       "infoUrl": info_url,
                        "authenticationUrl":
                            "{0!s}".format(auth_server),
                        "ocraSuite": ocrasuite,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Event handler for user API endpoint
Add event handler hook to user endpoints, to do something when a user is created or modified.

This is basically the result of #455


Discussion:
Just need to add the corresponding decorators

Resolving patch:
diff --git a/privacyidea/api/user.py b/privacyidea/api/user.py
--- a/privacyidea/api/user.py
+++ b/privacyidea/api/user.py
@@ -3,6 +3,8 @@
 # http://www.privacyidea.org
 # (c) cornelius kölbel, privacyidea.org
 #
+# 2019-09-04 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#            Add decorators for event handling
 # 2016-07-12 Cornelius Kölbel <cornelius@privacyidea.org>
 #            Add decorator for checking realmadmin realms during userlist
 # 2014-12-08 Cornelius Kölbel, <cornelius@privacyidea.org>
@@ -31,6 +33,8 @@
 from ..lib.policy import ACTION
 from privacyidea.api.auth import admin_required, user_required
 from privacyidea.lib.user import create_user, get_user_from_param, User
+from privacyidea.lib.event import event
+
 
 from flask import (g)
 from ..lib.user import get_user_list
@@ -47,6 +51,7 @@
 @prepolicy(realmadmin, request, ACTION.USERLIST)
 @prepolicy(check_base_action, request, ACTION.USERLIST)
 @user_required
+@event("user_list", request, g)
 def get_users():
     """
     list the users in a realm
@@ -110,6 +115,7 @@ def get_users():
 @user_blueprint.route('/<resolvername>/<username>', methods=['DELETE'])
 @admin_required
 @prepolicy(check_base_action, request, ACTION.DELETEUSER)
+@event("user_delete", request, g)
 def delete_user(resolvername=None, username=None):
     """
     Delete a User in the user store.
@@ -138,6 +144,7 @@ def delete_user(resolvername=None, username=None):
 @user_blueprint.route('/', methods=['POST'])
 @admin_required
 @prepolicy(check_base_action, request, ACTION.ADDUSER)
+@event("user_add", request, g)
 def create_user_api():
     """
     Create a new user in the given resolver.
@@ -180,6 +187,7 @@ def create_user_api():
 @user_blueprint.route('', methods=['PUT'])
 @user_blueprint.route('/', methods=['PUT'])
 @prepolicy(check_base_action, request, ACTION.UPDATEUSER)
+@event("user_update", request, g)
 def update_user():
     """
     Edit a user in the user store.


Test patch:
Repository: privacyidea/privacyidea

Issue:
pre-auth-event-handler loops over all token
Using a pre-eventhandler with the auth-event (as implemented in #1567), the [`check_condition()`](https://github.com/privacyidea/privacyidea/blob/5f282d70e9201f1638ccfa94217c66760dd4fc4a/privacyidea/lib/eventhandler/base.py#L338) function loops over all tokens (and creates them) causing a significant delay if there are a lot of token in the db.
Since we do not have a user object (yet), the SQL-query generated in [`get_tokens()`](https://github.com/privacyidea/privacyidea/blob/5f282d70e9201f1638ccfa94217c66760dd4fc4a/privacyidea/lib/token.py#L378) matches all tokens.
See also  #1436 / #1645 


Discussion:
How is this a bug?
This should be fixed if we implement #1810 

Resolving patch:
diff --git a/privacyidea/lib/eventhandler/base.py b/privacyidea/lib/eventhandler/base.py
--- a/privacyidea/lib/eventhandler/base.py
+++ b/privacyidea/lib/eventhandler/base.py
@@ -337,10 +337,13 @@ def check_condition(self, options):
         if serial:
             # We have determined the serial number from the request.
             token_obj_list = get_tokens(serial=serial)
-        else:
+        elif user:
             # We have to determine the token via the user object. But only if
             #  the user has only one token
             token_obj_list = get_tokens(user=user)
+        else:
+            token_obj_list = []
+
         if len(token_obj_list) == 1:
             # There is a token involved, so we determine it's resolvers and realms
             token_obj = token_obj_list[0]


Test patch:
Repository: privacyidea/privacyidea

Issue:
Usage of smsgateway endpoint does not get logged to audit
Due to a missing `after_request` decorator, the audit log did no get finalized.


Discussion:

Resolving patch:
diff --git a/privacyidea/api/before_after.py b/privacyidea/api/before_after.py
--- a/privacyidea/api/before_after.py
+++ b/privacyidea/api/before_after.py
@@ -215,6 +215,7 @@ def before_request():
 @smtpserver_blueprint.after_request
 @eventhandling_blueprint.after_request
 @radiusserver_blueprint.after_request
+@smsgateway_blueprint.after_request
 @periodictask_blueprint.after_request
 @privacyideaserver_blueprint.after_request
 @client_blueprint.after_request


Test patch:
diff --git a/tests/base.py b/tests/base.py
--- a/tests/base.py
+++ b/tests/base.py
@@ -1,7 +1,6 @@
 # -*- coding: utf-8 -*-
 
 import unittest
-import json
 import mock
 
 from privacyidea.app import create_app
@@ -190,6 +189,9 @@ def find_most_recent_audit_entry(self, **filter_data):
                                            data=sorted_filter,
                                            headers={"Authorization": self.at}):
             res = self.app.full_dispatch_request()
+            self.assertTrue(res.is_json, res)
+            result = res.json['result']
+            self.assertIn('auditdata', result['value'])
             # return the last entry
             return res.json["result"]["value"]["auditdata"][0]
 
diff --git a/tests/test_api_smsgateway.py b/tests/test_api_smsgateway.py
--- a/tests/test_api_smsgateway.py
+++ b/tests/test_api_smsgateway.py
@@ -1,7 +1,9 @@
-import json
+# -*- coding: utf-8 -*-
+
 from .base import MyApiTestCase
 from privacyidea.lib.policy import set_policy, delete_policy, SCOPE, ACTION
 
+
 class APISmsGatewayTestCase(MyApiTestCase):
 
     def test_01_crud_smsgateway(self):
@@ -17,6 +19,10 @@ def test_01_crud_smsgateway(self):
             detail = res.json.get("detail")
             self.assertEqual(result.get("value"), [])
 
+        # check that we have an entry in the audit log
+        audit_entry = self.find_most_recent_audit_entry(action="GET /smsgateway*")
+        self.assertEqual(audit_entry['success'], 1, audit_entry)
+
         # create an sms gateway definition
         param = {
             "name": "myGW",
Repository: privacyidea/privacyidea

Issue:
Allow file-syntax in email notification
In the email token we allow a file syntax like

    file:/etc/privacyidea/email-template.html

We need to also allow this in the notification handler.
Allow file-syntax in email notification
In the email token we allow a file syntax like

    file:/etc/privacyidea/email-template.html

We need to also allow this in the notification handler.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/eventhandler/usernotification.py b/privacyidea/lib/eventhandler/usernotification.py
--- a/privacyidea/lib/eventhandler/usernotification.py
+++ b/privacyidea/lib/eventhandler/usernotification.py
@@ -52,6 +52,7 @@
 from privacyidea.lib import _
 import logging
 import os
+import traceback
 
 log = logging.getLogger(__name__)
 
@@ -293,6 +294,16 @@ def do(self, action, options=None):
             # In case of "savefile" we do not need a recipient
             # Collect all data
             body = handler_options.get("body") or DEFAULT_BODY
+            if body.startswith("file:"):  # pragma no cover
+                # We read the template from the file.
+                filename = body[5:]
+                try:
+                    with open(filename, "r") as f:
+                        body = f.read()
+                except Exception as e:
+                    log.warning(u"Failed to read email template from file {0!r}: {1!r}".format(filename, e))
+                    log.debug(u"{0!s}".format(traceback.format_exc()))
+
             subject = handler_options.get("subject") or \
                       "An action was performed on your token."
             serial = request.all_data.get("serial") or \


Test patch:
Repository: privacyidea/privacyidea

Issue:
More detailed description of splitAtSign
We should have a more precise description how ``splitAtSign`` interacts with a given ``realm`` parameter.
We should describe, that it works with ``/validate/`` and ``/auth`` endpoint.

We can have a link for a more detailed description here:
https://github.com/privacyidea/privacyidea/blob/master/doc/configuration/system_config.rst#split--sign

Which links to this location
https://github.com/privacyidea/privacyidea/blob/master/doc/configuration/realms.rst
that has the detailed description.



Discussion:

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -151,6 +151,10 @@ def wrapper(decorator):
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 
+html_css_files = [
+    'css/custom.css',
+]
+
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
 #html_last_updated_fmt = '%b %d, %Y'


Test patch:
Repository: privacyidea/privacyidea

Issue:
Missing requirement in setup.py
The `flask-versioned` package is missing in `setup.py`s `install_requires` list. When installing privacyIDEA via `setup.py` or `pip` this will break.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -60,7 +60,8 @@ def get_file_list(file_path):
                     "flask-babel>=0.9",
                     "croniter>=0.3.8",
                     "oauth2client>=2.0.1",
-                    "configobj>=5.0.6"
+                    "configobj>=5.0.6",
+                    "flask-versioned>=0.9.4"
                     ]
 
 # For python 2.6 we need additional dependency importlib


Test patch:
Repository: privacyidea/privacyidea

Issue:
Inconsistency between documentation and implementation for API POST /smsgateway¶
Documentation states that the JSON Parameters includes option**s**.*
https://privacyidea.readthedocs.io/en/latest/modules/api/smsgateway.html

However, the implementation uses option.*
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/smsgateway.py
```python
for k, v in param.items():
        if k.startswith("option."):
            options[k[7:]] = v
```


Discussion:
Thank you for filing an issue and sharing your observations or ideas. Please be sure to provide as many information as possible to help us working on this issue.
Thanks for the notification, we'll fix this in the next update.

Resolving patch:
diff --git a/privacyidea/api/smsgateway.py b/privacyidea/api/smsgateway.py
--- a/privacyidea/api/smsgateway.py
+++ b/privacyidea/api/smsgateway.py
@@ -88,7 +88,7 @@ def set_gateway():
     :jsonparam name: The unique identifier of the SMS gateway definition
     :jsonparam module: The providermodule name
     :jsonparam description: An optional description of the definition
-    :jsonparam options.*: Additional options for the provider module (module
+    :jsonparam option.*: Additional options for the provider module (module
         specific)
     """
     param = request.all_data


Test patch:
Repository: privacyidea/privacyidea

Issue:
Add some diagrams to the documentation
It would be nice to have a few diagrams in the documentation. Specifically I think a deployment diagram giving an example for the components involved in a typical use of push tokens, as well as sequence diagrams for the three authentication modes would be nice.


Discussion:
This would be great.

It would be great if we could do a structural apptoach. I think there is a markdown extension to do diagrams.
(To avoid to actually *draw* stuff)
Do You mean the readthedocs documentation or the guthub docs?
RTD uses reST and github Markdown.
[PlantUML](http://plantuml.com/) looks promising. It works with [Markdown](https://blog.anoff.io/2018-07-31-diagrams-with-plantuml/) and [RestructuredText](https://build-me-the-docs-please.readthedocs.io/en/latest/Using_Sphinx/UsingGraphicsAndDiagramsInSphinx.html).
I was mainly referring to RTD. PlantUML is the approach I am favoring as well. I have written a few diagrams locally. I am not sure on the best way to add the though.
Apparently readthedocs.org has recently gained the ability to render PlantUML, however it is woefully undocumented (see [this issue](https://github.com/readthedocs/readthedocs.org/issues/3885)).


We have decided to have readthedocs render the diagrams through its internal plantuml support. This will require some changes to how readthedocs is configured in our project.
depends #1958 

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -58,7 +58,7 @@ def wrapper(decorator):
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = ['sphinx.ext.autodoc', 'sphinx.ext.imgmath', 'sphinx.ext.viewcode', 
-              'sphinxcontrib.autohttp.flask']
+              'sphinxcontrib.autohttp.flask', 'sphinxcontrib.plantuml']
 http_index_ignore_prefixes = ['/token']
 
 # Add any paths that contain templates here, relative to this directory.
@@ -273,3 +273,13 @@ def wrapper(decorator):
 
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
 #texinfo_show_urls = 'footnote'
+
+#
+# PlantUML
+#
+
+# Run plantUML under Java in headless mode. This is needed for compatibility with readthedocs.io.
+plantuml = 'java -Djava.awt.headless=true -jar /usr/share/plantuml/plantuml.jar'
+
+# Use SVG inside <object> in supported browsers (all except IE8), falling back to PNG.
+plantuml_output_format = 'svg'


Test patch:
Repository: privacyidea/privacyidea

Issue:
Missing policy CACONNECTORREAD
We are checking for the policy "CACONNECTORREAD", but it does not exist.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/policy.py b/privacyidea/lib/policy.py
--- a/privacyidea/lib/policy.py
+++ b/privacyidea/lib/policy.py
@@ -1511,6 +1511,11 @@ def get_static_policy_definitions(scope=None):
                                                  "CA Connector definitions."),
                                        "group": GROUP.SYSTEM,
                                        'mainmenu': [MAIN_MENU.CONFIG]},
+            ACTION.CACONNECTORREAD: {'type': 'bool',
+                                     "desc": _("Admin is allowed to read CA Connector "
+                                               "definitions."),
+                                     "group": GROUP.SYSTEM,
+                                     'mainmenu': [MAIN_MENU.CONFIG]},
             ACTION.MACHINERESOLVERWRITE: {'type': 'bool',
                                           'desc': _("Admin is allowed to "
                                                     "write and modify the "
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,7 @@
 import sys
 
 #VERSION="2.1dev4"
-VERSION="3.2.2dev2"
+VERSION="3.2.2dev3"
 
 # Taken from kennethreitz/requests/setup.py
 package_directory = os.path.realpath(os.path.dirname(__file__))


Test patch:
Repository: privacyidea/privacyidea

Issue:
url_decode does padding incorrectly
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/tokens/u2f.py#L53 adds padding to a string containing base64. Th padding should get the message length to a multiple of four, however the current implementation will add three bytes of padding where one is required and vice versa. This bug does not cause any issues, since the messages we decode all need two bytes of padding currently.


Discussion:
So what is your suggestion?
Fix it in which way?
Move the method to lib/utils.py?
Nono, all I am suggesting is we fix this problem. No need to move the function or anything. I just opened this issue so I don't forget.

Resolving patch:
diff --git a/privacyidea/lib/tokens/u2f.py b/privacyidea/lib/tokens/u2f.py
--- a/privacyidea/lib/tokens/u2f.py
+++ b/privacyidea/lib/tokens/u2f.py
@@ -50,7 +50,7 @@ def url_decode(url):
     :return: the decoded string
     :rtype: bytes
     """
-    pad_len = len(url) % 4
+    pad_len = -len(url) % 4
     padding = pad_len * "="
     res = base64.urlsafe_b64decode(to_bytes(url + padding))
     return res


Test patch:
Repository: privacyidea/privacyidea

Issue:
Initialize the resolver type of user object
It seems like that in certain corner cases (like with the token janitor and orphaned tokens) it can happen that the resolver type (``self.rtype``) of a user object is not initialized. Thus it will not exist and code could break later. See:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/user.py#L99


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/user.py b/privacyidea/lib/user.py
--- a/privacyidea/lib/user.py
+++ b/privacyidea/lib/user.py
@@ -94,6 +94,7 @@ def __init__(self, login="", realm="", resolver=""):
             resolver = ""
         self.resolver = resolver or ""
         self.uid = None
+        self.rtype = None
         # Enrich user object with information from the userstore or from the
         # usercache
         if login:


Test patch: