internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: privacyidea/privacyidea

Issue:
TTL of push token in minutes
the ttl of the push token is supposed to be in minutes! 
/Me/ should fix this. 


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/smsprovider/FirebaseProvider.py b/privacyidea/lib/smsprovider/FirebaseProvider.py
--- a/privacyidea/lib/smsprovider/FirebaseProvider.py
+++ b/privacyidea/lib/smsprovider/FirebaseProvider.py
@@ -115,7 +115,7 @@ def parameters(cls):
                                      ' Usually it is the endpoint /ttype/push of the privacyIDEA server.')},
                       FIREBASE_CONFIG.TTL: {
                           "required": True,
-                          "description": _('How long should the second step of the enrollment be accepted (in seconds).')
+                          "description": _('How long should the second step of the enrollment be accepted (in minutes).')
                       },
                       FIREBASE_CONFIG.PROJECT_ID: {
                           "required": True,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Documentation for queue
Add documentation for 
* what and why queue?
* how to configure queue
* queue worker


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/queue.py b/privacyidea/lib/queue.py
--- a/privacyidea/lib/queue.py
+++ b/privacyidea/lib/queue.py
@@ -53,6 +53,7 @@ def jobs(self):
     def register_job(self, name, func, args, kwargs):
         """
         Register a job with the collector.
+
         :param name: unique name of the job
         :param func: function of the job
         :param args: arguments passed to the job queue's ``register_job`` method
@@ -69,6 +70,7 @@ def register_app(self, app):
         Register all collected jobs with this application.
         This instance is shared between threads!
         This function should only be called once per process.
+
         :param app: privacyIDEA app
         """
         with app.app_context():
@@ -144,6 +146,7 @@ def wrap_job(name, result):
     The returned function will always return ``result``.
     This assumes that a queue is configured! Otherwise, calling the
     resulting function will fail with a ServerError.
+
     :return: a function
     """
     def caller(*args, **kwargs):
diff --git a/privacyidea/lib/queues/base.py b/privacyidea/lib/queues/base.py
--- a/privacyidea/lib/queues/base.py
+++ b/privacyidea/lib/queues/base.py
@@ -40,6 +40,7 @@ def __init__(self, options):
     def register_job(self, name, func):  # pragma: no cover
         """
         Add a job to the internal registry.
+
         :param name: Unique job name
         :param func: Function that should be executed by an external job queue
         """
@@ -48,6 +49,7 @@ def register_job(self, name, func):  # pragma: no cover
     def enqueue(self, name, args, kwargs):  # pragma: no cover
         """
         Schedule an invocation of a job on the external job queue.
+
         :param name: Unique job name
         :param args: Tuple of positional arguments
         :param kwargs: Dictionary of keyword arguments


Test patch:
Repository: privacyidea/privacyidea

Issue:
Push token custom question is not used, URL below QR
The configurable question for push token is not used when trying to authenticate, instead the title is used for both fields.

When rolling out a push token, below the QR Code still is the URL.



Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/pushtoken.py b/privacyidea/lib/tokens/pushtoken.py
--- a/privacyidea/lib/tokens/pushtoken.py
+++ b/privacyidea/lib/tokens/pushtoken.py
@@ -506,7 +506,7 @@ def create_challenge(self, transactionid=None, options=None):
             fb_gateway = create_sms_instance(fb_identifier)
             url = fb_gateway.smsgateway.option_dict.get(FIREBASE_CONFIG.REGISTRATION_URL)
             message_on_mobile = get_action_values_from_options(SCOPE.AUTH,
-                                                   PUSH_ACTION.MOBILE_TITLE,
+                                                   PUSH_ACTION.MOBILE_TEXT,
                                                    options) or DEFAULT_MOBILE_TEXT
             title = get_action_values_from_options(SCOPE.AUTH,
                                                    PUSH_ACTION.MOBILE_TITLE,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Realm-Select box with broken "placeholder"
In the login screen there is a realm select box.

The placeholder for the select box does not work:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/components/login/views/login.html#L63

We could either fix the placeholder or preselect the default realm.


Discussion:
Apart from that:
The loginControllers.js could be edited to the following:
$scope.piRealms = obj.val().mysplit(",").sort();

Sorting the realms alphabetically meets the user's acceptance slightly. 


Resolving patch:
diff --git a/privacyidea/webui/login.py b/privacyidea/webui/login.py
--- a/privacyidea/webui/login.py
+++ b/privacyidea/webui/login.py
@@ -103,8 +103,6 @@ def single_page_application():
             # The policy is still a boolean realm_dropdown action
             # Thus we display ALL realms
             realms = ",".join(get_realms())
-        if realms:
-            realms = "," + realms
 
     try:
         if is_remote_user_allowed(request):


Test patch:
Repository: privacyidea/privacyidea

Issue:
push token challenge in the database is not marked as successfully answered.
# What did you try to do?

I successfully authenticate with a push token. 
The Push Token App also confirms successful authentication.

# What outcome did you expect?

The column otp_valid in the challenge database should be set to one.

# What outcome did you experience?

The column otp_valid was not set to 1. It stayed at 0.

# Configuration

* privacyIDEA Version: 3.0
* Installation method: ubuntu
* mysql database

Obviously the otp_value is not saved to the database.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/pushtoken.py b/privacyidea/lib/tokens/pushtoken.py
--- a/privacyidea/lib/tokens/pushtoken.py
+++ b/privacyidea/lib/tokens/pushtoken.py
@@ -401,7 +401,7 @@ def api_endpoint(cls, request, g):
         serial = getParam(request.all_data, "serial", optional=False)
 
         if serial and "fbtoken" in request.all_data and "pubkey" in request.all_data:
-            # Do the 2nd step of the enrollment
+            log.debug("Do the 2nd step of the enrollment.")
             try:
                 token_obj = get_one_token(serial=serial,
                                           tokentype="push",
@@ -414,6 +414,7 @@ def api_endpoint(cls, request, g):
             details = token_obj.get_init_detail(init_detail_dict)
             result = True
         elif serial and "nonce" in request.all_data and "signature" in request.all_data:
+            log.debug("Handling the authentication response from the smartphone.")
             challenge = getParam(request.all_data, "nonce")
             serial = getParam(request.all_data, "serial")
             signature = getParam(request.all_data, "signature")
@@ -441,7 +442,9 @@ def api_endpoint(cls, request, g):
                                           padding.PKCS1v15(),
                                           hashes.SHA256())
                         # The signature was valid
+                        log.debug("Found matching challenge {0!s}.".format(chal))
                         chal.set_otp_status(True)
+                        chal.save()
                         result = True
                     except InvalidSignature as e:
                         pass


Test patch:
Repository: privacyidea/privacyidea

Issue:
Setting and then Clearing 'User Cache expiration in seconds' Breaks Authentication
# What did you try to do?

Clear/delete value for 'User Cache expiration in seconds' under System Config.

# What outcome did you expect?

Value reset to default of 0.

# What outcome did you experience?

Authentication breaks after value is cleared.  Error is logged to log file.

`
[2019-04-26 14:31:42,511][ERROR][privacyidea.lib.token:493] User information can not be retrieved: invalid literal for int() with base 10: ''
`

Changing UserCacheExpiration back to 0 directly in the config table in the pi database and restarting the Apache2 service resolves the issue.

# Configuration

* privacyIDEA Version: 3.0
* Installation method: ubuntu packages

## more details: 

* OS: Ubuntu 16.04LTS
* Webserver: Apache2
* Tokendatabase: MySQL



Discussion:
Thanks for the report.
I could reproduce this behavior, it looks like an incorrect value is written to the db.
The safest way is to assure, that sensible data is read here:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/usercache.py#L73

Resolving patch:
diff --git a/privacyidea/lib/usercache.py b/privacyidea/lib/usercache.py
--- a/privacyidea/lib/usercache.py
+++ b/privacyidea/lib/usercache.py
@@ -70,7 +70,12 @@ def get_cache_time():
     :return: UserCacheExpiration config value as a timedelta
     :rtype: timedelta
     """
-    seconds = int(get_from_config(EXPIRATION_SECONDS, '0'))
+    seconds = 0
+    try:
+        seconds = int(get_from_config(EXPIRATION_SECONDS, '0'))
+    except ValueError:
+        log.info(u"Non-Integer value stored in system config {0!s}".format(EXPIRATION_SECONDS))
+
     return datetime.timedelta(seconds=seconds)
 
 


Test patch:
diff --git a/tests/test_lib_usercache.py b/tests/test_lib_usercache.py
--- a/tests/test_lib_usercache.py
+++ b/tests/test_lib_usercache.py
@@ -71,8 +71,20 @@ def _delete_realm(self):
         delete_resolver(self.resolvername1)
 
     def test_00_set_config(self):
-        set_privacyidea_config(EXPIRATION_SECONDS, 600)
+        # Save wrong data in EXPIRATION_SECONDS
+        set_privacyidea_config(EXPIRATION_SECONDS, "wrong")
+        exp_delta = get_cache_time()
+        self.assertEqual(exp_delta, timedelta(seconds=0))
+        self.assertFalse(is_cache_enabled())
 
+        # Save empty data in EXPIRATION_SECONDS
+        set_privacyidea_config(EXPIRATION_SECONDS, "")
+        exp_delta = get_cache_time()
+        self.assertEqual(exp_delta, timedelta(seconds=0))
+        self.assertFalse(is_cache_enabled())
+
+        # Save real data in EXPIRATION_SECONDS
+        set_privacyidea_config(EXPIRATION_SECONDS, 600)
         exp_delta = get_cache_time()
         self.assertEqual(exp_delta, timedelta(seconds=600))
         self.assertTrue(is_cache_enabled())
Repository: privacyidea/privacyidea

Issue:
hide links in dropdown menu
If piExternalLinks is set to false, the links in the dropdown menu should also be hidden.
here:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/templates/menu.html#L129

compare here https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/templates/baseline.html#L17




Discussion:

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -17,7 +17,7 @@
 # built documents.
 #
 # The short X.Y version.
-version = '3.0'
+version = '3.0.1dev2'
 # The full version, including alpha/beta/rc tags.
 #release = '2.16dev5'
 release = version
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,7 @@
 import sys
 
 #VERSION="2.1dev4"
-VERSION="3.0"
+VERSION="3.0.1dev2"
 
 # Taken from kennethreitz/requests/setup.py
 package_directory = os.path.realpath(os.path.dirname(__file__))


Test patch:
Repository: privacyidea/privacyidea

Issue:
Make the title configurable
The title of the web page could be configurable.
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/static/templates/header.html#L27



Discussion:

Resolving patch:
diff --git a/privacyidea/webui/login.py b/privacyidea/webui/login.py
--- a/privacyidea/webui/login.py
+++ b/privacyidea/webui/login.py
@@ -77,6 +77,8 @@ def single_page_application():
     # Get the logo file
     logo = current_app.config.get("PI_LOGO", "privacyIDEA1.png")
     browser_lang = request.accept_languages.best_match(["en", "de", "de-DE"], default="en").split("-")[0]
+    # The page title can be configured in pi.cfg
+    page_title = current_app.config.get("PI_PAGE_TITLE", "privacyIDEA Authentication System")
     # check if login with REMOTE_USER is allowed.
     remote_user = ""
     password_reset = False
@@ -161,5 +163,6 @@ def single_page_application():
                            realms=realms,
                            external_links=external_links,
                            login_text=login_text,
-                           logo=logo)
+                           logo=logo,
+                           page_title=page_title)
 


Test patch:
Repository: privacyidea/privacyidea

Issue:
pskc export does not calculate mac
According to [RFC6030](https://tools.ietf.org/html/rfc6030#section-6.1.1) we need to calculate a MAC if an encryption algorithm without integrity checks is use (like `AES-128-CBC`).
We already create a MAC key and encrypt it accordingly but the MAC for the token secrets was not calculated.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/importotp.py b/privacyidea/lib/importotp.py
--- a/privacyidea/lib/importotp.py
+++ b/privacyidea/lib/importotp.py
@@ -46,6 +46,7 @@
 It is used for importing SafeNet (former Aladdin)
 XML files, that hold the OTP secrets for eToken PASS.
 '''
+import hmac
 import defusedxml.ElementTree as etree
 import re
 import binascii
@@ -55,7 +56,8 @@
 from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
 
 from privacyidea.lib.utils import (modhex_decode, modhex_encode,
-                                   hexlify_and_unicode, to_unicode, to_utf8)
+                                   hexlify_and_unicode, to_unicode, to_utf8,
+                                   b64encode_and_unicode)
 from privacyidea.lib.config import get_token_class
 from privacyidea.lib.log import log_with
 from privacyidea.lib.crypto import (aes_decrypt_b64, aes_encrypt_b64, geturandom)
@@ -647,6 +649,8 @@ def export_pskc(tokenobj_list, psk=None):
                 encrypted_otpkey = aes_encrypt_b64(psk, otpkey)
             else:
                 encrypted_otpkey = aes_encrypt_b64(psk, otpkey)
+            hm = hmac.new(key=mackey, msg=otpkey, digestmod='sha1')
+            mac_value = b64encode_and_unicode(hm.digest())
         except TypeError:
             # Some keys might be odd string length
             continue
@@ -672,7 +676,7 @@ def export_pskc(tokenobj_list, psk=None):
                              </xenc:CipherData>
                          </EncryptedValue>
                      </Secret>
-                     <ValueMAC>TODOmissing</ValueMAC>
+                     <ValueMAC>{value_mac}</ValueMAC>
                     <Time>
                         <PlainValue>0</PlainValue>
                     </Time>
@@ -690,7 +694,7 @@ def export_pskc(tokenobj_list, psk=None):
         </KeyPackage>""".format(serial=cgi.escape(serial), type=cgi.escape(type), otplen=otplen,
                                 issuer=cgi.escape(issuer), manufacturer=cgi.escape(manufacturer),
                                 counter=counter, timestep=timestep, encrypted_otpkey=encrypted_otpkey,
-                                timedrift=timedrift,
+                                timedrift=timedrift, value_mac=mac_value,
                                 suite=cgi.escape(suite)), "html.parser")
 
             soup.macmethod.insert_after(kp2)


Test patch:
Repository: privacyidea/privacyidea

Issue:
CSS cleanup
As mentioned in https://github.com/privacyidea/privacyidea/pull/1656/commits/34b97d179fab0979db412fc96affa2e22f8abfdc, we should change the way how we edit our css files.

An idea is to create a file for each region like menu, sidebar, content, etc. and put the changes where they belong to.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/tokens/u2ftoken.py b/privacyidea/lib/tokens/u2ftoken.py
--- a/privacyidea/lib/tokens/u2ftoken.py
+++ b/privacyidea/lib/tokens/u2ftoken.py
@@ -182,9 +182,9 @@
 
 """
 
-IMAGES = {"yubico": "static/css/FIDO-U2F-Security-Key-444x444.png",
-          "plug-up": "static/css/plugup.jpg",
-          "u2fzero.com": "static/css/u2fzero.png"}
+IMAGES = {"yubico": "static/img/FIDO-U2F-Security-Key-444x444.png",
+          "plug-up": "static/img/plugup.jpg",
+          "u2fzero.com": "static/img/u2fzero.png"}
 U2F_Version = "U2F_V2"
 
 log = logging.getLogger(__name__)


Test patch:
Repository: privacyidea/privacyidea

Issue:
Failed to enroll U2F token on latest Chrome
# What did you try to do?

Enroll U2F token on Chrome

# What outcome did you expect?

Being able to enroll U2F token on Chrome

# What outcome did you experience?

Failed to enroll U2F token on Chrome

# Configuration

* privacyIDEA Version: 2.23.5

* Installation method: from source


## more details: 

* OS: Amazon LInux

* Webserver: nginx

* Tokendatabase: postgresql


# Log file

U2F recently fails to enroll with error pop up: "Bad U2F Request / challenge must be base64url encoded"

My Chrome version is Version 74.0.3729.157 (Official Build) (64-bit)

I think this maybe related to this change in Chrome Crypto:
https://github.com/chromium/chromium/commit/ceb92979d1f3a77146ed68f8484636ce63073625#diff-bbd4efb6053a9f322d1ba351e036bd71

I am using PrivacyIDEA 2.23.5. Just want to make sure this is already fixed in 3.0?

Thank you and best regards,

Quynh Nguyen


Discussion:
@Mipronimo Isn't this something you talked about lately?
There is a problem with Google Chrome 72+ and u2f.
The login process works fine with Google Chrome. Only the enrollment fails.

To enroll the token, please use Firefox, but you'll need privacyIDEA 3.0 to do so.
Hi,

Since today, we have the same issue

PrivacyIdea Version : 2.23.2
Chrome Version :  74.0.3729.157 and 75.0.3770.80

By running Chrome with --disable-features=WebAuthenticationProxyCryptotoken it's working again but that not really a fix..

Resolving patch:
diff --git a/privacyidea/lib/tokens/u2ftoken.py b/privacyidea/lib/tokens/u2ftoken.py
--- a/privacyidea/lib/tokens/u2ftoken.py
+++ b/privacyidea/lib/tokens/u2ftoken.py
@@ -348,7 +348,7 @@ def get_init_detail(self, params=None, user=None):
             if not app_id:
                 raise TokenAdminError(_("You need to define the appId in the "
                                         "token config!"))
-            nonce = urlsafe_b64encode_and_unicode(geturandom(32))
+            nonce = url_encode(geturandom(32))
             response_detail = TokenClass.get_init_detail(self, params, user)
             register_request = {"version": U2F_Version,
                                 "challenge": nonce,


Test patch:
diff --git a/tests/test_lib_tokens_u2f.py b/tests/test_lib_tokens_u2f.py
--- a/tests/test_lib_tokens_u2f.py
+++ b/tests/test_lib_tokens_u2f.py
@@ -148,7 +148,7 @@ def test_01_create_token(self):
         version = registerRequest.get("version")
         self.assertEqual(version, "U2F_V2")
         challenge = registerRequest.get("challenge")
-        self.assertEqual(len(challenge), 44)
+        self.assertEqual(len(challenge), 43)
 
         # Init step 2
         token = init_token({"type": "u2f",
Repository: privacyidea/privacyidea

Issue:
TokenOwner migration fails with MySQL and non-ASCII UIDs
Suppose you are on privacyIDEA 2.23.5 with MySQL, have a LDAP resolver with ``uidtype=dn`` and a non-ASCII UID, like ``CN=Cornelius Kölbel,OU=devel,OU=frankfurt,DC=testfoo,DC=intranet``.

Then, the upgrade to 3.0.1 loses all token assignments, because the TokenOwner migration script fails:

```
Failed to migrate token assignment data!
'utf8' codec can't decode byte 0xf6 in position 14: invalid start byte
```

Unfortunately, this error causes the migration to lose *all* token assignments.

The problem is that flask-sqlalchemy implicitly sets ``charset=utf8`` as a connection option in case of mysql: 

https://github.com/pallets/flask-sqlalchemy/blob/419fa5ec65105866d3f0a1cf9ee6205dc69d6768/flask_sqlalchemy/__init__.py#L873

As Flask-Migrate/alembic do not use flask-sqlalchemy to create the SQL connection, the ``charset`` option is not set in case of the migration script, which causes the above error.


Discussion:
We can solve this be adding ``?charset=utf8`` to the SQLALCHEMY_URI in pi.cfg before running the schema upgrade. But how could we fix this in the migration script without changing the pi.cfg?


Resolving patch:
diff --git a/migrations/env.py b/migrations/env.py
--- a/migrations/env.py
+++ b/migrations/env.py
@@ -1,10 +1,12 @@
 from __future__ import with_statement
 from alembic import context
 from sqlalchemy import engine_from_config, pool
+from sqlalchemy.engine.url import make_url
 from logging.config import fileConfig
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
+
 config = context.config
 
 # Interpret the config file for Python logging.
@@ -16,7 +18,23 @@
 # from myapp import mymodel
 # target_metadata = mymodel.Base.metadata
 from flask import current_app
-config.set_main_option('sqlalchemy.url', current_app.config.get('SQLALCHEMY_DATABASE_URI'))
+
+
+def set_database_url(config):
+    url = current_app.config.get('SQLALCHEMY_DATABASE_URI')
+    try:
+        # In case of MySQL, add ``charset=utf8`` to the parameters (if no charset is set),
+        # because this is what Flask-SQLAlchemy does
+        if url.startswith("mysql"):
+            parsed_url = make_url(url)
+            parsed_url.query.setdefault("charset", "utf8")
+            url = str(parsed_url)
+    except Exception as exx:
+        print(u"Attempted to set charset=utf8 on connection, but failed: {}".format(exx))
+    config.set_main_option('sqlalchemy.url', url)
+
+
+set_database_url(config)
 target_metadata = current_app.extensions['migrate'].db.metadata
 
 # other values from the config, defined by the needs of env.py,


Test patch:
Repository: privacyidea/privacyidea

Issue:
Cleanup app config in sql audit
Some code cleanup.

The sqlaudit should use the ``self.config`` (which actualy contains the pi.cfg) and not the ``get_app_config_value``.

~~~~
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -45,7 +45,6 @@ from privacyidea.lib.pooling import get_engine
 from privacyidea.lib.utils import censor_connect_string
 from privacyidea.lib.lifecycle import register_finalizer
 from privacyidea.lib.utils import truncate_comma_list
-from privacyidea.lib.framework import get_app_config_value
 from sqlalchemy import MetaData, cast, String
 from sqlalchemy import asc, desc, and_, or_
 import datetime
@@ -91,7 +90,7 @@ class Audit(AuditBase):
         self.audit_data = {}
         self.sign_data = not self.config.get("PI_AUDIT_NO_SIGN")
         self.sign_object = None
-        self.verify_old_sig = get_app_config_value('PI_CHECK_OLD_SIGNATURES')
+        self.verify_old_sig = self.config.get('PI_CHECK_OLD_SIGNATURES')
         if self.sign_data:
             self.read_keys(self.config.get("PI_AUDIT_KEY_PUBLIC"),
                            self.config.get("PI_AUDIT_KEY_PRIVATE"))
~~~~


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -45,7 +45,6 @@
 from privacyidea.lib.utils import censor_connect_string
 from privacyidea.lib.lifecycle import register_finalizer
 from privacyidea.lib.utils import truncate_comma_list
-from privacyidea.lib.framework import get_app_config_value
 from sqlalchemy import MetaData, cast, String
 from sqlalchemy import asc, desc, and_, or_
 import datetime
@@ -91,7 +90,7 @@ def __init__(self, config=None):
         self.audit_data = {}
         self.sign_data = not self.config.get("PI_AUDIT_NO_SIGN")
         self.sign_object = None
-        self.verify_old_sig = get_app_config_value('PI_CHECK_OLD_SIGNATURES')
+        self.verify_old_sig = self.config.get('PI_CHECK_OLD_SIGNATURES')
         if self.sign_data:
             self.read_keys(self.config.get("PI_AUDIT_KEY_PUBLIC"),
                            self.config.get("PI_AUDIT_KEY_PRIVATE"))


Test patch: