Repository: privacyidea/privacyidea

Issue:
DB model incorrect for pidea_audit.success, causes auth failure due to underlying DB failure (type incompatibility)
<!--
Thank you for filing a bug within privacyIDEA.
Please read the following blog post.
https://www.privacyidea.org/getting-help/

The more information you provide, the easier it is to identify 
the problem!

Issues are not meant for asking questions!
If you simply wonder how things work and need some explanation 
please visit the community forum:

https://community.privacyidea.org

If you do not like the NO-WARRANTY of this software and need
reliable, professional help, please consider getting either
- consultancy 
  (https://netknights.it/en/leistungen/one-time-services/)
- or the privacyIDEA Enterprise Edition 
  (https://netknights.it/en/leistungen/service-level-agreements/)

If you still think, you should open an issue, please do so:
-->

# What did you try to do?

Authenticate via pam_privacyidea

# What outcome did you expect?

Successful authentication

# What outcome did you experience?
Login failed.

Error log:

ProgrammingError: (ProgrammingError) operator does not exist: integer = boolean
LINE 4: ...audit."user" = 'darthmdh' AND pidea_audit.success = false AND ...
HINT: No operator matches the given name and argument type(s).  You might need to add explicit type casts.

# Configuration

* privacyIDEA Version: 2.19.1

* Installation method:
Ubuntu packages (privacyidea PPA)

## more details: 

* OS: Linux

* Webserver: nginx

* Tokendatabase: Postgresql


# Other info
Since PostgreSQL is strongly typed, it appears the constant use of boolean types "true" and "false" with the pidea_audit.success value conflicts with its definition as an INTEGER field.  If its meant to be a BOOLEAN type then the field should be defined as such in the DB model.  This should work transparently with MySQL as that server aliases BOOLEAN to TINYINT internally so the definition is compatible.
Core problematic files seem to be:
lib/auditmodules/sqlaudit.py
models.py




Discussion:
Hm, interesting, I cannot reproduce this so far, authentication works for me using postgresql.
What version of postgresql are you running? Which SQLalchemy dialect are you using? (i.e. the part after ``postgresql+`` in the connect string)
Does this also happen if you authenticate using the REST API manually, e.g. ``curl 'http://privacyideaserver/validate/check?user=username&pass=password'``?
Sorry for the delay.
It was connecting to postgres 9.5.7 using psycopg2 dialect.  Unfortunately I can't live test it anymore as I replaced it with a clone using a local mysql instance instead to get over the hump, but I'll spin it up on a different IP (just need time!).  Unless someone else has +1'd this I'm happy to not make it a 2.20 item given I've worked around it.

Don't feel rushed. There is no *must* to make it a 2.20. Thanks!
Just to show, that we take it seriously to run on postgres.
We might remove it from the 2.20 milestone, I personally do not see it as a blocker, either! ;-)
Hi Cornelius,

I can reproduce this behavior in PrivacyIDEA 3.3.3 on Python3.

You'll get the error while both (1) using postgresql with psychopg2 dialect and (2) using an enabled authorization policy with auth_max_fail set e.g. to "10/12h". I suppose you'll get the error with any other authorization policy that is looking into the audit log.

With such a policy enabled, the admins/users then will get HTTP 500 errors after trying to login, and of course no successful authentication.

Here is a log entry of our new test system:

```
[2020-08-11 13:34:51,050][73605][140450041726848][ERROR][privacyidea.app:1891] Exception on /auth [POST]
Traceback (most recent call last):
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedFunction: operator does not exist: integer = boolean
LINE 4: ...n LIKE '%/validate/check' AND pidea_audit.success = false AN...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
cd 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/lib/prepolicy.py", line 151, in policy_wrapper
    return wrapped_function(*args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/lib/prepolicy.py", line 151, in policy_wrapper
    return wrapped_function(*args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/lib/prepolicy.py", line 151, in policy_wrapper
    return wrapped_function(*args, **kwds)
  [Previous line repeated 1 more time]
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/lib/postpolicy.py", line 107, in policy_wrapper
    response = wrapped_function(*args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/lib/postpolicy.py", line 107, in policy_wrapper
    response = wrapped_function(*args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/lib/postpolicy.py", line 107, in policy_wrapper
    response = wrapped_function(*args, **kwds)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/event.py", line 99, in event_wrapper
    f_result = func(*args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/api/auth.py", line 293, in get_auth_token
    superuser_realms)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 93, in policy_wrapper
    return self.decorator_function(wrapped_function, *args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 518, in login_mode
    return wrapped_function(*args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/auth.py", line 128, in check_webui_user
    check, details = check_user_pass(user_obj, password, options=options)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 93, in policy_wrapper
    return self.decorator_function(wrapped_function, *args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 186, in auth_cache
    res, reply_dict = wrapped_function(user_object, passw, options)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 93, in policy_wrapper
    return self.decorator_function(wrapped_function, *args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 254, in auth_user_does_not_exist
    return wrapped_function(user_object, passw, options)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 93, in policy_wrapper
    return self.decorator_function(wrapped_function, *args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 223, in auth_user_has_no_token
    return wrapped_function(user_object, passw, options)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 93, in policy_wrapper
    return self.decorator_function(wrapped_function, *args, **kwds)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/policydecorators.py", line 373, in auth_user_timelimit
    timedelta=tdelta)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/privacyidea/lib/auditmodules/sqlaudit.py", line 389, in get_count
    log_count = self.session.query(LogEntry).filter(filter_condition).count()
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3630, in count
    return self.from_self(col).scalar()
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3355, in scalar
    ret = self.one()
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3325, in one
    ret = self.one_or_none()
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3294, in one_or_none
    ret = list(self)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/privacyidea/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) operator does not exist: integer = boolean
LINE 4: ...n LIKE '%/validate/check' AND pidea_audit.success = false AN...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

[SQL: SELECT count(*) AS count_1 
FROM (SELECT pidea_audit.id AS pidea_audit_id, pidea_audit.date AS pidea_audit_date, pidea_audit.signature AS pidea_audit_signature, pidea_audit.action AS pidea_audit_action, pidea_audit.success AS pidea_audit_success, pidea_audit.serial AS pidea_audit_serial, pidea_audit.token_type AS pidea_audit_token_type, pidea_audit."user" AS pidea_audit_user, pidea_audit.realm AS pidea_audit_realm, pidea_audit.resolver AS pidea_audit_resolver, pidea_audit.administrator AS pidea_audit_administrator, pidea_audit.action_detail AS pidea_audit_action_detail, pidea_audit.info AS pidea_audit_info, pidea_audit.privacyidea_server AS pidea_audit_privacyidea_server, pidea_audit.client AS pidea_audit_client, pidea_audit.loglevel AS pidea_audit_loglevel, pidea_audit.clearance_level AS pidea_audit_clearance_level, pidea_audit.policies AS pidea_audit_policies 
FROM pidea_audit 
WHERE pidea_audit."user" = %(user_1)s AND pidea_audit.realm = %(realm_1)s AND pidea_audit.action LIKE %(action_1)s AND pidea_audit.success = false AND pidea_audit.date >= %(date_1)s) AS anon_1]
[parameters: {'user_1': 'testuser1', 'realm_1': 'testrealm1', 'action_1': '%/validate/check', 'date_1': datetime.datetime(2020, 8, 11, 1, 34, 51, 37998)}]
(Background on this error at: http://sqlalche.me/e/f405)
```

As I locked my admin account out of my test system, I was able to work around this via disabling the corresponding policy row in the policy table directly in the PostgreSQL database.

PS: Currently I am testing and extending our new PrivacyIDEA test system, so we can later Major-upgrade our 2FA servers from PrivacyIDEA 2 to PrivacyIDEA 3. Also now we are finally testing the His/HisInOne to PrivacyIDEA token migration (TAN lists) and the HisInOne PrivacyIDEA connector. First trials worked like a charm. As I am back in the 2FA business, you'll hear more from us in the next weeks. As soon as I have the development system running, there will also be time for pull requests. 

Best regards,

Bernd
Taking this to 3.5 to look into this.
In the case of @berndnicklas it happens in ``get_count``.

We should be able to simple cast all conditions of ``pidea_audit.success``  to an integer.

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -46,7 +46,7 @@
 from privacyidea.lib.pooling import get_engine
 from privacyidea.lib.utils import censor_connect_string
 from privacyidea.lib.lifecycle import register_finalizer
-from privacyidea.lib.utils import truncate_comma_list
+from privacyidea.lib.utils import truncate_comma_list, is_true
 from sqlalchemy import MetaData, cast, String
 from sqlalchemy import asc, desc, and_, or_
 import datetime
@@ -184,7 +184,7 @@ def _create_filter(param, timelimit=None):
                         # "success" is the only integer.
                         search_value = search_value.strip("*")
                         conditions.append(getattr(LogEntry, search_key) ==
-                                          int(search_value))
+                                          int(is_true(search_value)))
                     else:
                         # All other keys are compared as strings
                         column = getattr(LogEntry, search_key)
@@ -401,7 +401,7 @@ def get_count(self, search_dict, timedelta=None, success=None):
         filter_condition = self._create_filter(search_dict)
         conditions = [filter_condition]
         if success is not None:
-            conditions.append(LogEntry.success == success)
+            conditions.append(LogEntry.success == int(is_true(success)))
 
         if timedelta is not None:
             conditions.append(LogEntry.date >= datetime.datetime.now() -


Test patch: