Repository: privacyidea/privacyidea

Issue:
Allow to modify the truncated length of columns of pidea_audit
The lengths of the columns of the ``pidea_audit`` table are hardcoded in the ``audit_column_length`` dictionary in ``models.py``:

https://github.com/privacyidea/privacyidea/blob/96b3e18c63814b771242c1e42cd3ff7109e5b9db/privacyidea/models.py#L2555

If users want to increase the column lengths, they can manually alter the SQL table (see #525). However, if ``PI_AUDIT_SQL_TRUNCATE`` is True (as it is in new installations now), the audit class will truncate fields to the lengths specified in ``audit_column_length`` even if the actual column size might be greater. It would be nice if one could increase the truncated size without having to edit ``models.py``.

I think this could be fixed in two ways:
* We could allow users to overwrite parts of ``audit_column_length`` in ``pi.cfg`` and use these settings in ``models.py`` too. Then, if a user writes
  ```
  PI_AUDIT_COLUMN_LENGTHS = {"user": 40}
  ```
  in ``pi.cfg``, a ``pi-manage createdb`` would create an audit table with an enlarged ``user`` column, and also truncate values accordingly. This way might be tricky, because when we create the models, we do not have access to the Flask ``app.config`` because the Flask app doesn't exist yet. But there might be a way to circumvent that :-)
* Alternatively, we could require users to manually alter the column sizes, and only allow them to overwrite the truncated lengths, e.g.
    ```
    PI_AUDIT_SQL_TRUNCATE = True
    PI_AUDIT_SQL_TRUNCATED_LENGTHS = {"user": 40}
    ```
   which would cause the SQL audit module to truncate the ``user`` field to 40 characters, not 20.


Discussion:
I think we do not need to use the setting PI_AUDIT_COLUMN_LENGHT in ``pi-manage createdb``. Why? The DB is usually already created, when the user wants to enlarge the column.

So I am totally fine with the process of
1. User alters the column length manually.
2. Notifies privacyIDEA about the alternation.

But I am wondering if we can not determine by code within privacyidea, within the SQL Audit module, how long the column actually **is**.

I know that this is (in regards to the models definition) the wrong approach ;-)
We might also recommend to use another than the default table, if the user actually wants to modify the lenght?
I think you're right, we don't necessarily have to modify the column lengths already for ``createdb``.

By some quick googling, there doesn't seem to be a way to determine the actual column length without using commands that are database-specific, but I'll recheck.

Using another table would also be an option -- but currently I don't think there is a way to change the audit table name without changing ``models.py``?
> But I am wondering if we can not determine by code within privacyidea, within the SQL Audit module, how long the column actually **is**.

We could initialize the table with:
```
Table('pidea_audit', md, autoload=True, autoload_with=conn)
```
(found on [SO](https://stackoverflow.com/questions/38940682/how-can-i-get-column-name-and-type-from-an-existing-table-in-sqlalchemy))
But if we query the column type we get the SQL Type.

Ah, you're right, I missed this :)

This actually seems to work pretty nice with MariaDB at least:
```python
from sqlalchemy import create_engine, MetaData, Table, VARCHAR

conn = create_engine('mysql+pymysql://test:test@192.168.33.101/pi')
meta = MetaData(conn)

table = Table('pidea_audit', meta, autoload=True, autoload_with=conn)
for column in table.columns:
    if isinstance(column.type, VARCHAR):
        print(column.name, column.type, column.type.length)
    else:
        print(column.name, column.type)
```
prints:
```
id INTEGER(11)
date DATETIME
signature VARCHAR(620) 620
action VARCHAR(50) 50
success INTEGER(11)
serial VARCHAR(40) 40
token_type VARCHAR(12) 12
user VARCHAR(20) 20
realm VARCHAR(20) 20
resolver VARCHAR(50) 50
administrator VARCHAR(20) 20
action_detail VARCHAR(50) 50
info VARCHAR(50) 50
privacyidea_server VARCHAR(255) 255
client VARCHAR(50) 50
loglevel VARCHAR(12) 12
clearance_level VARCHAR(12) 12
policies VARCHAR(255) 255
```
I'll check with postgres
Does work the same with postgres.
That's nice!

So I guess the way to go would be to keep the field lengths in the model definition in ``models.py`` for table creation, but use the actual field lengths (as determined above) for truncation?
I think we will go with this suggestion:

PI_AUDIT_SQL_TRUNCATED_LENGTHS

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -33,6 +33,7 @@
     Optional:
     PI_AUDIT_SQL_URI = "sqlite://"
     PI_AUDIT_SQL_TRUNCATE = True | False
+    PI_AUDIT_SQL_COLUMN_LENGTH = {"user": 60, "info": 10 ...}
 
 If the PI_AUDIT_SQL_URI is omitted the Audit data is written to the
 token database.
@@ -86,7 +87,7 @@ class Audit(AuditBase):
     """
 
     is_readable = True
-    
+
     def __init__(self, config=None):
         super(Audit, self).__init__(config)
         self.name = "sqlaudit"
@@ -97,7 +98,11 @@ def __init__(self, config=None):
             self.read_keys(self.config.get("PI_AUDIT_KEY_PUBLIC"),
                            self.config.get("PI_AUDIT_KEY_PRIVATE"))
             self.sign_object = Sign(self.private, self.public)
-
+        # Read column_length from the config file
+        config_column_length = self.config.get("PI_AUDIT_SQL_COLUMN_LENGTH", {})
+        # fill the missing parts with the default from the models
+        self.custom_column_length = {k: (v if k not in config_column_length else config_column_length[k])
+                                     for k, v in column_length.items()}
         # We can use "sqlaudit" as the key because the SQLAudit connection
         # string is fixed for a running privacyIDEA instance.
         # In other words, we will not run into any problems with changing connect strings.
@@ -136,10 +141,10 @@ def _create_engine(self):
 
     def _truncate_data(self):
         """
-        Truncate self.audit_data according to the column_length.
+        Truncate self.audit_data according to the self.custom_column_length.
         :return: None
         """
-        for column, l in column_length.items():
+        for column, l in self.custom_column_length.items():
             if column in self.audit_data:
                 data = self.audit_data[column]
                 if isinstance(data, string_types):
@@ -207,10 +212,10 @@ def get_total(self, param, AND=True, display_error=True, timelimit=None):
         # if param contains search filters, we build the search filter
         # to only return the number of those entries
         filter_condition = self._create_filter(param, timelimit=timelimit)
-        
+
         try:
-            count = self.session.query(LogEntry.id)\
-                .filter(filter_condition)\
+            count = self.session.query(LogEntry.id) \
+                .filter(filter_condition) \
                 .count()
         finally:
             self.session.close()
@@ -222,7 +227,7 @@ def finalize_log(self):
         It should hash the data and do a hash chain and sign the data
         """
         try:
-            self.audit_data["policies"] = ",".join(self.audit_data.get("policies",[]))
+            self.audit_data["policies"] = ",".join(self.audit_data.get("policies", []))
             if self.config.get("PI_AUDIT_SQL_TRUNCATE"):
                 self._truncate_data()
             if "tokentype" in self.audit_data:
@@ -298,7 +303,7 @@ def _check_missing(self, audit_id):
         finally:
             # self.session.close()
             pass
-            
+
         return res
 
     @staticmethod
@@ -429,7 +434,7 @@ def search(self, search_dict, page_size=15, page=1, sortorder="asc",
                 log.debug("{0!s}".format(traceback.format_exc()))
 
         return paging_object
-        
+
     def search_query(self, search_dict, page_size=15, page=1, sortorder="asc",
                      sortname="number", timelimit=None):
         """
@@ -443,7 +448,7 @@ def search_query(self, search_dict, page_size=15, page=1, sortorder="asc",
         try:
             limit = int(page_size)
             offset = (int(page) - 1) * limit
-            
+
             # create filter condition
             filter_condition = self._create_filter(search_dict,
                                                    timelimit=timelimit)
@@ -458,7 +463,7 @@ def search_query(self, search_dict, page_size=15, page=1, sortorder="asc",
                     filter_condition).order_by(
                     asc(self._get_logentry_attribute("number"))).limit(
                     limit).offset(offset)
-                                         
+
         except Exception as exx:  # pragma: no cover
             log.error("exception {0!r}".format(exx))
             log.debug("{0!s}".format(traceback.format_exc()))
@@ -479,7 +484,7 @@ def clear(self):
         """
         self.session.query(LogEntry).delete()
         self.session.commit()
-    
+
     def audit_entry_to_dict(self, audit_entry):
         sig = None
         if self.sign_data:


Test patch:
diff --git a/tests/test_lib_audit.py b/tests/test_lib_audit.py
--- a/tests/test_lib_audit.py
+++ b/tests/test_lib_audit.py
@@ -306,6 +306,24 @@ def test_10_check_tokentype(self):
         self.assertEqual(audit_log.auditdata[0].get("token_type"), "spass")
 
 
+class AuditColumnLengthTestCase(OverrideConfigTestCase):
+    class Config(TestingConfig):
+        # this needs to exist on app creation
+        PI_AUDIT_SQL_COLUMN_LENGTH = {'user': 10}
+
+    def setUp(self):
+        self.Audit = getAudit(self.app.config)
+        self.Audit.clear()
+
+    def test_01_check_custom_column_lengths(self):
+        # Since changing the database during tests would be a hassle, we just
+        # reduce the custom length and check if it gets applied
+        self.Audit.log({"action": "test06", "tokentype": "spass",
+                        "user": "a_very_long_username@long_realm"})
+        self.Audit._truncate_data()
+        self.assertEqual(len(self.Audit.audit_data.get("user")), 10, self.Audit.audit_data)
+
+
 class AuditFileTestCase(OverrideConfigTestCase):
     class Config(TestingConfig):
         # this needs to exist on app creation