Project: talendesb
Issue TESB-11187: EventLogging: Improve database structure of Event Logging SQL database
Just like the SAM database structure the Event logging structure is far away from being optimal. Considering the possibly _large_ number of records the may end up inside the SQL tables esp. the primary keys / index definitions should be reconsidered. Note that this discussion is based on the default DDL (create.sql / create_ind.sql - appears to be for Derby). * Using a UUID as primary key appears to be a not extremely good idea. ** While it can be argued that by definition the UUID is pretty unique, it still doe _not_ not come with a guarantee to be (http://tinyurl.com/6z94fcv ). Apart from the fact that the low probability does not say anything about _when_ the algorithm will produce a duplicate - it just how often duplicates statistically arise number of calls. ( Black swans do exist - even though statistically they appear to be non-existent ;-) ) ** When considering that the DDL creates the ID field (regardless of whether is should be a primary key or not!) as a {code}ID VARCHAR(255) NOT NULL{code}, one should note that for today's 128 bit UUIDs a field length of 36 characters would be sufficient. Note that databases - on the physical page level - are rather unhappy with variable length keys. With fixed length keys there are much more efficient when maintaining index / table spaces - esp. finding out, on which physical page a record / index entry can be found. ** From perspective of the application, it can be fine to have this UUID as a column (if sized correctly), but it should not be a primary key, because - even with the length shortened as explained above, it will unnecessarily blow up the index space of the database. Instead, a serial (incrementable) number (like in SAM DB) would be a much better choice, because *** unique key generation can be made safely inside a database transaction and will be extremely cheap - esp. when considering that with batch inserts the software could obtain the keys for all records in the insert batch with one minimal operation (e.g. when having to insert 20 records we just need to increment the sequence field by 20 once, close the transaction, assign the keys and insert our batch while in parallel another request would be save to allocate new IDs for the batch of records it tries to insert - so even multiple Event Logging server processes could work efficiently in parallel using the same database). *** it takes much less space (and we are talking about a database where we have to expect large amounts of records) in the primary table *EVENTLOG* and the secondary table *EVENTLOG_CUSTOMINFO* *** while we do not have a strict sequence order in all case, the records are still _pretty_ much in sequence order - which could simplify garbage collection (recurring clean up of tables which is put on the customer's shoulders for now ...) and increase the chance of database internal cache hits when sequentially processing records. * Creating primary keys for secondary table *EVENTLOG_CUSTOMINFO* is unnecessarily inefficient (see also TESB-9724 !). ** We should not require a sequence field here! Instead we should use a fixed length composite key of EVENT_ID (assuming that it's "just" a number) and a short _local_ sequence number we can generate on the fly when inserting the custom properties (as we know how many custom properties come with the EVENT record we just assign numbers from 1.. ). ** Consequently the sub ID field does not have to be long (we save space on the primary index). Using this composite key will also increase the probability that the database stores the dependent records for the custom info physically in the same or a nearby located database segment - so the chances of a database cache hit is increased. ** With that kind of primary key, there's no need to add and maintain the additional index needed for joining *EVENTLOG* and *EVENTLOG_CUSTOMINFO* records - which again saves database space and provides more performance. So, please, please, remove this technical dept from Event Logging. For SAM it's already much harder than here, because SAM is already out with the customers and any schema change would require a data migration for customer data. With event logging we could avoid this from the start.
Story points: 2