opened
dfaller
username_0
There appear to be performance issues with the source_file and product_file tables due to them being split from the main scale_file table. If a query filters on columns from the scale_file table and source/product table, this can lead to inefficient queries due to matching up the two table in a large join with many rows. Design a solution around the issue.

One possibility is to move everything into one table (scale_file). This would mean that scale_file would contain fields relevant only to source files (e.g. is_parsed) and products (e.g. job). There would be a new field indicating the "type" of file (source or product). If the type is source, the source fields would be used, and vice versa. Multiple column indexes could then be used with the type field first so you could quickly retrieve the latest source files or latest products. I think this approach is the easiest in terms of the database structure, but might require some ugly Scale code changes. Also, the data migration required would probably be quite expensive.
Investigate performance of source and product files
issue
closed
Fizz11
username_1
issue