about to make major changes to the processing pipeline for optimisation
and compatability with new database storage system.