Started implementing dataset creation on a postgresql database instead of a pickled python object. This should let you create the dataset in batches (since it takes so long) and also avoids the issue of handling massive amounts of data going into RAM!
Seems to be working as far as creating the dataset in PostgreSQL using a specified set of files.
Improved skipping if file has already been processed.
Appears to be working of the database now... Need to check against the previous disk version to confirm.