Added postgres hooks to use that instead of SQLite due to the amount of work being done. After running a test with the full log it was discovered that the duplication checks were putting massive amounts of overhead into the processing. I have disabled this as to reduce the overhead we will do deduplication via a stored proc or something similar after the files are loaded. Tweaked the dev config to make logging be done at an info level. The debug level just puts out soooo much shit I don't need in normal dev processes. Since we're now loading up all the lines in one go I've disabled the file splitting. I have also added in a try catch block to avoid the problem of a file processing being crashed part way through. Instead the line is logged and we proceed to the next line. This way we can be assured that when a file is beginning processing that it will complete.