Made it use workers and commit periodically in the hope of minimising memory usage.