Do a commit during the processing instead of just at the end. An attempt to limit memory consumption