polish the logic to retry files that did not go through until all files have been processed.