Added double check that all partition data has been processed before ending