Fix fatal errors when deleting files.

Reroll #5327