Only remove files when they really exist to cope with inconsistent states