Only delete a file if it could be found