Only delete a file if it exists. This way we won't through IOExceptions when we don't actually care if the file exists or not.