Only remove the tempfile if it exists