Repository: pytorch/vision

Issue:
`zipfile.BadZipFile` exception when downloading WIDERFace (traced to `download_file_from_google_drive` function)
### 🐛 Describe the bug

While trying to download the WIDERFace dataset using the following code:

```python
from torchvision.datasets.widerface import WIDERFace
w = WIDERFace('.', split='train', download=True)
```

I ran into the following error

```none
$ python download_wider.py
Traceback (most recent call last):
  File "download_wider.py", line 2, in <module>
    w = WIDERFace('.', split='train', download=True)
  File "/home/josh/venv/lib/python3.7/site-packages/torchvision/datasets/widerface.py", line 72, in __init__
    self.download()
  File "/home/josh/venv/lib/python3.7/site-packages/torchvision/datasets/widerface.py", line 191, in download
    extract_archive(filepath)
  File "/home/josh/venv/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 409, in extract_archive
    extractor(from_path, to_path, compression)
  File "/home/josh/venv/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 284, in _extract_zip
    from_path, "r", compression=_ZIP_COMPRESSION_MAP[compression] if compression else zipfile.ZIP_STORED
  File "/usr/lib/python3.7/zipfile.py", line 1225, in __init__
    self._RealGetContents()
  File "/usr/lib/python3.7/zipfile.py", line 1292, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
```

Stepping through the code I found that when it reaches [torchvision/datasets/utils.py:243](https://github.com/pytorch/vision/blob/60132302b5db75477325e8fdbd33f5a33b8cf0a5/torchvision/datasets/utils.py#L243):

```python
243:        _save_response_content(itertools.chain((first_chunk,), response_content_generator), fpath)
```

the value of `first_chunk` contains:

```python
b'<!DOCTYPE html><html><head><title>Google Drive - Virus scan warning</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><style nonce="lakwxKwRcFErCkEI/ksjXg">/* Copyright 2022 Google Inc. All Rights Reserved. */\n.goog-inline-block{position:relative;display:-moz-inline-box;display:inline-block}* html .goog-inline-block,*:first-child+html .goog-inline-block{display:inline}.goog-link-button{position:relative;color:#15c;text-decoration:underline;cursor:pointer}.goog-link-button-disabled{color:#ccc;text-decoration:none;cursor:default}body{color:#222;font:normal 13px/1.4 arial,sans-serif;margin:0}.grecaptcha-badge{visibility:hidden}.uc-main{padding-top:50px;text-align:center}#uc-dl-icon{display:inline-block;margin-top:16px;padding-right:1em;vertical-align:top}#uc-text{display:inline-block;max-width:68ex;text-align:left}.uc-error-caption,.uc-warning-caption{color:#222;font-size:16px}#uc-download-link{text-decoration:none}.uc-name-size a{color:#15c;text-decoration:none}.uc-name-size a:visited{color:#61c;text-decoration:none}.uc-name-size a:active{color:#d14836;text-decoration:none}.uc-footer{color:#777;font-size:11px;padding-bottom:5ex;padding-top:5ex;text-align:center}.uc-footer a{color:#15c}.uc-footer a:visited{color:#61c}.uc-footer a:active{color:#d14836}.uc-footer-divider{color:#ccc;width:100%}</style><link rel="icon" href="null"/></head><body><div class="uc-main"><div id="uc-dl-icon" class="image-container"><div class="drive-sprite-aux-download-file"></div></div><div id="uc-text"><p class="uc-warning-caption">Google Drive can\'t scan this file for viruses.</p><p class="uc-warning-subcaption"><span class="uc-name-size"><a href="/open?id=15hGDLhsx8bLgLcIRD5DhYt5iBxnjNF1M">WIDER_train.zip</a> (1.4G)</span> is too large for Google to scan for viruses. Would you still like to download this file?</p><form id="downloadForm" action="https://docs.google.com/uc?export=download&amp;id=15hGDLhsx8bLgLcIRD5DhYt5iBxnjNF1M&amp;confirm=t" method="post"><input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/></form></div></div><div class="uc-footer"><hr class="uc-footer-divider"></div></body></html>'
```

and this is the only chunk in the `response_content_generator` as well.

This information is then written to the `WIDER_train.zip` file (which is supposed to be a zip file) and that causes the `BadZipFile` exception.

---

I'm not sure if this is the result of a recent change in google drive or related to my platform, or what. Either way it would be really nice if a more meaningful error could be presented on the `BadZipFile` file exception. Since this relies on the response from google drive, which is likely to change in the future, wouldn't it be nicer to have a more meaningful error message when things like this happen?

For example, wouldn't it be great if we had an error message that said something meaningful like

```
Oops, we tried to download the google drive file from

https://docs.google.com/uc?export=download&id=15hGDLhsx8bLgLcIRD5DhYt5iBxnjNF1M

but the file contains unexpected information and we don't know why.

You can download this file manually and place it at ./widerface/WIDER_train.zip to circumvent this issue.
```

rather than having the user step through the code to find the correct URL and destination path?

Also, why is this being unzipped before the hash is even checked? That seems like a potential security issue.

### Versions

```
Collecting environment information...
PyTorch version: 1.11.0+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.13.4
Libc version: glibc-2.26

Python version: 3.7.5 (default, Dec  9 2021, 17:04:37)  [GCC 8.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-99-generic-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 2080 Ti
GPU 1: NVIDIA GeForce RTX 2080 Ti

Nvidia driver version: 510.47.03
cuDNN version: 8.0.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.11.0+cpu
[pip3] torchaudio==0.11.0+cpu
[pip3] torchvision==0.12.0+cpu
[conda] Could not collect
```

cc @pmeier


Discussion:
Thanks for the report @josh-gleason. GDrive is just a pain in the behind to deal with. Since they don't have a proper API, we need to manually parse the response to find if anything is going wrong. For example, we have a proper error in case the daily quota for a file is exceeded:

https://github.com/pytorch/vision/blob/297e2b8746d32b46d20b0e6a6f867fde65e50e8d/torchvision/datasets/utils.py#L187-L191

I've never encountered the "Virus scan warning". Giving a proper error message doesn't seem hard here. I'll look into it.
Yeah I've run into my own problems automating Google drive downloads. Thanks for looking into this.

Unrelated, but how do you post those nice code blocks in github markdown?

Resolving patch:
diff --git a/torchvision/datasets/utils.py b/torchvision/datasets/utils.py
--- a/torchvision/datasets/utils.py
+++ b/torchvision/datasets/utils.py
@@ -11,6 +11,7 @@
 import urllib
 import urllib.error
 import urllib.request
+import warnings
 import zipfile
 from typing import Any, Callable, List, Iterable, Optional, TypeVar, Dict, IO, Tuple, Iterator
 from urllib.parse import urlparse
@@ -24,22 +25,31 @@
     _is_remote_location_available,
 )
 
-
 USER_AGENT = "pytorch/vision"
 
 
-def _urlretrieve(url: str, filename: str, chunk_size: int = 1024) -> None:
-    with open(filename, "wb") as fh:
-        with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": USER_AGENT})) as response:
-            with tqdm(total=response.length) as pbar:
-                for chunk in iter(lambda: response.read(chunk_size), ""):
-                    if not chunk:
-                        break
-                    pbar.update(chunk_size)
-                    fh.write(chunk)
+def _save_response_content(
+    content: Iterator[bytes],
+    destination: str,
+    length: Optional[int] = None,
+) -> None:
+    with open(destination, "wb") as fh, tqdm(total=length) as pbar:
+        for chunk in content:
+            # filter out keep-alive new chunks
+            if not chunk:
+                continue
+
+            fh.write(chunk)
+            pbar.update(len(chunk))
+
+
+def _urlretrieve(url: str, filename: str, chunk_size: int = 1024 * 32) -> None:
+    with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": USER_AGENT})) as response:
+        _save_response_content(iter(lambda: response.read(chunk_size), b""), filename, length=response.length)
 
 
 def gen_bar_updater() -> Callable[[int, int, int], None]:
+    warnings.warn("The function `gen_bar_update` is deprecated since 0.13 and will be removed in 0.15.")
     pbar = tqdm(total=None)
 
     def bar_update(count, block_size, total_size):
@@ -184,11 +194,20 @@ def list_files(root: str, suffix: str, prefix: bool = False) -> List[str]:
     return files
 
 
-def _quota_exceeded(first_chunk: bytes) -> bool:
+def _extract_gdrive_api_response(response, chunk_size: int = 32 * 1024) -> Tuple[bytes, Iterator[bytes]]:
+    content = response.iter_content(chunk_size)
+    first_chunk = None
+    # filter out keep-alive new chunks
+    while not first_chunk:
+        first_chunk = next(content)
+    content = itertools.chain([first_chunk], content)
+
     try:
-        return "Google Drive - Quota exceeded" in first_chunk.decode()
+        match = re.search("<title>Google Drive - (?P<api_response>.+?)</title>", first_chunk.decode())
+        api_response = match["api_response"] if match is not None else None
     except UnicodeDecodeError:
-        return False
+        api_response = None
+    return api_response, content
 
 
 def download_file_from_google_drive(file_id: str, root: str, filename: Optional[str] = None, md5: Optional[str] = None):
@@ -202,8 +221,6 @@ def download_file_from_google_drive(file_id: str, root: str, filename: Optional[
     """
     # Based on https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url
 
-    url = "https://docs.google.com/uc?export=download"
-
     root = os.path.expanduser(root)
     if not filename:
         filename = file_id
@@ -211,61 +228,34 @@ def download_file_from_google_drive(file_id: str, root: str, filename: Optional[
 
     os.makedirs(root, exist_ok=True)
 
-    if os.path.isfile(fpath) and check_integrity(fpath, md5):
-        print("Using downloaded and verified file: " + fpath)
-    else:
-        session = requests.Session()
-
-        response = session.get(url, params={"id": file_id}, stream=True)
-        token = _get_confirm_token(response)
-
-        if token:
-            params = {"id": file_id, "confirm": token}
-            response = session.get(url, params=params, stream=True)
-
-        # Ideally, one would use response.status_code to check for quota limits, but google drive is not consistent
-        # with their own API, refer https://github.com/pytorch/vision/issues/2992#issuecomment-730614517.
-        # Should this be fixed at some place in future, one could refactor the following to no longer rely on decoding
-        # the first_chunk of the payload
-        response_content_generator = response.iter_content(32768)
-        first_chunk = None
-        while not first_chunk:  # filter out keep-alive new chunks
-            first_chunk = next(response_content_generator)
-
-        if _quota_exceeded(first_chunk):
-            msg = (
-                f"The daily quota of the file {filename} is exceeded and it "
-                f"can't be downloaded. This is a limitation of Google Drive "
-                f"and can only be overcome by trying again later."
-            )
-            raise RuntimeError(msg)
-
-        _save_response_content(itertools.chain((first_chunk,), response_content_generator), fpath)
-        response.close()
+    if check_integrity(fpath, md5):
+        print(f"Using downloaded {'and verified ' if md5 else ''}file: {fpath}")
 
+    url = "https://drive.google.com/uc"
+    params = dict(id=file_id, export="download")
+    with requests.Session() as session:
+        response = session.get(url, params=params, stream=True)
 
-def _get_confirm_token(response: requests.models.Response) -> Optional[str]:
-    for key, value in response.cookies.items():
-        if key.startswith("download_warning"):
-            return value
+        for key, value in response.cookies.items():
+            if key.startswith("download_warning"):
+                token = value
+                break
+        else:
+            api_response, content = _extract_gdrive_api_response(response)
+            token = "t" if api_response == "Virus scan warning" else None
 
-    return None
+        if token is not None:
+            response = session.get(url, params=dict(params, confirm=token), stream=True)
+            api_response, content = _extract_gdrive_api_response(response)
 
+        if api_response == "Quota exceeded":
+            raise RuntimeError(
+                f"The daily quota of the file {filename} is exceeded and it "
+                f"can't be downloaded. This is a limitation of Google Drive "
+                f"and can only be overcome by trying again later."
+            )
 
-def _save_response_content(
-    response_gen: Iterator[bytes],
-    destination: str,
-) -> None:
-    with open(destination, "wb") as f:
-        pbar = tqdm(total=None)
-        progress = 0
-
-        for chunk in response_gen:
-            if chunk:  # filter out keep-alive new chunks
-                f.write(chunk)
-                progress += len(chunk)
-                pbar.update(progress - pbar.n)
-        pbar.close()
+        _save_response_content(content, fpath)
 
 
 def _extract_tar(from_path: str, to_path: str, compression: Optional[str]) -> None:


Test patch: