Repository: pytorch/vision

Issue:
Flaky behavior when downloading Google drive files
## 🐛 Bug

This is a rather difficult bug to diagnose because certain internet activity must be present. The issue is with `torhvision.datasets.utils.download_file_from_google_drive()`. It does not gracefully handle large files that have exceeded their daily download quota.

## To Reproduce
The following two prerequisites must be met in order to detect this issue.

* the quota for a google drive file must be exceeded for the day.
* file size should be relatively large (> 3 GB). It's not clear how file size impacts this behavior but I've experienced this behavior multiple times with large files.

```python
from torchvision.datasets.utils import *
# use the WIDERFACE training data file as an example
file_id = '0B6eKvaijfFUDQUUwd21EckhUbWs'
root = 'data_folder'
filename = 'WIDER_train.zip'
md5 = '3fedf70df600953d25982bcd13d91ba2'
download_file_from_google_drive(file_id, root, filename, md5)
```
will lead to the python session getting killed.

The python process hangs on the call to `torchvision.datasets.utils._quota_exceeded(...)`. My best guess is the code in this function is performing a string search that is either inefficient or causing python to search the entire data payload (resulting in a timeout).

```python
def _quota_exceeded(response: "requests.models.Response") -> bool:  # type: ignore[name-defined]
    return "Google Drive - Quota exceeded" in response.text
```


## Expected behavior
Calling `download_file_from_google_drive(...)` should not kill the session when download quota thresholds have been met on large files.

## Environment

```
PyTorch version: 1.8.0.dev20201021
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 8.4.0-3ubuntu2) 8.4.0
Clang version: 10.0.1-++20200708122807+ef32c611aa2-1~exp1~20200707223407.61 
CMake version: version 3.18.1

Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.1.105
GPU models and configuration: GPU 0: GeForce GTX 1650
Nvidia driver version: 450.80.02
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.8.0.dev20201021
[pip3] torchvision==0.9.0a0+9984146
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch-nightly
[conda] cudatoolkit               10.1.243             h6bb024c_0  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.1           py38hbc911f0_0  
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] pytorch                   1.8.0.dev20201021     py3.8_cpu_0  [cpuonly]  pytorch-nightly
[conda] torchvision               0.8.0a0+1fbd0b7          pypi_0    pypi
```

cc @pmeier


Discussion:
> causing python to search the entire data payload (resulting in a timeout).

IMO, this could be the offender. If I remember correctly, the `"Google Drive - Quota exceeded"` string is at the beginning of the the payload. Thus, we should switch to something like `response.iter_content` and check if the string is present.

@jgbradley1 Right now, https://docs.google.com/uc?export=download&id=0B6eKvaijfFUDQUUwd21EckhUbWs&confirm=o7Z2 (the URL we request from you sample above), does not trigger the daily quota exceeded. Could you ping me as fast as possible if you see this again, so I can take a look?
@pmeier on MiniImageNet I have this issue reproduced every time on my environment (Google Drive Daily quota shouldn't be exceeded)
Code:
```
from torchvision.datasets.utils import download_file_from_google_drive


if __name__ == '__main__':
    print("Start")
    folder = './miniimagenet'
    gdrive_id = '16V_ZlkW4SsnNDtnGmaBRq2OoPmUOc5mY'
    gz_filename = 'mini-imagenet.tar.gz'
    gz_md5 = 'b38f1eb4251fb9459ecc8e7febf9b2eb'

    download_file_from_google_drive(gdrive_id, folder, gz_filename, md5=gz_md5)
    print("Done")
```

This code downloads MiniImageNet dataset (the dataset itself is ~1Gb worth of data).
With `_quota_exceeded` as is currently implemented all the download is done in `"Google Drive - Quota exceeded" in response.text` line and no progress bar is shown. Then while the debugger still points to that line I seem to get a memory leak (more than 20Gb of RAM is used), then I just kill the process. The target file never appears on disk

If I comment out `_quota_exceeded`, I get immediately a progress bar and the file is downloaded.
The issue can be reproduced on both Windows 10 and Ubuntu 20.04 with PyTorch 1.7, TorchVision 0.8.0 and requests-2.25.0

@pmeier I can confirm I'm seeing the same behavior as @KhabarlakKonstantin for mini imagenet. At this time, the mini imagenet file has not exceeded the threshold.

The real issue here then is how we're checking for a quota exceeded error. A string search is just not going to be a viable solution for large files (1+ GB). According to the Google Drive API [docs](https://developers.google.com/drive/api/v3/handle-errors#resolve_a_403_error_daily_limit_exceeded), the http status code should be `403` when the quota threshold has truly been met.

Modifying `_quota_exceeded` to check the status code first could resolve the immediate issue (downloading large google drive files that have not met their download threshold). A `403` status is used for other reasons as well if you look through the documentation so we probably still want to leave the string search in for now (unless someone has a better suggestion).

```
def _quota_exceeded(response: "requests.models.Response") -> bool:  # type: ignore[name-defined]
    return (response.status_code == 403) and ("Google Drive - Quota exceeded" in response.text)
```

Using my modified function above, I am able to download mini imagenet and see the progress bar.
The mini imagenet file has now exceeded the threshold for today when downloading programatically. Here are some observations of what I can see.

* `response.status_code = 200`

Output of `print(response.text)` is
```
<!DOCTYPE html><html><head><title>Google Drive - Quota exceeded</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><link href=&#47;static&#47;doclist&#47;client&#47;css&#47;148676949&#45;untrustedcontent.css rel="stylesheet"><link rel="icon" href="//ssl.gstatic.com/images/branding/product/1x/drive_2020q4_32dp.png"/><style nonce="cpI7ZCLZSoij8xh5gvcKrw">#gbar,#guser{font-size:13px;padding-top:0px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}
</style><script nonce="cpI7ZCLZSoij8xh5gvcKrw"></script></head><body><div id=gbar><nobr><a target=_blank class=gb1 href="https://www.google.com/webhp?tab=ow">Search</a> <a target=_blank class=gb1 href="http://www.google.com/imghp?hl=en&tab=oi">Images</a> <a target=_blank class=gb1 href="https://maps.google.com/maps?hl=en&tab=ol">Maps</a> <a target=_blank class=gb1 href="https://play.google.com/?hl=en&tab=o8">Play</a> <a target=_blank class=gb1 href="https://www.youtube.com/?gl=US&tab=o1">YouTube</a> <a target=_blank class=gb1 href="https://news.google.com/?tab=on">News</a> <a target=_blank class=gb1 href="https://mail.google.com/mail/?tab=om">Gmail</a> <b class=gb1>Drive</b> <a target=_blank class=gb1 style="text-decoration:none" href="https://www.google.com/intl/en/about/products?tab=oh"><u>More</u> &raquo;</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a target="_self" href="/settings?hl=en_US" class=gb4>Settings</a> | <a target=_blank  href="//support.google.com/drive/?p=web_home&hl=en_US" class=gb4>Help</a> | <a target=_top id=gb_70 href="https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://docs.google.com/uc%3Fexport%3Ddownload%26id%3D16V_ZlkW4SsnNDtnGmaBRq2OoPmUOc5mY&service=writely&ec=GAZAMQ" class=gb4>Sign in</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div><div class="uc-main"><div id="uc-text"><p class="uc-error-caption">Sorry, you can&#39;t view or download this file at this time.</p><p class="uc-error-subcaption">Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.</p></div></div><div class="uc-footer"><hr class="uc-footer-divider">&copy; 2020 Google - <a class="goog-link" href="//support.google.com/drive/?p=web_home">Help</a> - <a class="goog-link" href="//support.google.com/drive/bin/answer.py?hl=en_US&amp;answer=2450387">Privacy & Terms</a></div></body></html>
```

I can still manually download the file from [here](https://drive.google.com/file/d/16V_ZlkW4SsnNDtnGmaBRq2OoPmUOc5mY/view) using a browser. Perhaps Google Drive allows more _browser-initiated downloads_ than programmatic-initiated downloads.

The most interesting behavior about this is the reported status code will switch randomly between `403` and `200`. It is not always consistent with the GDrive API documentation.

Final conclusion: the function `_quota_exceeded` works great only after a file has exceeded the daily quota threshold.
@jgbradley1 

> A string search is just not going to be a viable solution for large files (1+ GB)

True. I didn't think of this when I implemented the check. We should use `response.iter_content` to get the first chunk and only search in there. I'm not sure if this interferes with the saving of the file if the quota is not exceeded. I'm on it.

> the http status code should be `403` when the quota threshold has truly been met

This was the first thing I checked and I always got 200 back. I only went for checking the response content after that.

> the function `_quota_exceeded` works great only after a file has exceeded the daily quota threshold.

True again, but since it makes normal operation impossible, this is no solution. I'm going to revert the commit for now and send a proper fix later.

Resolving patch:
diff --git a/torchvision/datasets/utils.py b/torchvision/datasets/utils.py
--- a/torchvision/datasets/utils.py
+++ b/torchvision/datasets/utils.py
@@ -120,7 +120,9 @@ def list_files(root: str, suffix: str, prefix: bool = False) -> List[str]:
 
 
 def _quota_exceeded(response: "requests.models.Response") -> bool:  # type: ignore[name-defined]
-    return "Google Drive - Quota exceeded" in response.text
+    return False
+    # See https://github.com/pytorch/vision/issues/2992 for details
+    # return "Google Drive - Quota exceeded" in response.text
 
 
 def download_file_from_google_drive(file_id: str, root: str, filename: Optional[str] = None, md5: Optional[str] = None):


Test patch: