Repository: pytorch/vision

Issue:
INaturalist download is broken for version="2021_train"
*Originally reported in https://discuss.pytorch.org/t/inaturalist-download-crashes-towards-the-end/149006*

> *INaturalist* download proceeds fine up to the end, then crashes with this error:
> 
> ```py
> File "/scratch/user/myenvs/env/lib/python3.8/site-packages/torchvision/datasets/utils.py", line 152, in download_url
>     raise RuntimeError("File not found or corrupted.")
> ```
> Using torchvision automatic downloading of the `2021_train` dataset.

---

The error points to

https://github.com/pytorch/vision/blob/05eae32f9663bbecad10a8d367ccbec50130e2f5/torchvision/datasets/utils.py#L151-L152

So my guess is that the MD5 checksum that we have on record is wrong:

https://github.com/pytorch/vision/blob/05eae32f9663bbecad10a8d367ccbec50130e2f5/torchvision/datasets/inaturalist.py#L24

I'll download and check. Given that this archive is > 200GB, this will take a while. I've tried with `version="2021_valid"` and this works fine. Thus, I'm guessing this is not a systematic problem.

cc @pmeier @YosuaMichael


Discussion:
The checksum that I have computed after download

```
e0526d53c7f7b2e3167b2b43bb2690ed
```

differs from the one we have on record. I'll confirm with the user in the forum.
Any update on this?
@slerman12 Could you run 

```shell
$ md5sum train.tar.gz
```

for the file you downloaded and compare the output to https://github.com/pytorch/vision/issues/5817#issuecomment-1099448162?

If you don't have `md5sum` available, you can also use this Python snippet:

```py
import hashlib

def md5sum(path, chunk_size=1024 * 1024):
    checksum = hashlib.md5()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(chunk_size), b""):
            checksum.update(chunk)
    print(checksum.hexdigest())

md5sum("train.tar.gz")
```

If you get the same checksum as I do, I'll update the one on record. 

---

In any case, this will not help you until the next release (unless you are working with nightly builds or compiling from source). The only part that is missing on your side is the archive extraction. Since the checksum verification only happens after download, we can effectively skip it if the integrity check of the dataset passes:

https://github.com/pytorch/vision/blob/53c853282697819df2d756f5d30911fb9cf20cf3/torchvision/datasets/inaturalist.py#L80-L84

We only have to do the following

```shell
$ # same value as you passed to INaturlist(...)
$ ROOT=...
cd $ROOT
mkdir -p 2021_train
tar -xvf train.tar.gz -C 2021_train
```

Afterwards you just load the dataset from Python like you are used to, but drop the `download=True`. With that everything should be working fine. Let me know if it doesn't.
Yes, the output is `e0526d53c7f7b2e3167b2b43bb2690ed`.

Resolving patch:
diff --git a/torchvision/datasets/inaturalist.py b/torchvision/datasets/inaturalist.py
--- a/torchvision/datasets/inaturalist.py
+++ b/torchvision/datasets/inaturalist.py
@@ -22,7 +22,7 @@
     "2017": "7c784ea5e424efaec655bd392f87301f",
     "2018": "b1c6952ce38f31868cc50ea72d066cc3",
     "2019": "c60a6e2962c9b8ccbd458d12c8582644",
-    "2021_train": "38a7bb733f7a09214d44293460ec0021",
+    "2021_train": "e0526d53c7f7b2e3167b2b43bb2690ed",
     "2021_train_mini": "db6ed8330e634445efc8fec83ae81442",
     "2021_valid": "f6f6e0e242e3d4c9569ba56400938afc",
 }


Test patch: