Repository: pytorch/vision

Issue:
Invalid hash error on ImageNet dataset
In my environment, md5 value of `meta.bin` used in ImageNet dataset is different from the value defined in `imagenet.py`.

`meta.bin` is generated by `torch.save` in the code. I found python2 and3 generate different files.

md5sum hashes are as follows.

- (defined) `7e0d3cf156177e4fc47011cdd30ce706`
- (Python 2.7.16, Ubuntu) `a36fd93cf3900286d99e24ad0a73ce04`
- (Python 3.7.3, Ubuntu) `ca981e8aac175178e80e7949d90ee85c`

https://github.com/pytorch/vision/blob/9a481d0bec2700763a799ff148fe2e083b575441/torchvision/datasets/imagenet.py#L23-L26

https://github.com/pytorch/vision/blob/9a481d0bec2700763a799ff148fe2e083b575441/torchvision/datasets/imagenet.py#L117-L118


Discussion:
cc @pmeier 

This might be due to the fact that the information in `meta` is a python dict, which doesn't guarantee any order of how the keys are stored.
So I think we should not have just different hashes per python version, because it could also be system-dependent.

Maybe the simplest is to remove the `md5` checking from `meta.bin`?
> So I think we should not have just different hashes per python version, because it could also be system-dependent.

Agreed.

> This might be due to the fact that the information in `meta` is a python dict, which doesn't guarantee any order of how the keys are stored.

I will try if we get different hashes if we use an `OrderedDict`. If that is still the case:

> [...] remove the `md5` checking from `meta.bin`

Using `OrderedDict`s does *not* fix this. I'll send a fix soon. 
@fmassa Is the logic in `check_integrity` correct?

https://github.com/pytorch/vision/blob/6716fc514c9524abed4f8ca00e4424553990e315/torchvision/datasets/utils.py#L20-L24

Shouldn't we check if the `fpath` exist regardless of the `md5` check? Otherwise this function does no checking at all if `md5=None`.

Resolving patch:
diff --git a/torchvision/datasets/imagenet.py b/torchvision/datasets/imagenet.py
--- a/torchvision/datasets/imagenet.py
+++ b/torchvision/datasets/imagenet.py
@@ -20,11 +20,6 @@
     }
 }
 
-META_DICT = {
-    'filename': 'meta.bin',
-    'md5': '7e0d3cf156177e4fc47011cdd30ce706',
-}
-
 
 class ImageNet(ImageFolder):
     """`ImageNet <http://image-net.org/>`_ 2012 Classification Dataset.
@@ -70,7 +65,7 @@ def __init__(self, root, split='train', download=False, **kwargs):
                              for cls in clss}
 
     def download(self):
-        if not self._check_meta_file_integrity():
+        if not check_integrity(self.meta_file):
             tmpdir = os.path.join(self.root, 'tmp')
 
             archive_dict = ARCHIVE_DICT['devkit']
@@ -102,13 +97,10 @@ def download(self):
 
     @property
     def meta_file(self):
-        return os.path.join(self.root, META_DICT['filename'])
-
-    def _check_meta_file_integrity(self):
-        return check_integrity(self.meta_file, META_DICT['md5'])
+        return os.path.join(self.root, 'meta.bin')
 
     def _load_meta_file(self):
-        if self._check_meta_file_integrity():
+        if check_integrity(self.meta_file):
             return torch.load(self.meta_file)
         else:
             raise RuntimeError("Meta file not found or corrupted.",


Test patch: