How does one download a data set from a file automatically with Pytorch?

username: pinocchio
cooked: <p>I want to download a dataset from a specific url to specific path.</p>
<p>I tried the following:</p>
<pre><code class="lang-auto">    from torchvision.datasets.utils import download_and_extract_archive

    ## download mini-imagenet
    url = 'https://drive.google.com/file/d/1rV3aj_hgfNTfCakffpPm7Vhpr1in87CR'
    filename = 'miniImagenet.tgz'
    root = '~/tmp/'
    download_and_extract_archive(url, root, filename)
</code></pre>
<p>but it didn’t work.</p>
<p>Why? How do we fix it?</p>
<p>Error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/me/pytorch_playground.py", line 79, in &lt;module&gt;
    download_mini_imagenet()
  File "/Users/me/pytorch_playground.py", line 72, in download_mini_imagenet
    download_and_extract_archive(url, root, filename)
  File "/Users/me/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 268, in download_and_extract_archive
    extract_archive(archive, extract_root, remove_finished)
  File "/Users/me/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 250, in extract_archive
    raise ValueError("Extraction of {} not supported".format(from_path))
ValueError: Extraction of /Users/me/tmp/1rV3aj_hgfNTfCakffpPm7Vhpr1in87CR not supported
</code></pre>
<hr>
<p>related gitissue: <a href="https://github.com/pytorch/vision/issues/1028" rel="nofollow noopener">https://github.com/pytorch/vision/issues/1028</a></p>

username: simaiden
cooked: <p>That’s because with google drive you can’t get a direct download link. Try with <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/utils.py#L136" rel="nofollow noopener">this</a>:</p>
<pre><code class="lang-auto"> from torchvision.datasets.utils import download_file_from_google_drive
</code></pre>
<p>and extract by yourself or adapt the code and use <code>torchvision.datasets.utils.extract_archive</code></p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="simaiden" data-post="2" data-topic="74262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simaiden/48/21043_2.png" class="avatar"> simaiden:</div>
<blockquote>
<p>extract by yourself</p>
</blockquote>
</aside>
<p>how do you do extract the contents? Is it dependent on the zip file format? any examples?</p>

username: pinocchio
cooked: <p>Do you know how to NOT download the file if the dataset has already been download?</p>
<p>I was reading:</p>
<pre><code class="lang-auto">    def _check_integrity(self):
        zip_filename = self._get_target_folder()
        if not check_integrity(join(self.root, zip_filename + '.zip'), self.zips_md5[zip_filename]):
            return False
        return True
</code></pre>
<p>and it doesn’t seem to be doable with mine because I do not have an md5 has for this dataset…</p>

username: pinocchio
cooked: <p>Temporary solution:</p>
<pre><code class="lang-auto">def download_and_extract_miniImagenet(root):
    import os
    from torchvision.datasets.utils import download_file_from_google_drive, extract_archive

    ## download miniImagenet
    #url = 'https://drive.google.com/file/d/1rV3aj_hgfNTfCakffpPm7Vhpr1in87CR'
    file_id = '1rV3aj_hgfNTfCakffpPm7Vhpr1in87CR'
    filename = 'miniImagenet.tgz'
    download_file_from_google_drive(file_id, root, filename)
    fpath = os.path.join(root, filename) # this is what download_file_from_google_drive does
    ## extract downloaded dataset
    from_path = os.path.expanduser(fpath)
    extract_archive(from_path)
</code></pre>

username: pinocchio
cooked: <p>I am having errors with this: <a href="https://discuss.pytorch.org/t/how-does-one-download-dataset-from-gdrive-using-pytorch/74636" class="inline-onebox">How does one download dataset from gdrive using pytorch?</a></p>