Using a Google Cloud Storage bucket for dataset

cool_cheetah1: <p>I’ve been stuck with this for a while now, and I’m certain this is a problem people have dealt with in the past.</p>
<p>So I’m trying to use torch dataloaders on a Google Cloud VM with a GPU attached. I want to run training on ImageNet. So my options are to either get a persistent disk and download ImageNet onto it, or otherwise download ImageNet onto a GCS bucket.</p>
<p>If I were to go the bucket route, does anyone know what the best way to then have this interface with a torch dataloader would be? I’ve looked into WebDataset as well, and it seems cool - but I’d likely have to pass in URLs (which isn’t necessarily a problem).</p>
<p>Long story short, I’m not sure if I should be mounting the bucket, or if things “just work” if I were to use WebDataset, and overall just what the best/most economical path is going forward.</p>
<p>Thanks very much!</p>

ejguan: <p>You may checkout TorchData library we just released. We do have a DataPipe that takes URL from Google drive <a href="https://github.com/pytorch/data/blob/652986b14e893e08edaea2c519e21bc61706de5c/torchdata/datapipes/iter/load/online.py#L93" class="inline-onebox" rel="noopener nofollow ugc">data/online.py at 652986b14e893e08edaea2c519e21bc61706de5c · pytorch/data · GitHub</a><br>
Let us know if this covers your request</p>

Nilabhra: <p>I don’t think GDrive is the same as GCS. Being able to load data from AWS s3 buckets and GCP gcs are features I am looking forward to.</p>

nivek: <p>The <code>fsspec</code> and <code>iopath</code> <a href="https://pytorch.org/data/0.3.0/torchdata.datapipes.iter.html#io-datapipes" rel="noopener nofollow ugc">DataPipes here</a> should allow you to load data from S3. Let us know that works for you.</p>
<p>I don’t think we have looked in to Google Cloud Storage but I’ll make a note of it.</p>

Nilabhra: <p>Thanks! I’ll check if the functionalities allow for usage of GCS buckets.</p>

sephib: <p>Hi,<br>
I’ve also started to check this option out.<br>
what I have found is the following:</p>
<ol>
<li>there is an issue with setting credentials to <code>fsspec</code> in the <a href="https://github.com/pytorch/data/blob/cd38927904836f6f67ce33bfaee094fff4078402/torchdata/datapipes/iter/load/fsspec.py#L69" rel="noopener nofollow ugc">FSSpecFileListerIterDataPipe</a>. a possible solution is to add a <code>token</code> argument to <code>fsspec.core.url</code>)<br>
<code>fs, path = fsspec.core.url_to_fs(self.root, token=path/to/json)</code>
</li>
<li>the output from <a href="https://github.com/pytorch/data/blob/aab67f314d9c8329ea688e5841abc16cbe2c4f42/torchdata/datapipes/iter/load/fsspec.py#L710" rel="noopener nofollow ugc">fs.protocol</a> is a tuple <code>('gcs', 'gs')</code> .<br>
I’ve currently patched the code to take the relevant prefix for our GCS path.<br>
I hope to update once I get the pipeline running</li>
</ol>

nivek: <p>Hi <a class="mention" href="/u/sephib">@sephib</a>, thanks for spotting the issue, we appreciate it!</p>
<p>Please open an issue (and a PR if you have a proposed fix) <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc">on GitHub</a>.</p>

nivek: <p>We currently have a PR: <a href="https://github.com/pytorch/data/pull/495" class="inline-onebox" rel="noopener nofollow ugc">Improve fsspec DataPipe to accept extra keyword arguments by ejguan · Pull Request #495 · pytorch/data · GitHub</a></p>
<p>Let us know if it doesn’t resolve your need to pass a <code>token</code> argument.</p>

sephib: <p><img src="https://discuss.pytorch.org/images/emoji/apple/ok_hand/4.png?v=12" title=":ok_hand:t4:" class="emoji only-emoji" alt=":ok_hand:t4:" loading="lazy" width="20" height="20"><br>
BTW -   currently investigating 2 strange issues with the hotfix of the <code>token</code></p>
<ol>
<li>I tried to implement <a href="https://pytorch.org/data/beta/tutorial.html" rel="noopener nofollow ugc">the datapipline/dataloader tutorial</a>  - but when trying to iterate on <code>DataLoader</code> the system doesn’t do anything, doesn’t time out but keeps of running…</li>
</ol>
<pre><code class="lang-python">...
for idx, batch in enumerate(dl):
    print(idx)
</code></pre>
<ol start="2">
<li>Another issue is that when running twice the following example - on the first time the system works, but on the second time (without restarting the kernel) the <code>fs.protocole</code> in the <code>open_file_by_fsspec</code> class returns a <code>file_uri</code> without <code>gs://</code> thus failing <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20">
</li>
</ol>
<pre><code class="lang-python">datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
ds = Mapper(file_dp, PIL_open)
for i in ds:     
    print(f'{i=}')
    show_image(i)
</code></pre>

nivek: <p>For 1, can you let us know what version of the libraries (<code>torch</code> and <code>torchdata</code>) you have installed? And what is the exact code snippet that you are running?</p>
<ol start="2">
<li>By the hotfix, are you running the code from <a href="https://github.com/pytorch/data/pull/495/files" rel="noopener nofollow ugc">this PR</a> or the fix that you described above?</li>
</ol>
<p>cc: <a class="mention" href="/u/ejguan">@ejguan</a></p>

ejguan: <aside class="quote no-group" data-username="sephib" data-post="9" data-topic="146253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/48/50189_2.png" class="avatar"> sephib:</div>
<blockquote>
<ol start="2">
<li>Another issue is that when running twice the following example - on the first time the system works, but on the second time (without restarting the kernel) the <code>fs.protocole</code> in the <code>open_file_by_fsspec</code> class returns a <code>file_uri</code> without <code>gs://</code> thus failing <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></li>
</ol>
<pre><code class="lang-auto">datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
ds = Mapper(file_dp, PIL_open)
for i in ds:     
    print(f'{i=}')
    show_image(i)
</code></pre>
</blockquote>
</aside>
<p>What do you mean about running twice? You mean iterating <code>ds</code> twice or running the script twice?</p>
<p>Does your <code>image_bucket</code> start with <code>gs://</code>?</p>

sephib: <ol>
<li>the version are</li>
</ol>
<pre><code class="lang-auto">Name: torch
Version: 1.11.0

Name: torchdata
Version: 0.3.0
</code></pre>
<p>The above code is the exact code that I’m running<br>
here is a simpler version</p>
<pre data-code-wrap="python"><code class="lang-python">datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)
</code></pre>
<p>after this if you run</p>
<pre data-code-wrap="python"><code class="lang-python">datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)
</code></pre>
<p>again - you get a <code>FileNotFoundError</code> .<br>
Once I restart the kernel it works.<br>
If I run the script as below there is no problem - I’m guessing it is something regarding the how jupyter kernel uses fsspec <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>python my_script.py</p>
</blockquote>
<p>there is no problem. something is</p>
<ol start="2">
<li>the hotfix - is my local one - where I added the<code>token=</code> to <code>fsspec.core.url_to_fs</code></li>
</ol>

sephib: <p>yes - my URI <code>image_bucket</code> starts with <code>gs://</code></p>

ejguan: <aside class="quote no-group" data-username="sephib" data-post="12" data-topic="146253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/48/50189_2.png" class="avatar"> sephib:</div>
<blockquote>
<p>here is a simpler version</p>
<pre><code class="lang-auto">datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)
</code></pre>
<p>after this if you run</p>
<pre><code class="lang-auto">datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)
</code></pre>
<p>again - you get a <code>FileNotFoundError</code> .</p>
</blockquote>
</aside>
<p>This is interesting. I am not sure if this is something with the caching within <code>fsspec</code>. Could you please open an issue on Github?</p>

sephib: <p>Rergarding the <a href="https://pytorch.org/data/beta/tutorial.html" rel="noopener nofollow ugc">the datapipline/dataloader tutorial</a>  - this is the code that i’m trying to run  wich doesn’t return any answer.</p>
<pre><code class="lang-python">def build_datapipes(root_dir=image_bucket):
    datapipe = FSSpecFileLister(root=root_dir, masks=['*.png'])
    file_dp = datapipe.open_file_by_fsspec(mode='rb')  
    datapipe = file_dp.map(PIL_open)
    return datapipe

datapipe = build_datapipes()
dl = DataLoader(dataset=datapipe, batch_size=1, num_workers=1)
list(dl)
</code></pre>
<p>When killing the service I can see in the stack that it is waiting  on the <code>python3.9/multiprocessing/connection.py", line 936,</code></p>

ejguan: <p>Are you running this in ikernel? I believe notebook has some problems with multiprocessing (num_workers&gt;0)</p>

sephib: <p>No  - I’m running via a regular python environment</p>
<blockquote>
<p>python my_script.py</p>
</blockquote>

sephib: <p>Running with</p>
<blockquote>
<p>dl = DataLoader(dataset=datapipe)</p>
</blockquote>
<p>Solved the issue<br>
Will need to understand why the <code>num_workers</code> is causing issues on my setup</p>

sephib: <p>Opened an <a href="https://github.com/pytorch/data/issues/497" rel="noopener nofollow ugc">issue on GitHub</a></p>

sephib: <p>Hi<br>
here is a <a href="https://sephib.github.io/when-pytorch-datapipes-meets-gcs.html#when-pytorch-datapipes-meets-gcs" rel="noopener nofollow ugc">write up</a> of my solution</p>