Lazy loading of wide dataset

username: Kasper_Rasmussen
cooked: <p>Hi Pytorch community,</p>
<p>I am training a model on a very wide dataset (~500,000 features). To read the data from disc I use dask to load an <code>xarray.core.dataarray.DataArray</code> object to not load all the data in memory at once. I can load subsets of the data into memory with a numpy array as such: <code>xarray[0:64,:].values</code>. This loads 64 samples into memory in about 2 seconds. I then want to feed the data to the model one batch at a time, using batch size of 64, since this should have an estimated epoch time of 3 minutes given the total sample size (~6000). My problem is when I try to implement this functionality in the pytorch <code>Dataset</code>module. I want to create a class in which a <code>DataLoader</code> loads the data from disc into memory one batch at a time with a specified batch size. To that I made the following <code>Dataset</code>class and wrapped it in a <code>DataLoader</code>.</p>
<pre><code class="lang-auto">
class load_wide(Dataset):

    def __init__(self, xarray, labels):
        
        self.xarray = xarray
        self.labels = labels

    def __getitem__(self, item):
        
        data = self.xarray[item,:].values
        labels = self.labels[item]
        
        return data, labels

    def __len__(self):
        return len(self.xarray)

</code></pre>
<p>I then load the data like this:</p>
<pre><code class="lang-auto">train_ds = load_plink(xarray_train, labels_train)
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)

for i, data in enumerate(train_dl):
    feats, labels = data
    preds = net(feats)
    loss = criterion(preds, labels)
</code></pre>
<p>This works fine, but it takes about 90 seconds to load a batch resulting in unreasonable training time. What went wrong in my implementation of dataloading, that causes it to load the data so slowly. Using the DataLoader causes a 45-fold slowdown of dataloading. Can someone explain the cause of this?<br>
Also, how can you simultaneously evaluate the model on a validation set that is loaded one batch at a time?</p>

username: ptrblck
cooked: <p>I’m not familiar with the internal implementation of the <code>xarray</code>, but what seems to be different is the shuffling. Could you test your simple code snippet with random indices <code>xarray[0:64,:].values</code> instead of contiguous ones and compare the loading speed?</p>

username: Kasper_Rasmussen
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="141382">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>xarray[0:64,:].values</p>
</blockquote>
</aside>
<p>This was the exact cause of the issue. It seems like <code>xarray</code> is not a good fit to combine with pytorch’s dataloader class. I will look for alternative ways of loading the data. Thank you!</p>

username: JinroTorch
cooked: <p>Hello! Please refer to this, maybe this could help you as it helped me.</p>
<aside class="quote" data-post="2" data-topic="9387">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ghiggi/48/34489_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/deadlock-with-dataloader-and-xarray-dask/9387/2">Deadlock with DataLoader and xarray/dask</a> 
  </div>
  <blockquote>
    I experienced similar problems. 
Is something related to the fact that netCDF I/O is not thread-safe and there is some locking going on. 
If you chunk your dataset with ds=ds.chunk(&lt;your  chunk options&gt;), you save the dataset to disk in zarr format (ds.to_zarr()) and reopen the dataset with xr.open_zarr() then everything works fine <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title="wink" alt="wink" class="emoji"> 
I additionally suggest to set dask.config.set(scheduler='synchronous') to speed up the data loading if num_workers &gt; 0 in your DataLoader.
  </blockquote>
</aside>