How to Save DataLoader?

username: Noiran_Allen
cooked: <p>Hi,</p>
<p>I am new to PyTorch and currently experimenting on PyTorch’s DataLoader on Google Colab. My experiment often requires training time over 12 hours, which is more than what Google Colab offers. Due to this reason, I need to be able to save my optimizer, learning rate scheduler, and the state per specific epoch checkpoint (e.g., every epoch of multitude 5).</p>
<p>I also need to save the data loaded per mini-batch, of which the size is either 32 or 64. However, I could not save the current state of the DataLoader to be reused for other training epochs. How can I save the current state of DataLoader? Thanks in advance.</p>
<p>Note: May be linked with this <a href="https://github.com/pytorch/pytorch/issues/11813" rel="nofollow noopener">issue</a> on PyTorch’s Github.</p>

username: Balamurali_M
cooked: <p>Can’t we use random seed and get the same list ? Then, knowing the index will suffice.</p>

username: Eta_C
cooked: <p>Maybe</p>
<pre><code class="lang-python">import random
from torch.utils.data import Sampler, DataLoader

class MySampler(Sampler):
    def __init__(self, data_source):
        self.seq = list(range(len(data_source)))
    def __iter__(self):
        return iter(self.seq)

dataset = LoadYourDataset()
sampler = MySampler(dataset)
dataloader = DataLoader(dataset, shuffle=False, sampler=sampler)

for epoch in range(0, 999):
    random.shuffle(dataloader.sampler.seq)
    for i, (x, y) in enumerate(dataloader):
        # some code.
        # save i and dataloader.sampler.seq
</code></pre>
<p>I cannot promise that this code will work (just an idea).</p>

username: Noiran_Allen
cooked: <p>Thank you for the answers. After trying some codes of my own yesterday, I figured out that <code>DataLoader</code> can be saved directly using PyTorch’s <code>torch.save(dataloader_obj, 'dataloader.pth')</code>. The order of data is maintained so far, and the batches as well.</p>

username: Aminul_Huq
cooked: <p>How did you load after saving it ? Can you provide sample code please ?</p>

username: Noiran_Allen
cooked: <p>Oh just use the <code>torch.load</code> as usual if I’m not mistaken.</p>

username: Aminul_Huq
cooked: <p>Thanks,  I tried by creating a DataLoader() object and then loading into it and got errors. It seems just torch.load() is enough.</p>
<p>Thanks</p>

username: jchibane
cooked: <p>Actually, I can’t do torch.save on a dataloader - is it for sure working on your side? If yes, what pytorch version do you use?</p>

username: Noiran_Allen
cooked: <p>It worked on Pytorch 0.4.0 if I’m not mistaken.</p>

username: Fanny
cooked: <p>Hello,<br>
I’m on torch 1.2.0 and torch.save seems to work. However, when I try to load (with torch.load) my Dataloader in which I included transforms from torchvision, I get the following error</p>
<pre><code class="lang-auto"> File "C:\Users\USER\env\lib\site-packages\torch\serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\USER\env\lib\site-packages\torch\serialization.py", line 573, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'transforms'
</code></pre>
<p>Any idea how to solve this ?</p>

username: Noiran_Allen
cooked: <p>Have you tried importing the torchvision.transforms module? Also could you verify that torchvision is installed <a class="mention" href="/u/fanny">@Fanny</a> ?</p>

username: Fanny
cooked: <p>Yes <a class="mention" href="/u/noiran_allen">@Noiran_Allen</a> , the module is imported in my script and when I try to use transforms, for example with <code>tr = transforms.Resize(248)</code>, it does work.</p>

username: Fanny
cooked: <p>I think I found out the problem here. According to this topic <a href="https://discuss.pytorch.org/t/modulenotfounderror-no-module-named-network/71721/3" class="inline-onebox">ModuleNotFoundError: No module named 'network' - #3 by Oscar_Rangel</a> concerning the saving of full models, ptrblck said that :</p>
<blockquote>
<p>If you are using this approach:  <code>model = torch.load(path)</code> , you would need to make sure that all necessary files are in the corresponding folders as they were while storing the model.</p>
</blockquote>
<p>It looks like it does apply here too when I use the same organisation as the place where I saved the dataLoader, when I saved it.</p>

username: Butters
cooked: <p>As a side remark: apparently, it is recommended to use the extension “.pt” rather than “.pth” when saving model checkpoints (and maybe for the same reason also data loaders):</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/59095824/what-is-the-difference-between-pt-pth-and-pwf-extentions-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/4317729/asil" target="_blank" rel="noopener nofollow ugc">
    <img alt="Asil" src="https://www.gravatar.com/avatar/67f8824e56fe983925eee0002f7c177a?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/59095824/what-is-the-difference-between-pt-pth-and-pwf-extentions-in-pytorch" target="_blank" rel="noopener nofollow ugc">What is the difference between .pt, .pth and .pwf extentions in PyTorch?</a>
</h4>

<div class="tags">
  <strong>python, serialization, deep-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/4317729/asil" target="_blank" rel="noopener nofollow ugc">
    Asil
  </a>
  on <a href="https://stackoverflow.com/questions/59095824/what-is-the-difference-between-pt-pth-and-pwf-extentions-in-pytorch" target="_blank" rel="noopener nofollow ugc">08:33PM - 28 Nov 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>