Loading huge data functionality

Xia_Yandi: <p>Do you have any plan on implementing big data files loading functionality?</p>
<p>Suppose I have 300G data files for training, and I can’t load them all into memory.</p>
<p>For now, I am using TensorFlow, and they provide producer/consumer style data loading:<br>
<a href="https://www.tensorflow.org/how_tos/reading_data/" class="onebox" target="_blank" rel="nofollow noopener">https://www.tensorflow.org/how_tos/reading_data/</a></p>
<p>With it, I don’t need to read all data set into memory at once, and I can load data in parallel fashion.</p>
<p>Any plan on similar functionality?</p>
<p>Thanks.</p>

NgPDat: <p>You can already do that with <a href="https://github.com/pytorch/tnt" rel="nofollow noopener">Pytorchnet</a>.<br>
Concretely, you pass a list of data files into <code>tnt.ListDataset</code>, then wrap it with <code>torch.utils.DataLoader</code>.<br>
Example code:</p>
<pre><code class="lang-auto">def load_func(line):
    # a line in 'list.txt"

    # Implement how you load a single piece of data here

    # assuming you already load data into src and target respectively
    return {'src': src, 'target': target} # you can return a tuple or whatever you want it to

def batchify(batch):
    # batch will contain a list of {'src', 'target'}, or how you return it in load_func.

    # Implement method to batch the list above into Tensor here

    # assuming you already have two tensor containing batched Tensor for src and target
    return {'src': batch_src, 'target': batch_target} # you can return a tuple or whatever you want it to


dataset = ListDataset('list.txt', load_func) #list.txt contain list of datafiles, one per line
dataset = DataLoader(dataset=dataset, batch_size=50, num_workers=8, collate_fn=batchify) #This will load data when needed, in parallel, up to &lt;num_workers&gt; thread.

for x in dataset: #iterate dataset
    print(x)
</code></pre>
<p>There are surely other way to do it. Hope this helps.</p>

apaszke: <p>Just define a <code>Dataset</code> object, that only loads a list of files in <code>__init__</code>, and loads them every time <code>__getindex__</code> is called. Then, wrap it in a <code>torch.utils.DataLoader</code> with multiple workers, and you’ll have your files loaded lazily in parallel.</p>
<pre><code class="lang-auto">class MyDataset(torch.utils.Dataset):
    def __init__(self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__(self, idx):
        return load_file(self.data_files[idx])

    def __len__(self):
        return len(self.data_files)


dset = MyDataset()
loader = torch.utils.DataLoader(dset, num_workers=8)
</code></pre>

Xia_Yandi: <p>That is so cool! Thanks!</p>

Morpheus_Hsieh: <p>Sorry, new to PyTorch. How might one adapt the above method to <code>pytorch/examples/word_language_model/</code>? Currently, it seems to load the entire data into cuda, which is causing OOM errors.</p>

apaszke: <p>It would require rewriting the whole data loading part. It would need to go over all of it to gather all the tokens, and then lazily load only the batches you request. You’d need to add a proper <code>torch.utils.Dataset</code> subclass that does it.</p>

Morpheus_Hsieh: <p>Thanks. I did something that may sound stupid.<br>
I removed <code>data.cuda()</code> from <code>batchify()</code> and added<br>
<code>data = data.cuda()</code><br>
<code>target = target.cuda()</code><br>
in <code>get_batch()</code>.<br>
It seems to be running now, but I wonder if this is a quick (and correct) fix.</p>

apaszke: <p>Yeah, that’s correct too!</p>

Morpheus_Hsieh: <p>That is a relief! However, I am noticing a huge and growing amount of memory usage now, after running for 40+ hours. Is there anything that I am missing? Perhaps every time when <code>get_batch()</code> is run, it is creating <code>Variable</code>s and they are kept in memory after the batch is finished? Thanks.</p>

apaszke: <p>Is that CPU memory or GPU memory? Everything should get freed from time to time. Are you just running the example with that single modification?</p>

Xia_Yandi: <p>I wrote something follow your instruction. But it doesn’t work for me.</p>
<p>Here is what I do:</p>
<pre><code class="lang-auto">    def _load_hdf5_file(hdf5_file):
        f = h5py.File(hdf5_file, "r")
        data = []
        for key in f.keys():
            data.append(f[key])
        return tuple(data)


    class HDF5Dataset(Dataset):
        def __init__(self, data_files):
            self.data_files = sorted(data_files)

        def __getitem__(self, index):
            return _load_hdf5_file(self.data_files[index])

        def __len__(self):
            return len(self.data_files)

    train_set = HDF5Dataset(train_files) # there is only one file in train_files, i.e. train_files = ["foo_1"]
    train_loader = DataLoader(dataset=train_set,
                              batch_size=train_batch_size,
                              shuffle=True,
                              num_workers=2)
</code></pre>
<p>And during iteration, I got this error:</p>
<pre><code class="lang-auto">    Traceback (most recent call last):
      File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
      File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in __call__
      File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
      File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
    OSError: [Errno 24] Too many open files: '/tmp/pymp-Y6oJsO'
    Process Process-1:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
      File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put
      File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 17, in send
      File "/usr/lib/python2.7/pickle.py", line 224, in dump
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/multiprocessing/forking.py", line 67, in dispatcher
      File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/multiprocessing/forking.py", line 66, in dispatcher
      File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/reductions.py", line 116, in reduce_storage
      File "/usr/lib/python2.7/multiprocessing/reduction.py", line 145, in reduce_handle
    OSError: [Errno 24] Too many open files
</code></pre>
<p>And the program never terminates.</p>
<p>Did I do anything wrong? Thanks</p>
<p>By the way, I don’t know if it is appropriate to ask here, how can I post python style code here like you did? Thanks.</p>

apaszke: <p>It seems that your system allows only for a small number of open files. Can you try adding <code>torch.multiprocessing.set_sharing_strategy('file_system')</code> at the top of your script and try again?</p>
<p>Just append <code>python</code> <a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code">after the three backticks</a> to add syntax highlighting.</p>

Xia_Yandi: <p>I added the line, and I got this error:</p>
<pre><code class="lang-auto">Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
    data_queue.put((idx, samples))
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put
    return send(obj)
  File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 17, in send
    ForkingPickler(buf, pickle.HIGHEST_PROTOCOL).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 67, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 66, in dispatcher
    rv = reduce(obj)
  File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/reductions.py", line 109, in reduce_storage
    metadata = storage._share_filename_()
RuntimeError: $ Torch: unable to mmap memory: you tried to mmap 0GB. at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/TH/THAllocator.c:317
</code></pre>
<p>Thanks</p>

Morpheus_Hsieh: <p>That was CPU memory that I noticed using <code>htop</code>. Besides the above modification, I used another corpus which is about 9 GB. Anyway, it finished without OOM so I guess it’s fine <img src="//discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=5" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:">.</p>

apaszke: <p>That was probably an out of memory error. If the data or the code is public (or if you could just isolate the data loading into a separate script), I could run it myself and make sure it doesn’t leak. But we have some tests for that.</p>

Xia_Yandi: <p>Hi,</p>
<p>I just wrote a simple demo code to reproduce the error. The code is here:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/xiayandi/Pytorch_text_classification" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/11317896?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/xiayandi/Pytorch_text_classification" target="_blank" rel="nofollow noopener">xiayandi/Pytorch_text_classification</a></h3>

<p>A simple implementation of CNN based text classification in Pytorch - xiayandi/Pytorch_text_classification</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The data is randomly generated, but everything is the same as mine except the actual values.</p>
<p>You could run main.py to reproduce the error.</p>
<p>I don’t know if this will be useful, my system is:<br>
Ubuntu 16.04, TitanX, cuda 8.0, pip installed pytorch</p>
<p>Thanks!</p>

Morpheus_Hsieh: <p>Thanks so much. I literally just modified a very small part of the original example code <code>main.py</code></p>
<pre><code class="lang-auto">--- a/word_language_model/main.py
+++ b/word_language_model/main.py
@@ -57,8 +57,9 @@ def batchify(data, bsz):
     nbatch = data.size(0) // bsz
     data = data.narrow(0, 0, nbatch * bsz)
     data = data.view(bsz, -1).t().contiguous()
-    if args.cuda:
-        data = data.cuda()
+
+    # if args.cuda:
+    #     data = data.cuda()
     return data
 
 eval_batch_size = 10
@@ -103,6 +104,9 @@ def get_batch(source, i, evaluation=False):
     seq_len = min(args.bptt, len(source) - 1 - i)
     data = Variable(source[i:i+seq_len], volatile=evaluation)
     target = Variable(source[i+1:i+1+seq_len].view(-1))
+    if args.cuda:
+        data = data.cuda()
+        target = target.cuda()
     return data, target
</code></pre>

apaszke: <p>I’ve started a script running <a class="mention" href="/u/morpheus_hsieh">@Morpheus_Hsieh</a>’s script, I’ll try the demo later. Thanks!</p>

itzjustricky: <p>for the code snippet that you provided.</p>
<pre><code>class MyDataset(torch.utils.Dataset):
    def __init__(self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__(self, idx):
        return load_file(self.data_files[idx])

    def __len__(self):
        return len(self.data_files)
</code></pre>
<p>Are the file paths stored in self.data_files suppose to represent each batch of data (or data per loop) returned by iterating loader?</p>

smth: <p>it is data per instance of the loop</p>