Strange error when parsing JSON in multiple worker data loader

rui_zhang_331: <p>Hi, we have enabled the multi worker data loader to load 10K+ training data files, the speed is pretty good with multiple workers, however, we also try to leverage the capability of multi worker to not only read data line by line, but parsing the line to JSON Dict, here we have problem</p>
<pre><code class="lang-auto">ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "/home/miniconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 480, in _try_get_batch
    data = self.data_queue.get(timeout=timeout)
  File "/home/miniconda/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/home/miniconda/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
  File "/home/miniconda/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 65, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 95106) is killed by signal: Segmentation fault.
</code></pre>
<p>searched around and it seems point to shared memory problem, However, I think our system has sufficient shared memory.</p>
<p>another interesting point is : when I just parse the data line by line, I do not have this issue:</p>
<pre><code class="lang-auto">        with open(current_file, mode='rb') as f:
            text = f.read().decode('utf-8')
            all_data.extend(text.split('\n'))
</code></pre>
<p>but if I add a JSON parse logic after read line by line , it will report this error</p>
<pre><code class="lang-auto">with open(current_file, mode='rb') as f:
            text = f.read().decode('utf-8')
            all_data.extend(text.split('\n'))

        json_data = []
        for line in all_data:
            try:
                json_data.append(json.loads(line))
            except:
                break
return json_data
</code></pre>
<p>so I wonder whether this is indeed shared memory issue? it is very rough error message. Also I wonder what is really shared memory used in multi worker data loader case, should we have multiple worker/process run independently and provide data to GPU finally? I am not seeing much need to share large chunk of memory besides some worker coordination?</p>

ptrblck: <p>Are you seeing the same error, if you reduce the number of workers?<br>
If not, then it could point to an insufficient amount of shared memory. Could you check, how much shared memory your system is using at the moment?</p>
<aside class="quote no-group" data-username="rui_zhang_331" data-post="1" data-topic="85733">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rui_zhang_331/48/25620_2.png" class="avatar"> rui_zhang_331:</div>
<blockquote>
<p>Also I wonder what is really shared memory used in multi worker data loader case</p>
</blockquote>
</aside>
<p>Each workers is a process, which needs to communicate with the main process e.g. via shared memory. Multi-threading is not really an option in Python due to the GIL.</p>

rui_zhang_331: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Yes I am seeing this error even if I have worker reduced to 2, and also I believe I have enough shared memory as below(confirmed our training platform add all memory as shared memory)</p>
<p>BTW, if I have N workers, dose it means I need O(N*size(batch)) shared memory?</p>
<pre><code class="lang-auto">
------ Messages Limits --------

max queues system wide = 32000

max size of message (bytes) = 8192

default max size of queue (bytes) = 16384

------ Shared Memory Limits --------

max number of segments = 4096

max seg size (kbytes) = 18014398509465599

max total shared memory (kbytes) = 18014398509481980

min seg size (bytes) = 1

------ Semaphore Limits --------

max number of arrays = 32000

max semaphores per array = 32000

max semaphores system wide = 1024000000

max ops per semop call = 500

semaphore max value = 32767
</code></pre>

rui_zhang_331: <p>Also I have general question regarding multi processing in pytorch, if I have multi worker dataloader already, and I start my main training proces followling</p>
<p><a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices</a></p>
<p>In this case, the main process will spin off multi process, and data loader will also have multi process, will the dataloader and training share same processes underneath in OS? or both of them create new processes?</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters

if __name__ == '__main__':
    num_processes = 4
    model = MyModel()
    # NOTE: this is required for the ``fork`` method to work
    model.share_memory()
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>

rui_zhang_331: <p>Update thread again, it seems JSON parsing introduce a lot of memory overhead in python, suspect that is problem, also JSON parsing is very slow. Any better JSON lib recommended? I am not sure whether it is JSON memory overhead that cause multi worker shm error? how do we estimate how much shm is needed for our given case?</p>

ptrblck: <p>The <code>DataLoader</code> processes should be independent from the <code>DistributedDataParallel</code> processes.<br>
You could try to use <code>pandas</code> to read and process your JSON data (I’m always using it, as it’s quite easy to further process the <code>DataFrames</code>).</p>
<p>Let me know, if that helps.</p>

rui_zhang_331: <p>Thanks, will try Pandas, our training record is split into multiple files, each files is multiple lines of string, each line is JSON string. so a little bit more to work on JSON, but definitely try Pandas</p>
<p>BTW, regarding The  <code>DataLoader</code>  processes should be independent from the  <code>DistributedDataParallel</code>  processes., so let’s see we have 32 data loader workers, and we create 16 multiprocessing for our training, then is it 48 process running? or little bit complicated and can not simply add together?</p>

ptrblck: <p>It might depend how you are using <code>DistributedDataParallel</code>.<br>
The fastest and recommended way would be to create a process for each GPU, which would also mean that these process use their own <code>DataLoaders</code>.<br>
For 8 GPUs (8 processes in DDP), and e.g. <code>num_workers=2</code>, you should see 8*2=16 processes.</p>

rui_zhang_331: <p>Got it ,Thanks, I tried Pandas it is much faster but also get same segmental fault error after a while, so I wonder is there some way I can use to calculate how much shared memory do I need? with number of workers and training data size, can we roughly estimate how much shared memory needed?</p>

ptrblck: <p>I don’t know, if this is really related to shared memory, as your system seems to have enough.<br>
Could you run the code via:</p>
<pre><code class="lang-auto">$ gdb --args python my_script.py
...
Reading symbols from python...done.
(gdb) run
...
(gdb) backtrace
...
</code></pre>
<p>and post the backtrace here?</p>

rui_zhang_331: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Very interesting I did not have any stacktrace</p>
<pre><code class="lang-auto">
(gdb) run

Starting program: /home/miniconda/bin/python performance.py

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

[New Thread 0x7fffa60a6700 (LWP 61963)]

[New Thread 0x7fffa58a5700 (LWP 61964)]

[New Thread 0x7fffa10a4700 (LWP 61965)]

[New Thread 0x7fff9e8a3700 (LWP 61966)]

[New Thread 0x7fff9c0a2700 (LWP 61967)]

[New Thread 0x7fff998a1700 (LWP 61968)]

[New Thread 0x7fff970a0700 (LWP 61969)]

[New Thread 0x7fff9489f700 (LWP 61970)]

[New Thread 0x7fff9409e700 (LWP 61971)]

[New Thread 0x7fff8f89d700 (LWP 61972)]

[New Thread 0x7fff8d09c700 (LWP 61973)]

[New Thread 0x7fff8a89b700 (LWP 61974)]

[New Thread 0x7fff8809a700 (LWP 61975)]

[New Thread 0x7fff85899700 (LWP 61976)]

[New Thread 0x7fff83098700 (LWP 61977)]

[New Thread 0x7fff80897700 (LWP 61978)]

[New Thread 0x7fff7e096700 (LWP 61979)]

[New Thread 0x7fff7d895700 (LWP 61980)]

[New Thread 0x7fff7b094700 (LWP 61981)]

[New Thread 0x7fff78893700 (LWP 61982)]

[New Thread 0x7fff74092700 (LWP 61983)]

[New Thread 0x7fff71891700 (LWP 61984)]

[New Thread 0x7fff6f090700 (LWP 61985)]

[Thread 0x7fff7e096700 (LWP 61979) exited]

[Thread 0x7fff6f090700 (LWP 61985) exited]

[Thread 0x7fff74092700 (LWP 61983) exited]

[Thread 0x7fff7b094700 (LWP 61981) exited]

[Thread 0x7fff80897700 (LWP 61978) exited]

[Thread 0x7fff83098700 (LWP 61977) exited]

[Thread 0x7fff85899700 (LWP 61976) exited]

[Thread 0x7fff8809a700 (LWP 61975) exited]

[Thread 0x7fff8a89b700 (LWP 61974) exited]

[Thread 0x7fff8d09c700 (LWP 61973) exited]

[Thread 0x7fff8f89d700 (LWP 61972) exited]

[Thread 0x7fff9409e700 (LWP 61971) exited]

[Thread 0x7fff9489f700 (LWP 61970) exited]

[Thread 0x7fff970a0700 (LWP 61969) exited]

[Thread 0x7fff998a1700 (LWP 61968) exited]

[Thread 0x7fff9c0a2700 (LWP 61967) exited]

[Thread 0x7fff9e8a3700 (LWP 61966) exited]

[Thread 0x7fffa10a4700 (LWP 61965) exited]

[Thread 0x7fffa58a5700 (LWP 61964) exited]

[Thread 0x7fffa60a6700 (LWP 61963) exited]

[Thread 0x7fff71891700 (LWP 61984) exited]

[Thread 0x7fff78893700 (LWP 61982) exited]

[Thread 0x7fff7d895700 (LWP 61980) exited]

total_files = 5040.  //customer comments

[New Thread 0x7fff6f090700 (LWP 62006)]

[New Thread 0x7fff71891700 (LWP 62007)]

start to get data //customer comments

[New Thread 0x7fff74092700 (LWP 62008)]

start to get data

[New Thread 0x7fff78893700 (LWP 62009)]

start to get data

start to get data

current file len = 570 //customer comments

current file len = 566

current file len = 567

current file len = 567

start to get data

start to get data

start to get data

start to get data

current file len = 568

current file len = 569

current file len = 570

current file len = 570

ERROR: Unexpected segmentation fault encountered in worker.

ERROR: Unexpected segmentation fault encountered in worker.

Traceback (most recent call last):

File "/home/zhrui/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data

data = self._data_queue.get(timeout=timeout)

File "/home/miniconda/lib/python3.6/multiprocessing/queues.py", line 104, in get

if not self._poll(timeout):

File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 257, in poll

return self._poll(timeout)

File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 414, in _poll

r = wait([self], timeout)

File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 911, in wait

ready = selector.select(timeout)

File "/home/miniconda/lib/python3.6/selectors.py", line 376, in select

fd_event_list = self._poll.poll(timeout)

File "/home/zhrui/.local/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler

_error_if_any_worker_fails()

RuntimeError: DataLoader worker (pid 62005) is killed by signal: Segmentation fault.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "performance.py", line 62, in &lt;module&gt;

main()

File "performance.py", line 48, in main

for i,batch in enumerate(rl_data_loader):

File "/home/zhrui/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__

data = self._next_data()

File "/home/zhrui/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data

idx, data = self._get_data()

File "/home/zhrui/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data

success, data = self._try_get_data()

File "/home/zhrui/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data

raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))

RuntimeError: DataLoader worker (pid(s) 62005) exited unexpectedly

[Thread 0x7fff78893700 (LWP 62009) exited]

[Thread 0x7fff74092700 (LWP 62008) exited]

[Thread 0x7fff71891700 (LWP 62007) exited]

[Thread 0x7fff6f090700 (LWP 62006) exited]

[Inferior 1 (process 61952) exited with code 01]

(gdb) backtrace

No stack.
</code></pre>

rui_zhang_331: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="85733">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It might depend how you are using  <code>DistributedDataParallel</code> .</p>
</blockquote>
</aside>
<p>Actually I did not use DistributedDataParallel, we are still using one machine with single GPU but 24 cores of CPU, right now we don’t think GPU is bottleneck, since CPU may not able to feed data quick enough</p>

rui_zhang_331: <p>Update thread, following this thread:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/1355" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/1355" target="_blank" rel="nofollow noopener">possible deadlock in dataloader</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-04-25" data-time="19:21:31" data-timezone="UTC">07:21PM - 25 Apr 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-10-09" data-time="17:07:47" data-timezone="UTC">05:07PM - 09 Oct 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/zym1010" target="_blank" rel="nofollow noopener">
          <img alt="zym1010" src="https://avatars3.githubusercontent.com/u/4273204?v=4" class="onebox-avatar-inline" width="20" height="20">
          zym1010
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">the bug is described at pytorch/examples#148. I just wonder if this is a bug in PyTorch itself, as the example code...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It seems if I put too complicated logic like parsing JSON or iterate over list in <code> __get_item__</code>, it will fail, if I just put a simple string split logic, it will pass. same as some one mentioned in github thread</p>
<p>so here comes my question, if I can not put much complicated data process in data_loader, should I have another multi processing logic in our main train.py and put some data cook job there to leverage the multi process in pytorch? what is best proactice</p>
<p>neverthless, I think that github bug should not be closed, issue is still there</p>

ptrblck: <p>Did you try all suggestions in the linked issue, such as increasing the number of segments?</p>

rui_zhang_331: <p>Will try, Thanks, a little bit hard to increase segment, however, I tested with smaller file size and number of files, still same problem, 4096 segment looks definitely enough for me, and also my problem happens almost right after dataloader started, so I don’t believe the data is filling to shm enough yet.</p>
<p>I am trying to use multi processing not only on data loader, but general training program also, so training will have N processing, each processing will have M dataloader processing, total <code>NxM</code> threads/process, but I am not sure whether all independent processing’s dataloader can coordinate to read correct data index</p>

rui_zhang_331: <p>Update this thread a bit,  I refactor the training code using pytorch multi processing to speed up some data processing in CPU side(in order to feed to GPU faster ), <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices</a></p>
<p>In each processing func, I also use multi-worker data loader to speed up the data loading processing time.<a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html</a></p>
<p>I put my heaving CPU JSON parsing not in dataloader, but in main training process, and the problem seems gone, I don’t know why but anyway it seems working. but have a follow up question: suppose I have  <code>N</code>  processing, each has  <code>M</code>  dataloader worker, so total  <code>NxM</code>  underneath threading there.</p>
<p>If in my dataloader, I want to get all data in a index way, which means  <code>__get_item__(self, idx)</code>  in M data loader in N different processing can work together to process different index, how can I ensure they do not process duplicate or miss process some?</p>

ptrblck: <aside class="quote no-group" data-username="rui_zhang_331" data-post="16" data-topic="85733">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rui_zhang_331/48/25620_2.png" class="avatar"> rui_zhang_331:</div>
<blockquote>
<p>If in my dataloader, I want to get all data in a index way, which means <code>__get_item__(self, idx)</code> in M data loader in N different processing can work together to process different index, how can I ensure they do not process duplicate or miss process some?</p>
</blockquote>
</aside>
<p>I’m not sure I understand the question completely.<br>
The <code>DataLoader</code> will use multiple processes and make sure that each worker gets unique indices.<br>
Are you using another multiprocessing wrapper on top of multiple <code>DataLoaders</code>?<br>
If so, what’s the reason for it? Couldn’t you increase the number of workers in a <code>DataLoader</code> instead?</p>

rui_zhang_331: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Yes, somehow I am using a multi processing framework like this</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    num_processes = 8

    can_do_distributed = torch.distributed.is_available() and torch.cuda.is_available()
    multiprocessing.spawn(fn=single_process_main, nprocs=num_processes, args=[params])
</code></pre>
<p>and in <code>single_process_main</code>,</p>
<pre><code class="lang-auto">def single_process_main(*args):
     
     data_loader = DataLoader(dataset, num_workers=4, batch_size=1)
     for loo in range(0, epochs):
          # some JSON parsing logic
          data_cook()
          train()
</code></pre>
<p>so we leverage multi processing, then each processing with multi worker dataloader, I assume we will have 8x4=32 underneath worker, but I knew dataloader 4 worker will fetch different data, but not sure whether 8 different processing 's dataloader will fetch different data, it seems not.</p>
<p>The reason why I am using this is because if I put JSON parsing in <code>__get_item__</code> in dataloader, it will have the error as indicated above, thie multi processing and multi worker seems resolve the problem</p>

ptrblck: <p>If you need this workflow to avoid the JSON issue, you could use a <code>SubsetRandomSampler</code> for each <code>DataLoader</code>, so that each <code>DataLoader</code> only loads a subset of the data.<br>
However, I’m not sure how <code>train</code> is used.<br>
Are you training 8 models simultaneously?</p>

rui_zhang_331: <p>No, actually I only train one model, train function is simple NN , and want to have multi processing to process different  training data segment(same as multi worker data loader way), but looks like it is not easy to do so</p>