How does dataloader create a tensor batch

Mika_S: <p>How does the dataloader create a batch of tensors? Is it simply doing:<br>
<code>torch.cat([tensor_1, tensor_2])</code> ?</p>
<p>Trying to understand whether there is more optimization than that or not.</p>

ptrblck: <p>Depending on the type your <code>__getitem__</code> method from the <code>Dataset</code> returns, the <a href="https://github.com/pytorch/pytorch/blob/a996b1d653f5fc1e3fe28eaca41b8fd42178574a/torch/utils/data/_utils/collate.py#L43" rel="nofollow noopener"><code>default_collate</code></a> will use <code>torch.stack</code>, create a <code>torch.tensor</code> etc.</p>

rui_zhang_331: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,  we want to put some JSON parsing logic into dataloader to leverage multiple worker capabity, can we have return type just a dict? it seems if we use former 1.0.0 version it will return pure dict, but later version will create tensor?</p>
<p>here is our <code>__get_item__</code></p>
<pre><code class="lang-auto">with open(current_file, mode='rb') as f:
            text = f.read().decode('utf-8')
            all_data.extend(text.split('\n'))

        json_data = []
        for line in all_data:
            try:
                json_data.append(json.loads(line))
            except:
                break
return json_data
</code></pre>

ptrblck: <p>It seems you could still return a <code>dict</code>.<br>
At least this small code snippet works in the nightly binary from approx. a week ago:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        pass
    
    def __getitem__(self, index):
        return {'x': torch.randn(3, 24, 24), 'y': torch.randint(0, 10, (1,))}
    
    def __len__(self):
        return 10

dataset = MyDataset()
loader = DataLoader(dataset, batch_size=5, num_workers=0)

for batch in loader:
    print(batch['x'].shape)
    print(batch['y'].shape)
</code></pre>
<p>Are you getting an error and if so, could you post the error message?</p>

rui_zhang_331: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>My error msg is as:</p>
<pre><code class="lang-auto">ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "/home/miniconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 480, in _try_get_batch
    data = self.data_queue.get(timeout=timeout)
  File "/home/miniconda/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/home/miniconda/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
  File "/home/miniconda/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 65, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 95106) is killed by signal: Segmentation fault.
</code></pre>
<p>One interest thing is</p>
<p>when I just parse the data line by line(each line is JSON string), I do not have this issue:</p>
<pre><code class="lang-auto">        with open(current_file, mode='rb') as f:
            text = f.read().decode('utf-8')
            all_data.extend(text.split('\n'))
</code></pre>
<p>but if I add a JSON parse logic after read line by line , it will report this error</p>
<pre><code class="lang-auto">with open(current_file, mode='rb') as f:
            text = f.read().decode('utf-8')
            all_data.extend(text.split('\n'))

        json_data = []
        for line in all_data:
            try:
                json_data.append(json.loads(line))
            except:
                break
return json_data
</code></pre>
<p>searched around and found out multiple github link pointed to this is due to lack of shm, however I have sufficient shm size, I understand there will be some JSON memory overhead, but even I decrease the number of worker into 2, and data set is very small, it still have same problem. I kind of doubt it is shm related. any clue?</p>
<p>Also another interesting part is when I directly return JSON.loads item as <code>__getitem__</code>,  it auto convert to tensor type ,is it something new pytorch dataloader automatically do?</p>

ptrblck: <p>Let’s continue the discussion in <a href="https://discuss.pytorch.org/t/strange-error-when-parsing-json-in-multiple-worker-data-loader/85733">this thread</a> please.</p>