Dataloader fails when using num_workers > 1

username: sivanand.achanta
cooked: <p>I am using torch.distributed to launch and distributed training task. I am also trying to use “num_workers &gt; 1” to optimize the training speed.</p>
<p>However, I get the following “Segmentation Fault” error whenever I using  “num_workers &gt; 1” …</p>
<p>Error: DataLoader worker (pid(s) 17423) exited unexpectedly<br>
Traceback (most recent call last):<br>
File “/miniconda/envs/iris/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 990, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/miniconda/envs/iris/lib/python3.6/multiprocessing/queues.py”, line 104, in get<br>
if not self._poll(timeout):<br>
File “/miniconda/envs/iris/lib/python3.6/multiprocessing/connection.py”, line 257, in poll<br>
return self._poll(timeout)<br>
File “/miniconda/envs/iris/lib/python3.6/multiprocessing/connection.py”, line 414, in _poll<br>
r = wait([self], timeout)<br>
File “/miniconda/envs/iris/lib/python3.6/multiprocessing/connection.py”, line 911, in wait<br>
ready = selector.select(timeout)<br>
File “/miniconda/envs/iris/lib/python3.6/selectors.py”, line 376, in select<br>
fd_event_list = self._poll.poll(timeout)<br>
File “/miniconda/envs/iris/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 17423) is killed by signal: Segmentation fault.</p>
<p>I tried to analyze the root-cause of this error … in my case this error comes in the following scenario:<br>
Whenever I send the metrics (using tensorboard or equivalent) … I get the above segmentation fault, once the training resumes …</p>
<p>I use a pre-condition to check if its the master-worker and send the metrics only using the master-process … once the training resumes I get the SegFault Error … Is there a way to prevent the DataLoader to use master worker for dataloading … or is there a different way in which we can address this issue … Please let me know …</p>
<p>The Error is also reported sometimes in a different was as below:<br>
File “dist_pytorch/train_simple.py”, line 112, in train<br>
for data in iter(dataloader):<br>
File “/miniconda/envs/iris/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 517, in <strong>next</strong><br>
data = self._next_data()<br>
File “/miniconda/envs/iris/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 1182, in _next_data<br>
idx, data = self._get_data()<br>
File “/miniconda/envs/iris/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 1148, in _get_data<br>
success, data = self._try_get_data()<br>
File “/miniconda/envs/iris/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 986, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/queues.py”, line 113, in get<br>
return _ForkingPickler.loads(res)<br>
File “/miniconda/envs/iris/lib/python3.7/site-packages/torch/multiprocessing/reductions.py”, line 282, in rebuild_storage_fd<br>
fd = df.detach()<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/resource_sharer.py”, line 57, in detach<br>
with _resource_sharer.get_connection(self._id) as conn:<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/resource_sharer.py”, line 87, in get_connection<br>
c = Client(address, authkey=process.current_process().authkey)<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/connection.py”, line 498, in Client<br>
answer_challenge(c, authkey)<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/connection.py”, line 742, in answer_challenge<br>
message = connection.recv_bytes(256)         # reject large message<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/connection.py”, line 216, in recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/connection.py”, line 407, in _recv_bytes<br>
buf = self._recv(4)<br>
File “/miniconda/envs/iris/lib/python3.7/multiprocessing/connection.py”, line 379, in _recv<br>
chunk = read(handle, remaining)<br>
ConnectionResetError: [Errno 104] Connection reset by peer</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sivanand.achanta" data-post="1" data-topic="151689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sivanand.achanta/48/48130_2.png" class="avatar"> sivanand.achanta:</div>
<blockquote>
<p>I use a pre-condition to check if its the master-worker and send the metrics only using the master-process … once the training resumes I get the SegFault Error</p>
</blockquote>
</aside>
<p>Could you explain what exactly you did which seems to cause the segfault? Based on your description I guess you are checking the worker id somehow inside the <code>Dataset</code>?</p>

username: sivanand.achanta
cooked: <p>Hi,</p>
<p>I have not added anything to lead to a SegFault … I was only mentioning the program flow which I thought was somehow leading to this error … my guess might have been wrong …</p>
<p>I am attaching a sample training program which might explain the above condition …</p>
<p>import torch<br>
import torch.nn as nn<br>
from torch.utils.data import Dataset, DataLoader<br>
from torch.utils.data.distributed import DistributedSampler<br>
import numpy as np<br>
import argparse<br>
import os<br>
import turibolt as bolt<br>
L2_Loss = nn.MSELoss()</p>
<h1>
<a name="h-1" class="anchor" href="#h-1"></a>---------------------------------------------------------------------------------------------------------------------</h1>
<p>class SpeechDataset(Dataset):<br>
def <strong>init</strong>(self):<br>
self.T_MAX = 200<br>
def <strong>len</strong>(self):<br>
return 40000<br>
def <strong>getitem</strong>(self, idx):<br>
mels = np.random.randn(self.T_MAX, 512)<br>
return torch.from_numpy(mels).type(torch.FloatTensor).transpose(0, 1)</p>
<h1>
<a name="h-2" class="anchor" href="#h-2"></a>---------------------------------------------------------------------------------------------------------------------</h1>
<p>class MelsCollate():<br>
“”" Zero-pads model inputs and targets based on number of frames per step<br>
“”"<br>
def <strong>init</strong>(self):<br>
pass</p>
<pre><code>def __call__(self, batch):
    """Collate's training batch from normalized text and mel-spectrogram
    PARAMS
    ------
    batch: [text_normalized, mel_normalized]
    """
    # Right zero-pad all mel sequences to max mel length
    melseq_lengths, ids_sorted_decreasing = torch.sort(
        torch.LongTensor([x.size(1) for x in batch]),
        dim=0, descending=True)
    max_mel_len = melseq_lengths[0]
    mels_padded = torch.FloatTensor(len(batch), 512, max_mel_len)

    # initialize with zeros
    mels_padded.zero_()
    # copy
    for i in range(len(ids_sorted_decreasing)):
        mels = batch[ids_sorted_decreasing[i]]
        mels_padded[i, :, :mels.size(1)] = mels
    return mels_padded
</code></pre>
<h1>
<a name="h-3" class="anchor" href="#h-3"></a>----------------------------------------------------------------------------------------------------------------------</h1>
<p>class BasicConvBlock(nn.Module):<br>
def <strong>init</strong>(self, ndim_in, ndim_out, kernel_size, dilation_factor=1):<br>
super(BasicConvBlock, self).<strong>init</strong>()<br>
self.relu = nn.ReLU()<br>
padding=int((kernel_size - 1) / 2)*dilation_factor<br>
self.conv1 = nn.Conv1d(ndim_in, ndim_out, kernel_size, stride=1, dilation=dilation_factor, padding=padding)</p>
<pre><code>def forward(self, input):
    output = self.relu(self.conv1(input)) + input
    return output
</code></pre>
<h1>
<a name="h-4" class="anchor" href="#h-4"></a>----------------------------------------------------------------------------------------------------------------------</h1>
<p>class ConvStack(nn.Module):<br>
def <strong>init</strong>(self, ndim_in, ndim_out):<br>
super(ConvStack, self).<strong>init</strong>()<br>
self.dilation_factors = [1, 2, 4, 8, 16, 1, 2, 4, 8, 16]<br>
self.conv_stack = nn.ModuleList([BasicConvBlock(ndim_in=ndim_in, ndim_out=ndim_out, kernel_size=21, dilation_factor=dilation_factor) <br>
for dilation_factor in self.dilation_factors])</p>
<pre><code>def forward(self, input):
    output = input
    for conv_block in self.conv_stack:
        output = conv_block(output)
    return output
</code></pre>
<h1>
<a name="h-5" class="anchor" href="#h-5"></a>----------------------------------------------------------------------------------------------------------------------</h1>
<p>def train_step(data, model, optimizer, scheduler, device):<br>
data = data.to(device)<br>
optimizer.zero_grad()<br>
pred_mels = model(data)<br>
total_loss = L2_Loss(pred_mels, data)<br>
total_loss.backward()<br>
optimizer.step()<br>
scheduler.step()<br>
return total_loss</p>
<h1>
<a name="h-6" class="anchor" href="#h-6"></a>----------------------------------------------------------------------------------------------------------------------</h1>
<p>def train(local_rank, send_bolt_metrics):<br>
torch.cuda.set_device(local_rank)<br>
num_epochs = 5</p>
<pre><code>#setup the distributed backend for managing the distributed training
torch.distributed.init_process_group('nccl')

# dataset
dataset = SpeechDataset()
dist_sampler = DistributedSampler(dataset)
dataloader = DataLoader(dataset, batch_size=16, num_workers=2, sampler=dist_sampler, collate_fn=MelsCollate())

# model
model = ConvStack(512, 512)

#set the cuda device to a GPU allocated to current process
device = torch.device('cuda', local_rank)
model = model.to(device)
model = torch.nn.parallel.DistributedDataParallel(model,  device_ids=[local_rank],
                                                      output_device=local_rank)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0003, betas=(0.9, 0.98), eps=1e-09)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[i*100000 for i in range(1, 6)], gamma=0.5)

step = 0
for epoch in range(num_epochs):
    for data in iter(dataloader):
        loss = train_step(data, model, optimizer, scheduler, device)
        step += 1
        if step % 10 == 0 and local_rank==0:
            print(epoch, step, loss.detach().cpu().numpy())
            # send metrics
            if send_bolt_metrics:
                bolt.send_metrics({'loss': loss.detach().cpu().item()})
            # save model
            torch.save(model.state_dict, 'checkpoints/model_' + str(step) + '.pt')
</code></pre>
<p><span class="hashtag">#prase</span> the local_rank argument from command line for the current process<br>
parser = argparse.ArgumentParser()<br>
parser.add_argument("–local_rank", default=0, type=int)<br>
parser.add_argument("–send_bolt_metrics", default=False, action=‘store_true’)<br>
args = parser.parse_args()<br>
os.makedirs(‘checkpoints’, exist_ok=True)<br>
train(args.local_rank, args.send_bolt_metrics)</p>
<p>The error occurs when the send_bolt_metrics=True … otherwise the training seems to proceed fine …</p>
<p>bolt.send_metrics is a mechanism via which the metrics are sent to a webpage for visualization … similar to tensorboard …</p>
<p>Somehow it seems as if “connection” is reset by this bolt.send_metrics which gives rise to this error: “connection reset by peer” …</p>
<p>I am not sure how to avoid this …</p>