akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: enforce fail at context_gpu

Vimos: <p>When I use dataloader of <code>VisionDataset</code> together with <code>Detectron</code> pretrained models, if I load the model first, then dataloader fails with the following error, otherwise the dataloader runs OK.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-8-0923c73e3ef2&gt; in &lt;module&gt;
      1 
----&gt; 2 for batch in train_loader:
      3     display(to_img(batch[0][0]))
      4     im_name = batch[2][0]
      5     premise, hypothesis, annotator_labels, gold_label = batch[1][0]

~/anaconda3/envs/rcqa/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    580                 self.reorder_dict[idx] = batch
    581                 continue
--&gt; 582             return self._process_next_batch(batch)
    583 
    584     next = __next__  # Python 2 compatibility

~/anaconda3/envs/rcqa/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _process_next_batch(self, batch)
    606                 raise Exception("KeyError:" + batch.exc_msg)
    607             else:
--&gt; 608                 raise batch.exc_type(batch.exc_msg)
    609         return batch
    610 

RuntimeError: Traceback (most recent call last):
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in &lt;listcomp&gt;
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "&lt;ipython-input-2-49e9c213df04&gt;", line 88, in __getitem__
    img = self.transform(img)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 92, in __call__
    return F.to_tensor(pic)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 79, in to_tensor
    img = torch.ByteTensor(torch.ByteStorage.from_buffer(pic.tobytes()))
RuntimeError: [enforce fail at context_gpu.cu:322] error == cudaSuccess. 3 vs 0. Error at: /opt/conda/conda-bld/pytorch_1556653114079/work/caffe2/core/context_gpu.cu:322: initialization error
</code></pre>
<p>Any thoughts?</p>

ptrblck: <p>Are you preloading the data (and target) tensors onto the GPU before passing them to <code>Dataset</code>?<br>
This might cause multiple initializations of the CUDA context, which might yield this error.<br>
You could try to set <code>num_workers=0</code> in your <code>DataLoader</code>, use CPU tensors, or try to set the <code>spawn</code> method for <code>multiprocessing</code>.</p>

Vimos: <p>Thanks, setting <code>num_workers=0</code> works well.</p>
Asynchronous Allreduce gradients

hpc-unex: <p>Hello, I want to know how the MPI_Allreduce works in asynchronous mode when the gradients are calculated. Suppose we have 3 processes. If the first epoch is finished and only one process have update the gradients, when it takes the gradients from a shared buffer it takes NaN in the SUM of the process that havent finished ?? Im pretty lost here because Allreduce is a blocking primitive but the training doesnt stop for it.</p>

pietern: <p>What do you mean by “training doesn’t stop”?</p>
<p>Also, how do you run allreduce in asynchronous mode? The synchronization done by <code>torch.nn.parallel.DistributedDataParallel</code> is done implicitly, when you make autograd compute gradients for your model parameters. It doesn’t return until all the allreduce calls have finished (or in the case of CUDA tensors, until all the NCCL allreduce kernels have been queued).</p>