System is getting hang while using multiple gpu`s in inference

username: Billie
cooked: <p>I have trained my model on a single <code>gpu</code> machine while training i have <code>wrapped</code> my model with <code>torch.nn.DataParallel</code> class. I called the training with the command <code>CUDA_VISIBLE_DIVICES=0 python train.py</code>.</p>
<p>Training went fine but when i tried to do inference on this model from the command <code>CUDA_VISIBLE_DIVICES=0,1 python test.py</code> it’s getting hang.</p>
<p>Inference is working fine when i call single gpu with command <code>CUDA_VISIBLE_DIVICES=0 python test.py</code>.</p>

username: ptrblck
cooked: <p>Was your setup working before using multiple GPUs?<br>
If so, do you know what changed, e.g. did you update the driver or anything else?<br>
If not, are you able to communicate between different devices and e.g. send a tensor from A to B?</p>
<pre><code class="lang-python">x = torch.randn(1, device="cuda:0")
y = x.to("cuda:1")
</code></pre>
<p>If not, could you check if IOMMU is enabled and disable it if needed as described <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">here</a>?</p>
<p>Btw. we recommend using <code>DistributedDataParallel</code> for a better performance as <code>DataParallel</code> is also in maintenance mode.</p>

username: Billie
cooked: <p>The setup was working well before using multiple GPUs, and GPUs are able to communicate between devices.</p>
<p>if I do not give the environment variable <code>CUDA_VISIBLE_DIVICES</code> then it throws below error:</p>
<pre><code class="lang-auto">ERROR:root:Caught TypeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() missing 2 required positional arguments: 'input' and 'text'
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/LPR/__init__.py", line 138, in feedforward
    batch_plates = self.license_plate.fetch_plate_number(batch, self.long_size)
  File "/opt/conda/lib/python3.8/site-packages/LPR/__init__.py", line 102, in fetch_plate_number
    detected_text_dict = self.ocr.fetch_text(img_association[item])
  File "/opt/conda/lib/python3.8/site-packages/LPR/OCR/__init__.py", line 172, in fetch_text
    preds = self.model(image, text_for_pred, is_train=False) #torch.unsqueeze(image,0)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
    output.reraise()
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
TypeError: Caught TypeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() missing 2 required positional arguments: 'input' and 'text'

DEBUG:root:Entity ended here with no valid Licence plate detection and id is: my_id
</code></pre>

username: ptrblck
cooked: <p>That’s an interesting error. Does a single GPU run? If so, could you try to use <code>DDP</code> as it’s also the supported distributed backend?</p>

username: Billie
cooked: <p>it is running fine in single GPU machine no matter if i use the environment variable <code>CUDA_VISIBLE_DIVICES</code> or not.</p>