CUDA_VISIBLE_DEVICES=0,1 python xxx.py, not work

Yeah: <p>I have 2 GPUs,</p>
<p>when I want to use one of GPUs to train, with the following code, both work.<br>
CUDA_VISIBLE_DEVICES=0 python xxx.py,<br>
CUDA_VISIBLE_DEVICES=1 python xxx.py,</p>
<p>However, when I want to use 2 GPUs to train, with the following code,<br>
CUDA_VISIBLE_DEVICES=0,1 python xxx.py,<br>
it doesn’t work anymore. Only the default GPU:0 is used for training, when the memory of GPU:0 run out of, the training will be terminated with error ‘out of memory’. The GPU:1 is lying idle and not be used. Why?</p>
<p>the GPU information is showed in the following:<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  Tesla K80           Off  | 00006B71:00:00.0 Off |                    0 |<br>
| N/A   54C    P0    82W / 149W |   8772MiB / 11441MiB |     40%      Default |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  Tesla K80           Off  | 000096F1:00:00.0 Off |                    0 |<br>
| N/A   25C    P8    32W / 149W |     11MiB / 11441MiB |      0%      Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    0      2495      C   python                                      8759MiB |<br>
±----------------------------------------------------------------------------+</p>
<p>Could someone explain this situation? and What should I do to love that?</p>
<p>Thanks in advance and really appreciate for any feedback.</p>

JuanFMontesinos: <p>that is a cuda enviroment variable.<br>
It’s not pytorch.<br>
What it means is that you can call any bash command preceded by that. It basically manages which devices the called process can see.<br>
So in your case it means that your python kernel can only see one of the gpus. Both if you set 0,1. But it doesn’t mean that pytorch is gonna train on both automatically.<br>
To train on several gpus you should use modules like <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html</a></p>

Yeah: <p>Thanks for your information. Actually I have tried</p>
<p>model.cuda()<br>
model = torch.nn.DataParallel(model, device_ids=[0, 1])</p>
<p>but there is always the following errors:</p>
<p>File “/home/speech/treelstm_nlg/HRED/ContextLSTMLayer.py”, line 21, in forward<br>
output, (hn, cn) = self.rnn(x, (h_0, c_0))<br>
File “/home/speech/treelstm_nlg/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/speech/treelstm_nlg/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 579, in forward<br>
self.check_forward_args(input, hx, batch_sizes)<br>
File “/home/speech/treelstm_nlg/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 534, in check_forward_args<br>
‘Expected hidden[0] size {}, got {}’)<br>
File “/home/speech/treelstm_nlg/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 196, in check_hidden_size<br>
raise RuntimeError(msg.format(expected_hidden_size, list(hx.size())))<br>
RuntimeError: Expected hidden[0] size (1, 64, 300), got [1, 128, 300]</p>
<p>why for that?</p>
<p>By the way, what is the difference between DataParallel and DistributedDataParallel, I seems the official documents suggest more DistributedDataParallel. In my case, some computation code is written by myself, so I am not sure, which is the best choice.</p>
<p>Thanks in advance!! Really appreciate for any feedback.</p>

JuanFMontesinos: <p>Hmmm So basically it does a copy of the model in each gpu. Then, when you call forward, it takes the batch, created as many chunks as gpus a send them to the corresponding one.</p>
<p>it assumes the batch dimension is the dimension 0. I think this issue can happens due to he fact rnn operates as temporal dimension first. Soo how is the input to your network sorted?</p>
<p>WRT DistributedDP I don’t really know why. It wasn’t the case some versions ago. It seems the performance of DP is worse (<a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead</a>)</p>
<p>Anyway both will probably work in a similar way.</p>

Yeah: <p>In my case, batch_first=True. Isn’t it right?</p>

JuanFMontesinos: <p>Hmmm then I don’t really know.<br>
I mean, the module does nothing but splitting the batch into two chunks.<br>
Are you sure the error doesn’t raise without it?</p>
<p>Can you try to post a standalone script?</p>