Pytorch GPU utilization zero

sajastu: <p>Hi,</p>
<p>I’m working with Huggingface Transformers library to run pegasus model. It appears that the trainer class of transformers automatically handles the multi gpu training when the GPU devices are known (i.e., CUDA_VISIBLE_DEVICES flag). However, when I’m using multi-gpu for training, only one of GPUs is in near to 100% utilization and others are literally zero. Following is the command + nvidia output. I’m using pytoch 1.6.0 cuda 10.</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1,2 python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google/pegasus-reddit_tifu \
    --do_predict \
    --train_file $DS_BASE_DIR/train.json \
    --validation_file $DS_BASE_DIR/validation.json \
    --test_file $DS_BASE_DIR/test.json \
    --output_dir /home/code-base/user_space/saved_models/pegasus/ \
    --per_device_train_batch_size=2 \
    --per_device_eval_batch_size=2 \
    --overwrite_output_dir \
    --predict_with_generate \
    --text_column text \
    --summary_column summary \
    --num_beams 5
</code></pre>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   76C    P0   290W / 300W |  16082MiB / 16160MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   43C    P0    72W / 300W |   4060MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   43C    P0    72W / 300W |   4044MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
</code></pre>
<p>Any hint/advice?</p>

ptrblck: <aside class="quote no-group" data-username="sajastu" data-post="1" data-topic="123982">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sajastu/48/14519_2.png" class="avatar"> sajastu:</div>
<blockquote>
<p>It appears that the trainer class of transformers automatically handles the multi gpu training when the GPU devices are known (i.e., CUDA_VISIBLE_DEVICES flag).</p>
</blockquote>
</aside>
<p>Could you link to the trainer code, which would implement this logic?<br>
Based on the output of <code>nvidia-smi</code> it doesn’t seem to be the case or work as intended.</p>