Can't utilise more than one GPU with transformers model

nima_rafiee: <p>Hi.<br>
I’m trying to use MarianModels from Huggingface library for back-translation. I’m using 8 GPU’s however nvidia-smi shows the utilization of only one GPU although I used nn.Dataparallel().  Can anyone help me with this issue?</p>
<pre><code class="lang-auto">import torch
import torchtext
from transformers import MarianMTModel, MarianTokenizer
</code></pre>
<pre><code>target_langs = ['fr,wa,frp,oc,ca,rm,lld,fur,lij,lmo,es,pt,gl,lad,an,mwl,it,co,nap,scn,vec,sc,ro,la']

def translate(texts, model, tokenizer, language="fr"):
    
    with torch.no_grad():
        template = lambda text: f"{text}" if language == "en" else f"&gt;&gt;{language}&lt;&lt; {text}"
        src_texts = [template(text) for text in texts]
        encoded = tokenizer.prepare_seq2seq_batch(src_texts, 
                                                      truncation=True, 
                                                      max_length=300, return_tensors="pt").to(device)   
        translated = model.module.generate(**encoded).to(device)
        translated_texts = tokenizer.batch_decode(translated, skip_special_tokens=True)
        return translated_texts


def back_translate(texts, source_lang="en", target_lang="fr"):
    # Translate from source to target language
    fr_texts = translate(texts, target_model, target_tokenizer, 
                         language=target_lang)

    # Translate from target language back to source language
    back_translated_texts = translate(fr_texts, en_model, en_tokenizer, 
                                      language=source_lang)
    
    return back_translated_texts



target_model_name = 'Helsinki-NLP/opus-mt-en-de'
target_tokenizer = MarianTokenizer.from_pretrained(target_model_name)
target_model = MarianMTModel.from_pretrained(target_model_name)

en_model_name = 'Helsinki-NLP/opus-mt-de-en'
en_tokenizer = MarianTokenizer.from_pretrained(en_model_name)
en_model = MarianMTModel.from_pretrained(en_model_name)

target_model = nn.DataParallel(target_model)    
target_model = target_model.to(device) # same performance  if I add .half()
target_model.eval()

en_model = nn.DataParallel(en_model)    
en_model = en_model.to(device)# same performance if I add .half()
en_model.eval()    

for i , (x1 , x2 , label) in enumerate(loader):
    with torch.no_grad():
        ## x1 and x2 are batches of strings. 
        bk_x1 = back_translate(x1, source_lang="en", target_lang=np.random.choice(target_langs))   
        bk_x2 = back_translate(x2, source_lang="en", target_lang=np.random.choice(target_langs))
</code></pre>
<pre><code class="lang-auto">



here are GPU's performances:   low utilization due to small batch size 16 but if I increase the batch size I got Cuda out of memory error. also, I can see only one GPU is used for processing so might be that the Marian model can not be parallelized correctly. which explains the reason for Cuda's out of memory error and slow performance.  if so what would be the solution?

     |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GeForce GTX 108...  Off  | 00000000:1B:00.0 Off |                  N/A |
    | 42%   78C    P2   199W / 250W |   9777MiB / 11178MiB |     91%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   1  GeForce GTX 108...  Off  | 00000000:1C:00.0 Off |                  N/A |
    | 29%   36C    P8    10W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   2  GeForce GTX 108...  Off  | 00000000:1D:00.0 Off |                  N/A |
    | 31%   36C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   3  GeForce GTX 108...  Off  | 00000000:1E:00.0 Off |                  N/A |
    | 35%   41C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   4  GeForce GTX 108...  Off  | 00000000:3D:00.0 Off |                  N/A |
    | 29%   34C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   5  GeForce GTX 108...  Off  | 00000000:3F:00.0 Off |                  N/A |
    | 30%   31C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   6  GeForce GTX 108...  Off  | 00000000:40:00.0 Off |                  N/A |
    | 31%   38C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   7  GeForce GTX 108...  Off  | 00000000:41:00.0 Off |                  N/A |
    | 30%   37C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A     58780      C   python                          10407MiB |
    |    1   N/A  N/A     58780      C   python                              0MiB |
    |    2   N/A  N/A     58780      C   python                              0MiB |
    |    3   N/A  N/A     58780      C   python                              0MiB |
    |    4   N/A  N/A     58780      C   python                              0MiB |
    |    5   N/A  N/A     58780      C   python                              0MiB |
    |    6   N/A  N/A     58780      C   python                              0MiB |
    |    7   N/A  N/A     58780      C   python                              0MiB |
    +-----------------------------------------------------------------------------+






</code></pre>
<p>FYI: I’m using</p>
<pre><code class="lang-auto">pytorch 1. 1.7.0
transformers 4.0.1
cudda 10.1
</code></pre>
<pre><code class="lang-auto"></code></pre>

nima_rafiee: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>.  I wonder if you possibly have any idea regarding my issue.</p>

ptrblck: <p>Are you seeing no utilization on the other GPUs at all or just a low utilization?<br>
In the latter case, note that <code>nn.DataParallel</code> has some drawbacks, such as a higher memory usage on the default device, as well as worse performance than <code>DistributedDataParallel</code>, which is why we recommend to use DDP.</p>

nima_rafiee: <p>thanks for your reply. According to what nvidia-smi shows the other gpu’s have zero utilization. I tried to use the <code>DistributedDataParallel</code>  the utilization problem is solved but seems there are problems with 1D batch normalization in distributed training ( I mentioned mine in <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-when-using-distributed-training/106969">106969</a> but I see others also have the same problem)</p>