Error with DistributedDataParallel()

gnadaf: <p>Hi,<br>
I am trying to wrap DistributedDataParallel() with the Transformer model.</p>
<p>But, I am facing the below error</p>
<p><strong>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘torchtext.data.example.Example’&gt;</strong></p>
<p>Code:</p>
<blockquote>
<p>def train_1(args):<br>
<span class="hashtag-raw">#init_process_group</span><br>
<span class="hashtag-raw">#rank</span> = args.nr * args.gpus + gpu<br>
rank = int(os.environ[‘LOCAL_RANK’])<br>
gpu = torch.device(f’cuda:{rank}')<br>
torch.distributed.init_process_group(backend=‘nccl’, init_method=‘env://’)<br>
torch.cuda.set_device(gpu)</p>
<pre><code>TEXT = torchtext.data.Field(tokenize=get_tokenizer("basic_english"),
                            init_token='&lt;sos&gt;',
                            eos_token='&lt;eos&gt;',
                            lower=True)
train_txt, val_txt, test_txt = torchtext.datasets.WikiText2.splits(TEXT)
TEXT.build_vocab(train_txt)

batch_size = 20
eval_batch_size = 10

sampler =  torch.utils.data.distributed.DistributedSampler(train_txt);
loader = torch.utils.data.DataLoader(train_txt, shuffle=(sampler is None), sampler=sampler)

bptt = 35


ntokens = len(TEXT.vocab.stoi)  # the size of vocabulary
emsize = 200  # embedding dimension
nhid = 200  # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2  # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 2  # the number of heads in the multiheadattention models
dropout = 0.2  # the dropout value
model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout).to(gpu)

#DDP
model = nn.parallel.DistributedDataParallel(model,device_ids=[gpu])

criterion = nn.CrossEntropyLoss()
lr = 5.0  # learning rate
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)


def train():
    model.train()  # Turn on the train mode
    total_loss = 0.
    start_time = time.time()
    ntokens = len(TEXT.vocab.stoi)
    for i, (data, targets) in enumerate(loader):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output.view(-1, ntokens), targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()
        total_loss += loss.item()
        log_interval = 200
        if batch % log_interval == 0 and batch &gt; 0:
            cur_loss = total_loss / log_interval
            elapsed = time.time() - start_time
            
            print('| epoch {:3d} | {:5d}/{:5d} batches | '
                'lr {:02.2f} | ms/batch {:5.2f} | '
                'loss {:5.2f} | ppl {:8.2f} | device{:3d}'.format(
                epoch, batch, len(train_data) // bptt, scheduler.get_lr()[0],
                            elapsed * 1000 / log_interval,
                cur_loss, math.exp(cur_loss), torch.cuda.current_device()))
            total_loss = 0
            start_time = time.time()
</code></pre>
</blockquote>

Abhilash_Srivastava: <p>Please share the full error log. Most likely this error is not due to DistributedDataParallel.</p>

gnadaf: <p>Traceback (most recent call last):<br>
File “standard_ddp_7.py”, line 219, in <br>
main()<br>
File “standard_ddp_7.py”, line 216, in main<br>
train_1(args)<br>
File “standard_ddp_7.py”, line 174, in train_1<br>
train()<br>
File “standard_ddp_7.py”, line 127, in train<br>
for i, (data, targets) in enumerate(loader):<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 435, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 475, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py”, line 85, in default_collate<br>
raise TypeError(default_collate_err_msg_format.format(elem_type))<br>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘torchtext.data.example.Example’&gt;<br>
Traceback (most recent call last):<br>
File “standard_ddp_7.py”, line 219, in <br>
main()<br>
File “standard_ddp_7.py”, line 216, in main<br>
train_1(args)<br>
File “standard_ddp_7.py”, line 174, in train_1<br>
train()<br>
File “standard_ddp_7.py”, line 127, in train<br>
for i, (data, targets) in enumerate(loader):<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 435, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 475, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py”, line 85, in default_collate<br>
raise TypeError(default_collate_err_msg_format.format(elem_type))<br>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘torchtext.data.example.Example’&gt;<br>
Traceback (most recent call last):<br>
File “/home/nadaf/anaconda3/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/nadaf/anaconda3/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/distributed/launch.py”, line 260, in <br>
main()<br>
File “/home/nadaf/anaconda3/lib/python3.8/site-packages/torch/distributed/launch.py”, line 255, in main<br>
raise subprocess.CalledProcessError(returncode=process.returncode,<br>
subprocess.CalledProcessError: Command ‘[’/home/nadaf/anaconda3/bin/python’, ‘-u’, ‘standard_ddp_7.py’]’ returned non-zero exit status 1.</p>

gnadaf: <p>Hi <a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a>,<br>
Attached error log</p>

Abhilash_Srivastava: <aside class="quote no-group quote-modified" data-username="gnadaf" data-post="1" data-topic="103442">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/cdc98d/48.png" class="avatar"> gnadaf:</div>
<blockquote>
<pre><code class="lang-auto">    for i, (data, targets) in enumerate(loader):
        optimizer.zero_grad()
        output = model(data)
        ....
</code></pre>
</blockquote>
</aside>
<p>Try modifying this to:</p>
<pre><code class="lang-auto">    for i, batch in enumerate(loader):
        data, targets = batch.text, batch.target
        optimizer.zero_grad()
        output = model(data)
        ....
</code></pre>

gnadaf: <aside class="quote no-group" data-username="Abhilash_Srivastava" data-post="5" data-topic="103442">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhilash_srivastava/48/33127_2.png" class="avatar"> Abhilash_Srivastava:</div>
<blockquote>
<p>Try modifying this to:</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a></p>
<p>I modified the code as you suggested, but still facing the same error</p>

gnadaf: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<a class="mention" href="/u/alband">@albanD</a><br>
Could you please have a look</p>

ptrblck: <p>The error doesn’t seem to be related to DDP, but the <code>DataLoader</code> and <code>torchtext</code>:</p>
<pre><code class="lang-python">TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘torchtext.data.example.Example’&gt;
</code></pre>
<p>I’m unsure how the <code>torchtext.data.example.Example</code> class is implemented, but could you try to return tensors in the <code>Dataset.__getitem__</code> instead of this object?</p>

gnadaf: <p>I am loading datset like this</p>
<blockquote>
<p>train_txt, val_txt, test_txt = torchtext.datasets.WikiText2.splits(TEXT)</p>
</blockquote>

gnadaf: <p><a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a> <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Still facing the same error:</p>
<p>I do not see any online example where we wrap the transformer model with DDP.</p>
<p>Transformer mode:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" target="_blank" rel="noopener nofollow ugc">Sequence-to-Sequence Modeling with nn.Transformer and TorchText — PyTorch...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Abhilash_Srivastava: <p>Are you able to run and train your model successfully (say for a smaller dataset) without DDP?</p>

gnadaf: <p>Yes, without DDP, the model works fine</p>