AssertionError while using DataParallelModel
Author: sajidrahman
Pull request: no
Labels: name: wontfix

Hi,

I'm trying to use _Load Balancing during multi-GPU_ environment. I'm following the tutorial by @thomwolf published at [medium](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255). I'm fine-tuning GPT-2 small for a classification task. Here're the steps I've followed so far:

1. Copy [parallel.py](https://gist.github.com/thomwolf/7e2407fbd5945f07821adae3d9fd1312?source=post_page---------------------------) in local directory
2. Add `from torch.nn.parallel.distributed import DistributedDataParallel` in parallel.py file (otherwise getting an error 'DistributedDataParallel' not found)
3. After loading GPT2Pretrained model, define the parallel model:
```
model = DataParallelModel(model, device_ids=[0, 1])  
parallel_loss = DataParallelCriterion(model, device_ids=[0,1])
```
4. Now during training, got the following error. Complete stacktrace is as follows:

>  AssertionError                            Traceback (most recent call last)
> <ipython-input-135-05384873e022> in <module>
>      19 
>      20 #         losses = model(input_ids, mc_token_ids, lm_labels=lm_labels, mc_labels=mc_labels)
> ---> 21         losses = parallel_loss(input_ids, mc_token_ids, lm_labels=lm_labels, mc_labels=mc_labels)
>      22 
>      23         lm_loss, clf_loss = losses
> 
>  ~/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
> 
>     491             result = self._slow_forward(*input, **kwargs)
>     492         else:
> --> 493             result = self.forward(*input, **kwargs)
>     494         for hook in self._forward_hooks.values():
>     495             hook_result = hook(self, input, result)
> 
>  ~/github_repos/pytorch-pretrained-BERT/examples/parallel.py in forward(self, inputs, *targets, **kwargs)
> 
>     158             return self.module(inputs, *targets[0], **kwargs[0])
>     159         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
> --> 160         outputs = _criterion_parallel_apply(replicas, inputs, targets, kwargs)
>     161         #return Reduce.apply(*outputs) / len(outputs)
>     162         #return self.gather(outputs, self.output_device).mean()
> 
>  ~/github_repos/pytorch-pretrained-BERT/examples/parallel.py in _criterion_parallel_apply(modules, inputs, targets, kwargs_tup, devices)
> 
>     165 
>     166 def _criterion_parallel_apply(modules, inputs, targets, kwargs_tup=None, devices=None):
> --> 167     assert len(modules) == len(inputs)
>     168     assert len(targets) == len(inputs)
>     169     if kwargs_tup:
> 
> AssertionError: 

From the stacttrace, I'm not sure why module length needs to be equal of inputs length. Am I missing something here? I'm using Python 3.6 with PyTorch version 1.1.0. Any help/pointers will be highly appreciated. Thanks!

Comments:
You don't need to use this method here because the models have built-in losses computation.
Just feed the labels and you will get the loss back (see the doc/docstrings of the models).

Hi @thomwolf, thanks for the suggestion. After following your advice, I'm not getting the error anymore, but now I'm a bit confused about the `backward()` pass. Now the **losses** variable contains a list of tensors of loss calculations per gpu and I'm not sure how I can enforce each individual model sitting in each gpu to perform backprop. Following is a sample code of what I've done so far and the sample output:

```
losses:[[tensor(98.5968, device='cuda:0', grad_fn=<NllLossBackward>), tensor(0.7206, device='cuda:0', grad_fn=<BinaryCrossEntropyWithLogitsBackward>)], [tensor(100.5673, device='cuda:1', grad_fn=<NllLossBackward>), tensor(0.6629, device='cuda:1', grad_fn=<BinaryCrossEntropyWithLogitsBackward>)]] 
 lm_loss: (tensor(98.5968, device='cuda:0', grad_fn=<NllLossBackward>), tensor(100.5673, device='cuda:1', grad_fn=<NllLossBackward>))
clf_loss:(tensor(0.7206, device='cuda:0', grad_fn=<BinaryCrossEntropyWithLogitsBackward>), tensor(0.6629, device='cuda:1', grad_fn=<BinaryCrossEntropyWithLogitsBackward>))

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-55-10ac591c2408> in <module>
     26         lm_loss, clf_loss = zip(*losses)
     27         print('losses:{} \n lm_loss: {}\nclf_loss:{}\n'.format(losses, lm_loss, clf_loss))
     28 
---> 29         loss = (args.lm_coef * lm_loss.to(device) + clf_loss.to(device)).to(device)
     30 
     31         print(loss)

AttributeError: 'tuple' object has no attribute 'to'
```
Obviously I can deal with this 'tuple' error, but I'm confused what should I do next with this? Should I call `loss.backward()` per each cuda devices? How will I then gather gradient values in that case?   Please excuse me for any naive assumptions I'm making here. Your input would be highly appreciated :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @thomwolf , I also experience this imbalanced GPU usage when using the trainer function. Can I ask the reason that the DataParallelModel that you discussed on the Medium post, is not applied as default in the trainer function? Thank you.