when using multiple GPUs, `loss.mean()` may have subtle bias 
Author: Lvzhh
Pull request: no
Labels: name: wontfix

The problem is that, when the input is distributed to multiple GPUs, the input on each GPU may have different `batch_size`. 

For example, if you have 2 GPUs and the total batch_size is 13, then the `batch_size` for each GPU will be 7 and 6 respectively, `loss.mean()` will not give the exact loss. Although it may have little influence on the training of the model, it is not the exact result.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L1006-L1007

Comments:
You could fix it with something like this
```
batch_size = torch.tensor(data.shape[1]).to(device)
dist.all_reduce(batch_size, op=dist.ReduceOp.SUM)
dist.all_reduce(loss, op=dist.ReduceOp.SUM)
mean_loss = loss/batch_size
```

Thanks for your solution. I don't think it could fix it, because the `loss` returned by each GPU is already averaged over its `batch_size`.

@Lvzhh the examples (like `run_squad.py`) only work in two settings for multi-gpu:

- using `DataParallel` (not distributed, triggered when you simply run the script on a multi-gpu server): in this case the batch size should be a multiple of the number of gpu or an error is thrown. This is the case in which this `loss = loss.mean()` will be triggered. No risk of differing batch sizes here.
- using `DistributedDataParallel` with a *single* GPU assigned to each process (triggered when you run the example using PyTorch `distributed.launch` script for instance): this only works when 1 GPU is assigned to each process (see [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L853-L857)). In this case `loss = loss.mean()` is not triggered. There is no averaging of the loss over the GPU. What happens here is that it's the gradients which are averaged (read PyTorch [DDP doc](https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel) for instance). The batch size should be the same for each script so no problem here also.

So I don't think the problem you are describing will show up in practice.


@thomwolf I'm using the first multi-gpu setting `DataParallel`. And I tried to find some code in `run_squad.py` to ensure that the `batch_size` be a multiple of the number of gpu, but I didn't find.

And I just run this command [here](https://github.com/huggingface/pytorch-pretrained-BERT#squad) with `--train_batch_size 13` using 2 GPUs, and print the size of `start_logits` and its device, the first dimension (`batch_size`) on each GPUs are actually different (7 vs 6).

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/pytorch_pretrained_bert/modeling.py#L1204-L1206

Oh you are right, this error is not there anymore.
We should probably just check for this again, I'll fix the examples.

`DataParallel` is in the process of being replaced by `DistributedDataParallel` which is pretty much always faster due to releasing the GiL so maybe you should try DDP.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.