Repository: OpenNMT/OpenNMT-py

Issue:
Difference in training results between single GPU and multi-GPU
For single GPU and multi-GPU training, the single card is 4% higher than the multi-card training. What is the cause?
Here is my configuration：
<img width="150" alt="image" src="https://user-images.githubusercontent.com/109410944/226503405-f70fa165-2fe6-4c23-b1bd-e4d4e7610c32.png">
<img width="207" alt="image" src="https://user-images.githubusercontent.com/109410944/226503453-a0e1ba34-47ff-482e-86ad-13602c338ef1.png">



Discussion:
what do you mean by "is 4% higher" ?
Forgot to write that BLEU is 4% higher on a single card than on a multi-card
is it just one run each or same obseravtion over several runs ? can you post the end of the logs for each ? (with last steps or train + validation)
By the way, at the beginning of multi-card training, you did not average batch_size and then distribute data like TORCH.DISTRIBUTED. Each card has the same initial batch_size, right?


On the left is the final training result of a single card, and on the right is the multi-card：
<img width="623" alt="image" src="https://user-images.githubusercontent.com/109410944/226545929-7f4553ae-6575-4cb4-9a37-4a23a07368d6.png">

what version of OpenNMT-py do you use ?
master

if you're 100% sure of the process (same data -train - valid) so on, my suggestion would be to test v3.0.4 to see if it comes from a recent commit (I changed striding since 3.0.4).
Another test could be to revert https://github.com/OpenNMT/OpenNMT-py/issues/2295 back to async_op=True
to see if this has a real impact.
If you have time to test these 2 leads, it might be helpful for others.

Ok, thank you for your reply. I'll have a try
Can you explain how multi-card data parallelism works in your latest version? thank you
I am not sure what you are asking. We have always been using torch distributed, which spawn one process per GPU. In V3, the new Dataloader will itself spawn another num_workers processes per GPU. The only change made recently is the striding.
stride = num_workers x nb_gpu
instead of reading data 1 line and skip stride lines to read the next in each process, we now
read stride lines in each process.
This is just to be able to read consecutive lines when reading "doc level" datasets.
batch_size is divided into multiple copies and distributed to each GPU model in parallel. How do you do it in the code? I can't find it.
https://forum.opennmt.net/t/number-of-samples-in-one-training-step-together-with-accumulation/4348

Resolving patch:
diff --git a/onmt/utils/distributed.py b/onmt/utils/distributed.py
--- a/onmt/utils/distributed.py
+++ b/onmt/utils/distributed.py
@@ -52,7 +52,7 @@ def all_reduce_buffer():
             offset += numel
 
         # all-reduce and rescale
-        torch.distributed.all_reduce(buffer_t[:offset], async_op=True)
+        torch.distributed.all_reduce(buffer_t[:offset], async_op=False)
         buffer_t.div_(rescale_denom)
 
         # copy all-reduced buffer back into tensors
@@ -68,7 +68,7 @@ def all_reduce_buffer():
         # print(filled, sz)
         if sz > buffer_size:
             # tensor is bigger than buffer, all-reduce and rescale directly
-            torch.distributed.all_reduce(t, async_op=True)
+            torch.distributed.all_reduce(t, async_op=False)
             t.div_(rescale_denom)
         elif filled + sz > buffer_size:
             # buffer is full, all-reduce and replace buffer with grad


Test patch: