Better undestanding how data is loded in datamodule setup method for multi GPU setting in NLP
I want to better understand the setup and prepare_data methods in multi gpu scenariu in context of NLP and text processing.
I have prepared the DataModule which process json line file with pairs of sentence for translation task. The file contains 10M lines.
In prepare_data() I open the data file, read it to memory, do some basic filtering (remove to long sentences and do some sorting based on length in order to group similar length sentences together) then I write it to another file (filtered_data.json).
Next in the setup() method I read filtered_data.json and split it to train and valid.
I can perform split deterministically so train and valid splits will always have the same elements or I can split randomly then each GPU process will have a different train and valid sets.
When using it in multi-gpu (2 GPUs) each process will have its own copy of the train and valid set (am I right?). Which approach is better in context data utilization, random or deterministically?
I do not fully understand how distributed DataLoader handles these two approaches? Could someone explain it in detail?
If data are loaded deterministically then all GPU processes, especially forward and backward pass will return the same values (for gpu 1 and 2), it is efficient? How gradients are merged and how network weight updates will be performed.
Or maybe the second (random split) approach is better because gradients computed on different samples and merged from 2 gpus will result in a better estimation of the true gradient.
I have prepared the DataModule which process json line file with pairs of sentence for translation task. The file contains 10M lines.
In prepare_data I open the data file, read it to memory, do some basic filtering (remove to long sentences and do some sorting based on length in order to group similar length sentences together) then I write it to another file (filtered_data.json).

Do all of that either offline in a different script, or do it in the prepare_data hook.

Next in the setup method I read filtered_data.json and split it to train and valid.

Sounds good. Each GPU/node will run the same, so you will have the same train and val split in all of them (initially). Don't split the data differently for each GPU, that part will be done by the DistributedSampler [1].

I do not fully understand how distributed DataLoader handles these two approaches? Could someone explain it in detail?

Lightning takes your DataLoader and adds a DistributedSampler. The DS knows on which GPU it is and will sample only a portion of your data on one GPU and another portion on the other GPU. Each GPU sees a different split of train and a different split of val.

If data are loaded deterministically then all GPU processes, especially forward and backward pass will return the same values (for gpu 1 and 2), it is efficient? How gradients are merged and how network weight updates will be performed.

As explained above, the dataloader on each GPU will return different samples on each GPU automatically. Each GPU will have the same network weights, uses different data to compute gradients, then gradients are averaged so each GPU gets the same update and starts with the same weights for the next forward/backward [2].

Or maybe the second (random split) approach is better because gradients computed on different samples and merged from 2 gpus will result in a better estimation of the true gradient.

Yes, again this is automatically done for you.
References:
[1] DistributeSampler
[2] Distributed Training in PyTorch
[3] Multi-GPU training in PyTorch