What if I load data in __init__ function of LightningDataModule
Hi,
I see the doc describing the functions of LightningDataModule.
https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/datamodules.html#Defining-The-MNISTDataModule
Here is my thinking. If some variable, e.g., a transform, can be defined in init function, and later shared across different GPUs. Theoretically, if we load data in init, the data should also be able to transfer to different GPUs similarly. In the case of a single machine with multiple GPUs, the data will be copied multiple times in the memory. In the case of multiple machines, the data will broadcast through the network from the main node to other nodes. Broadcasting large data through networks may have efficiency issue, which is why we had better load data in the setup function.
Please let me know whether my analysis is correct or not. Basically, I am not clear about how the variables, e.g., i.e. self.something, defined in init are shared across multiple GPUs/machines. Thanks!
@zhiqiangdon

lightning just runs prepare_data on the main process before the distributed process actually starts so there is no blocking happening behind the scenes.
To tackle this issue we have prepare_data_per_node. A node is just a machine. If they share the disk then prepare_data_per_node should be set to False.
User runs the __init__ function when they initialize the DataModule, lightning just send to across devices.