What is the best practice to share a massive CPU tensor over multiple processes in pytorch-lightning DDP mode (read-only + single machine)?
Hi everyone, I wonder what is the best practice to share a massive CPU tensor over multiple processes in pytorch-lightning DDP mode (read-only + single machine)?
I think torch.Storage.from_file with share=True may suit my needs, but I can’t find a way to save storage and read it as a tensor. (see here for details)
I also tried to copy training data to /dev/shm (reference) and run DDP with 8 GPUs, but nothing is different. The memory usage when running with 8 GPUs is the same as before, but I tested with a single process, loading the dataset may occupy more than 1 GB of memory. Am I missing something here?
For torch.shared_memory, how should I pass the same reference to all processes in pytorch-lightning pure DDP mode?
Thank you.
I found that torch.Storage.from_file suits my needs and it can reduce the memory usage in my Lightning DDP program.
For the way to create a storage file, see here.