What are ones options for manually defining the parallelization?
(Q1) Does PyTorch Lightning enable parallelization across multiple dimension or does it only allow data parallelism?
The FlexFlow implements the parallelism across 4 different dimensions ("SOAP": the sample, operator, attribute and parameter dimensions). (Q2) Over which of these does PyTorch-Lightning do parallelization? (Q3) Does PyTorch-Lightning's API give an option to manually control the parallelization for each and every layer individually?
Dear @roman955b,
1 ) Currently, Lightning automatically implement distributed data parallelism. However, we are currently working on making manual parallelization for users who want deeper control of the parallelisation schema.
2 ) Lightning supports only (S, P) with DeepSpeed, FSDP integrations.
3 ) Yes, we are currently working on this. Here is an issue to track the conversation #9375
Best,
T.C