question: What are ones options for manually defining the parallelization?
context: (Q1) Does PyTorch Lightning enable parallelization across multiple dimension or does it only allow data parallelism?
  The FlexFlow implements the parallelism across 4 different dimensions ("SOAP": the sample, operator, attribute and parameter dimensions). (Q2) Over which of these does PyTorch-Lightning do parallelization? (Q3) Does PyTorch-Lightning's API give an option to manually control the parallelization for each and every layer individually?
answer: Dear @roman955b,
  1 ) Currently, Lightning automatically implement distributed data parallelism. However, we are currently working on making manual parallelization for users who want deeper control of the parallelisation schema.
  2 ) Lightning supports only (S, P) with DeepSpeed, FSDP integrations.
  3 ) Yes, we are currently working on this. Here is an issue to track the conversation #9375
  Best,
  T.C