Expectations for custom data parallel implementations
I have a need to use a custom [DistributedDataParallel](https://pytorch.org/docs/stable/notes/ddp.html) implementation. I'd like to do this with the existing DDPBackend today.
With Lightning, the API and docs are unclear as to whether I need to extend LightningDistributedDataParallel, or if I can directly extend torch DistributedDataParallel.
The LightningModule API docs suggest configure_ddp should work with torch DistributedDataParallel: https://pytorch-lightning.readthedocs.io/en/latest/lightning-module.html#configure-ddp
However, there are spots in Lightning which rely on checking isinstance of the custom Lightning overrides: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/model_connector.py#L31-L34 1
The Lightning DDP also forwards calls to train/val/test step. Is this a requirement for custom DDP implementations when used with Lightning?
TLDR: should I subclass LightningDistributedDataParallel or DistributedDataParallel when I implement the model hook for configure_ddp?
Also asked here: https://forums.pytorchlightning.ai/t/expectations-for-custom-data-parallel-implementations/162
@ananthsub Not sure if you have seen it already, there is now a DDP Plugin in which you can override the DistributedDataParallel. It can be passed into the trainer via plugins list.
https://pytorch-lightning.readthedocs.io/en/latest/plugins.html?highlight=DDPPlugin