Remove parameters from autograd backward hook
Hello,
I am trying to remove some layers from DistributedDataParallel to prevent them being synchronized between devices.
I spent last 6 hours googling, and I have found out, that there's a attribute _ddp_params_and_buffers_to_ignore which can be set to module that is passed to DistributedDataParallel constructor. I've implemented custom strategy plugin to Trainer, I have checked that the parameters are then passed to a parameters_to_ignore attribute of the DistributedDataParallel but somehow if I check gradients, of the layer, they are always the same.
Is there some simpler way to remove some layer / module from being synchronized between more devices in DDP strategy?
Thank you in advance for any help!
Okay, It was my mistake, I deeply apologize for wasting your time there. The layer indeeds gets removed from the DistributedDataParallel (or rather not even getting there).
But I've found another error when trying to set the _ddp_params_and_buffers_to_ignore inside the LightningModule, so I've created issue here - #11844 .
Thank you anyway!