How to sync buffers in multi-gpu training
I know that parameters are indirectly synced in multi-gpu via grad-syncing.  But how to sync buffers that are not updated via gradient?
I find that I can use all_reduce() or all_gather() method manually in ddp doc, but what pytorch-lightning does under the hood? Does it sync buffers automatically or offer an interface to do that?
Considering pl handles single and multi-gpu without any need to change Modules, do it manually will result in ugly judgement code whether the training is in ddp mode inside an independent module.
Dear @MeteorsHub,
Great question. Lightning doesn't do any magic there. It relies on PyTorch DistributedDataParallel to handle multi-gpu via grad-syncing.
The buffers are being synced on start using _sync_params_and_buffers: https://github.com/pytorch/pytorch/blob/3e3acf8a9ac005db3094f23bd41a5fbc0c3c154b/torch/nn/parallel/distributed.py#L570
If you need to access this private function for any reason, you can do the following:

trainer.training_type_plugin.model._sync_params_and_buffers(authoritative_rank=0)