question: How to sync buffers in multi-gpu training
context: I know that parameters are indirectly synced in multi-gpu via grad-syncing.  But how to sync buffers that are not updated via gradient?
  I find that I can use all_reduce() or all_gather() method manually in ddp doc, but what pytorch-lightning does under the hood? Does it sync buffers automatically or offer an interface to do that?
  Considering pl handles single and multi-gpu without any need to change Modules, do it manually will result in ugly judgement code whether the training is in ddp mode inside an independent module.
answer: Dear @MeteorsHub,
  Great question. Lightning doesn't do any magic there. It relies on PyTorch DistributedDataParallel to handle multi-gpu via grad-syncing.
  The buffers are being synced on start using _sync_params_and_buffers: https://github.com/pytorch/pytorch/blob/3e3acf8a9ac005db3094f23bd41a5fbc0c3c154b/torch/nn/parallel/distributed.py#L570
  If you need to access this private function for any reason, you can do the following:
  
  trainer.training_type_plugin.model._sync_params_and_buffers(authoritative_rank=0)