question: DDP specifics: Single node function execution, test loss sync, num_workers, sync_batchnorm, precision
context: ❓ Questions and Help
  What is your question?
  This is about DDP specifics and about the handling of functions in a script that we only want executed once (not for every GPU).
  I think they are missing from the doc and I couldn't find answers elsewhere. I apologize if they have been covered already.
  Questions:
  
  Does the regular torch.save(model.state_dict(), path) call work normally or does DDP complicate things for multiple GPUS?
  Does DDP run all the functions of a script for every GPU? For example, in the following script will the delete_and_remake_folder function be executed multiple times (that would result in conflicts)? Is there a way to specify functions to be run only once?
  Am I correct that sync_batchnorm=True and precision=16 work in DDP?
  Does the trainer.test() function automatically aggregate results accross devices or is it required to set self.log(loss,  sync_dist=True) in the model?
  Am I correct in assuming that, if we set num_workers=X in a Dataloader, the actual CPU core usage will be X*N for N GPUS?
  
  Questions 1-4 are summarized in the following script and whether it works/can work.
  def main():
     delete_and_remake_folder() # I only want to run once
     model = Model()
     trainer = Trainer(gpus = 8, backend='ddp, sync_batchnorm=True, precision=16)
     trainer.fit()
     trainer.test()
     torch.save(model.pt_model.state_dict(), save_dir) # I probably only want to run once (?)
answer: yes, but you might run into the issue in your 2nd question
  yes, yes if self.global_rank == 0: do stuff
  yes
  no, yes (think of it like another val loop)
  yes
  
  I'll close this for now, if you have more questions please use our forums! https://forums.pytorchlightning.ai/