DDP specifics: Single node function execution, test loss sync, num_workers, sync_batchnorm, precision
❓ Questions and Help
What is your question?
This is about DDP specifics and about the handling of functions in a script that we only want executed once (not for every GPU).
I think they are missing from the doc and I couldn't find answers elsewhere. I apologize if they have been covered already.
Questions:

Does the regular torch.save(model.state_dict(), path) call work normally or does DDP complicate things for multiple GPUS?
Does DDP run all the functions of a script for every GPU? For example, in the following script will the delete_and_remake_folder function be executed multiple times (that would result in conflicts)? Is there a way to specify functions to be run only once?
Am I correct that sync_batchnorm=True and precision=16 work in DDP?
Does the trainer.test() function automatically aggregate results accross devices or is it required to set self.log(loss,  sync_dist=True) in the model?
Am I correct in assuming that, if we set num_workers=X in a Dataloader, the actual CPU core usage will be X*N for N GPUS?

Questions 1-4 are summarized in the following script and whether it works/can work.
def main():
   delete_and_remake_folder() # I only want to run once
   model = Model()
   trainer = Trainer(gpus = 8, backend='ddp, sync_batchnorm=True, precision=16)
   trainer.fit()
   trainer.test()
   torch.save(model.pt_model.state_dict(), save_dir) # I probably only want to run once (?)
yes, but you might run into the issue in your 2nd question
yes, yes if self.global_rank == 0: do stuff
yes
no, yes (think of it like another val loop)
yes

I'll close this for now, if you have more questions please use our forums! https://forums.pytorchlightning.ai/