Repository: Lightning-AI/pytorch-lightning

Issue:
Tensorboard logging in multi-gpu setting not working properly?
Hi there :)

I have a question (that may be an issue with the code or just my ignorance).
(b.t.w. I am using the latest version, pytorch-lightning==0.4.9)

If I set the trainer 

`trainer = Trainer(experiment=exp, gpus=[0])`

I can see the corresponding logging (scalars and hyperparameters) in Tensorboard.
If I change it to distributed training (keeping the rest of the code unchanged) :

`
trainer = Trainer(experiment=exp, gpus=[0,1], distributed_backend='ddp')`

The Tensorboard logging stops working at least for scalars and hyperparameters, I see nothing except the experiment name.

In both cases 'exp' is a Experiment instantiated like this:

`    exp = Experiment(save_dir=/SOME/PATH, name=NAME, version=VERSION, description=DESCRIPTION)`

This picture illustrates the problem.

![Untitled](https://user-images.githubusercontent.com/2588285/65029817-b175d500-d93e-11e9-9f3f-3e350d4188d4.png)

In the picture the red arrows point to the "distributed" experiment, with no drawing in the chart. The other two (the ones that appear in the chart) are the very same, except than the run in single GPU.

Am I missing something or do I need to add extra configuration to make the logging work in multi-gpu with the ddp setting? Or is it a bug?

Thank you! :)



Discussion:
could you post the full code that uses the trainer?
Do you mean this part?

```
def main(hparams):
    model = MyCustomizedLightningModel(hparams)

    # harcoded path here
    exp = Experiment(save_dir='/DATA/exp3',
                     name='classif',
                     version='0.1',
                     description='description goes here')

    exp.argparse(hparams)

    trainer = Trainer(min_nb_epochs=1,
                      max_nb_epochs=5, nb_sanity_val_steps=1, experiment=exp, gpus=[0,1], distributed_backend='ddp')
    trainer.fit(model)
```

Only when I remove the "distributed_backend='ddp'" (and the second gpu from the list) to make it single-gpu it does fill the charts and hparams report in tensorboard, like in the picture above.
not sure what it would be, but try an integer and make sure to change it between experiments

```
    exp = Experiment(save_dir='/DATA/exp3',
                     name='classif',
                     version=1,  # increase every new experiment
                     description='description goes here')
```
I have tried with integer version numbers, but no luck.

![Untitled](https://user-images.githubusercontent.com/2588285/65042639-8cdb2680-d959-11e9-901b-dd72c30426a6.png)

The tensorboard writing/no-writing seems to be triggered by the "distributed_backend='ddp'" absence/presence in the Trainer instantiation.
Maybe when it is in distributed mode it writes to some other default location? Or there is an if/else related to the 'ddp' that prevents the SumaryWriter add the values to the report?
not sure... are you using jupyter notebook? jupyter behaves weird with tensorboard logger
No, I am coding using an IDE.

Don't worry anyway, sure there are more urgent tasks. For me it's ok for now, I can do most of the trainings on a single GPU :-)
Just wanted to know if I was doing something wrong. Maybe it has to do with something in my system. If I discover anything suspicious I will inform here.
If more people starts reporting about this problem then you can invest more attention on this.
Thanks for your time!
Actually I also observed the similar things when training with 'ddp' option. Seems that tfevents file is not correctly written (no file size increased during the training and no meta_tags, metrics file are generated). I also can do most of thing using 'dp' option now....
Seems like self.proc_rank is not well set in 'ddp'. I print self.proc_rank in "trainer" and found all process has proc_rank=0 which makes self.experiment to be working as debug mode. 

the following code change in ddp_train could fix the issues 

![image](https://user-images.githubusercontent.com/1920881/65133888-65aa5500-da3e-11e9-88f6-005006bae90f.png)

the code you show is from an old version of lightning. update to the latest to see if you still have an issue.

Or do you mean swapping the order of these lines?
https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py#L773-L774

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -767,12 +767,6 @@ def ddp_train(self, gpu_nb, model):
         except Exception:
             self.node_rank = 0
 
-        # recover original exp before went into process
-        # init in write mode only on proc 0
-        if self.experiment is not None:
-            self.experiment.debug = self.proc_rank > 0
-            self.experiment = self.experiment.get_non_ddp_exp()
-
         # show progbar only on prog_rank 0
         self.show_progress_bar = self.show_progress_bar and self.node_rank == 0 and gpu_nb == 0
 
@@ -781,7 +775,11 @@ def ddp_train(self, gpu_nb, model):
         self.world_size = self.nb_gpu_nodes * self.num_gpus
 
         # let the exp know the rank to avoid overwriting logs
+        # recover original exp before went into process
+        # init in write mode only on proc 0
         if self.experiment is not None:
+            self.experiment = self.experiment.get_non_ddp_exp()
+            self.experiment.debug = self.proc_rank > 0
             self.experiment.rank = self.proc_rank
 
         # set up server using proc 0's ip address


Test patch: