WandDBLogger ddp race condition
When I run trainer(gpus=2, logger=wandblogger()) it tries to connect to wandb twice (I think) and then one of those attempts fails.
How to fix?
[original feature] (#627)
Code
logger = WandbLogger(name=model.output_dir.name)
logger.log_hyperparams(args) # you can see 1 copy of this on wandb.com

checkpoint_callback = ModelCheckpoint(
    filepath=str(model.output_dir / "{epoch}-{val_avg_rouge2:.4f}"), monitor="val_loss", mode="min", save_top_k=1,
)

train_params = dict(
    accumulate_grad_batches=args.gradient_accumulation_steps,
    gpus=2,
    max_epochs=2,
    checkpoint_callback=checkpoint_callback,
    callbacks=[LoggingCallback()],
    val_check_interval=0.25
    logger=logger,
    weights_summary=None,
    use_amp=True
)
trainer = pl.Trainer(**train_params)
trainer.fit(model)
Epic traceback
wandb: Tracking run with wandb version 0.8.36
wandb: Run data is saved locally in wandb/run-20200527_155653-1yd9jb0c
wandb: Syncing run cnn_12_9_no_teacher_mgpu
wandb: ⭐ View project at https://app.wandb.ai/sshleifer/transformers_fork-examples_summarization_bart
wandb: 🚀 View run at https://app.wandb.ai/sshleifer/transformers_fork-examples_summarization_bart/runs/1yd9jb0c
wandb: Run `wandb off` to turn off syncing.

GPU available: True, used: True
No environment variable for node rank defined. Set as 0.
CUDA_VISIBLE_DEVICES: [0,1]
Using 16bit precision.
MASTER_ADDR environment variable is not defined. Set as localhost
initializing proc_rank 0 world 2
MASTER_ADDR environment variable is not defined. Set as localhost
initializing proc_rank 1 world 2
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.
here's the 2nd connection
wandb: Tracking run with wandb version 0.8.36
wandb: Run data is saved locally in wandb/run-20200527_155706-1yd9jb0c
wandb: Syncing run cnn_12_9_no_teacher_mgpu
wandb: ⭐ View project at https://app.wandb.ai/sshleifer/transformers_fork-examples_summarization_bart
wandb: 🚀 View run at https://app.wandb.ai/sshleifer/transformers_fork-examples_summarization_bart/runs/1yd9jb0c
wandb: Run `wandb off` to turn off syncing.
Then error.
Traceback (most recent call last):                                                                                                                                                                                                 [69/1405]
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 389, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in run_pretrain_routine
    self.configure_checkpoint_callback()
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_config.py", line 58, in configure_ch
wandb: Waiting for W&B process to finish, PID 8253
eckpoint_callback
    "checkpoints"
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb: Program ended successfully.
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/wandb/apis/file_stream.py", line 213, in _thread_body
    self._send(ready_chunks)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/wandb/apis/file_stream.py", line 249, in _send
    self._client.post, self._endpoint, json={'files': files}))
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/wandb/apis/file_stream.py", line 227, in _handle_response
    raise response
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/wandb/util.py", line 590, in request_with_retry
    response.raise_for_status()
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.wandb.ai/files/sshleifer/transformers_fork-examples_summarization_bart/a199yu5d/file_stream

wandb: Syncing 7 W&B file(s) and 0 media file(s)
wandb: Process crashed early, not syncing files