Pickle error from Trainer.fit when using MLFlowLogger and distributed data parallel without SLURM
🐛 Bug
Trainer.fit fails with a pickle error when the logger is MLFlowLogger, and distributed_backend='ddp' on GPUs but without SLURM.
To Reproduce
Steps to reproduce the behavior:

Instantiate MLFlowLogger in Pytorch 0.5.3.2 with Pytorch 1.3.1 and MLFlow 1.4.0. The execution environment has environment variables MLFLOW_TRACKING_URI, and also MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD to connect to the MLflow tracking server with HTTP Basic Authentication. The MLflow tracking server is also v1.4.0.
Instantiate Trainer with MLFlowLogger instance as logger, distributed_backend='ddp' and with the gpus parameter on a machine with NVIDIA GPUs but without SLURM.
Run Trainer.fit

From the error output, it looks like multiprocessing is attempting to pickle the nested function in MLflow function _get_rest_store:
ayla.khan@gpu12:~/photosynthetic$ python test_mlflow.py
Traceback (most recent call last):
  File "test_mlflow.py", line 71, in <module>
    trainer.fit(model)
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 343, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 162, in spawn
    process.start()
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/mnt/unihome/home/CORP/ayla.khan/miniconda2/envs/photosynthetic/lib/python3.6/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_get_rest_store.<locals>.get_default_host_creds'

Code sample
Sample code tested with a very simple test model (gist):
test_hparams = Namespace()
model = XORGateModel(test_hparams)

logger = MLFlowLogger(experiment_name='test_lightning_logger',
                                          tracking_uri=os.environ['MLFLOW_TRACKING_URI'])
trainer = pl.Trainer(logger=logger, distributed_backend='ddp', gpus='-1')
trainer.fit(model)

Expected behavior
Trainer.fit runs without error.
Environment
(photosynthetic) ayla.khan@gpu12:~/photosynthetic$ python collect_env.py
Collecting environment information...
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: CentOS Linux 7 (Core)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti
GPU 4: GeForce GTX 1080 Ti
GPU 5: GeForce GTX 1080 Ti
GPU 6: GeForce GTX 1080 Ti
GPU 7: GeForce GTX 1080 Ti

Nvidia driver version: 440.33.01
cuDNN version: /usr/local/cuda-10.0/lib64/libcudnn.so.7

Versions of relevant libraries:
[pip] numpy==1.16.4
[pip] pytorch-lightning==0.5.3.2
[pip] pytorch-toolbelt==0.2.1
[pip] torch==1.3.1
[pip] torchsummary==1.5.1
[pip] torchvision==0.4.2
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py36he904b0f_0
[conda] mkl_fft                   1.0.15           py36ha843d7b_0
[conda] mkl_random                1.1.0            py36hd6b4f25_0
[conda] pytorch                   1.3.1           py3.6_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] pytorch-lightning         0.5.3.2                  pypi_0    pypi
[conda] pytorch-toolbelt          0.2.1                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.4.2                py36_cu101    pytorch