Testing in dp mode uses only one of the GPUs
🐛 Bug

To Reproduce
Steps to reproduce the behavior:
Run a test without training


Code sample

Modified from the conference-seed repo
trainer = Trainer(
            gpus="-1",
            distributed_backend='dp',
        )
trainer.test(model)
Expected behavior

Environment

pl version: 0.6.0
PyTorch Version (e.g., 1.0): 1.2
OS (e.g., Linux): Ubuntu
How you installed PyTorch (conda, pip, source): pip
Build command you used (if compiling from source):
Python version: 3.6
CUDA/cuDNN version: 10.1
GPU models and configuration:
Any other relevant information:

Additional context