Repository: Lightning-AI/pytorch-lightning

Issue:
Named converted to regular tuples when sent to the gpu.
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
Named tuples returned from `Dataset` get converted to regular tuples when sent to the gpu.
This happens because `isinstance(instance_of_a_named_tuple, tuple)` evaluates to True in `distrib_parts.py`
https://github.com/PyTorchLightning/pytorch-lightning/blob/67d5f4dc392250d23bfeb11aba45e919a99ff1c0/pytorch_lightning/trainer/distrib_parts.py#L463
### To Reproduce
```python
import pytorch_lightning as pl
from collections import namedtuple
import torch
import numpy

NamedTupleDemoInput = namedtuple('DemoInput', ['x1', 'x2', 'y'])

class NamedTupleDemoDataset:
    def __len__(self):
        return 30000

    def __getitem__(self, index):
        x1 = numpy.random.uniform(0, 100)
        x2 = numpy.random.uniform(0, 100)
        y = 2*x1 + 3*x2 + numpy.random.normal(0, 0.05)
        return NamedTupleDemoInput(x1, x2, y)

class WeightedSum(torch.nn.Module):
    def __init__(self):
        super(WeightedSum, self).__init__()
        self.a = torch.nn.Parameter(torch.zeros(1))
        self.b = torch.nn.Parameter(torch.zeros(1))

    def forward(self, x1, x2):
        return self.a * x1 + self.b * x2

class NamedTupleDemo(pl.LightningModule):

    def __init__(self):
        super(NamedTupleDemo, self).__init__()
        self.model = WeightedSum()

    def forward(self, x1, x2):
        return self.model(x1, x2)

    def train_dataloader(self):
        return torch.utils.data.DataLoader(NamedTupleDemoDataset(), batch_size=128)

    def training_step(self, batch, batch_index):
        yhat = self.forward(batch.x1, batch.x2)
        return {'loss': torch.nn.functional.mse_loss(batch.y, yhat)}

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=1e-2)

if __name__ == '__main__':
    module = NamedTupleDemo()
    pl.Trainer(max_epochs=20, gpus=1).fit(module)
    print(f'a={float(module.model.a)} b={float(module.model.b)}')

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

```
Traceback (most recent call last):
  File "demo.py", line 48, in <module>
    pl.Trainer(max_epochs=20, gpus=1).fit(module)
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 749, in fit
    self.single_gpu_train(model)
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/distrib_parts.py", line 491, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 910, in run_pretrain_routine
    self.train()
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 384, in train
    self.run_training_epoch()
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 456, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 633, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 597, in optimizer_closure
    output_dict = self.training_forward(split_batch, batch_idx, opt_idx, self.hiddens)
  File "/home/n/repos/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 770, in training_forward
    output = self.model.training_step(*args)
  File "demo.py", line 40, in training_step
    yhat = self.forward(batch.x1, batch.x2)
AttributeError: 'tuple' object has no attribute 'x1'
```
<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
Minimal means having the shortest code but still preserving the bug. -->

### Expected behavior
Namedtuples returned from the dataset should be keep their original fields.  

### Environment
* CUDA:
        - GPU:
                - GeForce RTX 2080 Ti
        - available:         True
        - version:           10.2
* Packages:
        - numpy:             1.18.3
        - pyTorch_debug:     False
        - pyTorch_version:   1.5.0
        - pytorch-lightning: 0.7.4rc5
        - tensorboard:       2.2.1
        - tqdm:              4.45.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - ELF
        - processor:
        - python:            3.8.2
        - version:           #1 SMP PREEMPT Sun, 05 Apr 2020 05:13:14 +0000

<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/distrib_parts.py b/pytorch_lightning/trainer/distrib_parts.py
--- a/pytorch_lightning/trainer/distrib_parts.py
+++ b/pytorch_lightning/trainer/distrib_parts.py
@@ -461,10 +461,15 @@ def __transfer_data_to_device(self, batch, device, gpu_id=None):
 
         # when tuple
         if isinstance(batch, tuple):
-            batch = list(batch)
-            for i, x in enumerate(batch):
-                batch[i] = self.__transfer_data_to_device(x, device, gpu_id)
-            return tuple(batch)
+            # when namedtuple
+            if hasattr(batch, '_fields'):
+                elem_type = type(batch)
+                return elem_type(*(self.__transfer_data_to_device(x, device, gpu_id) for x in batch))
+            else:
+                batch = list(batch)
+                for i, x in enumerate(batch):
+                    batch[i] = self.__transfer_data_to_device(x, device, gpu_id)
+                return tuple(batch)
 
         # when dict
         if isinstance(batch, dict):


Test patch:
diff --git a/tests/models/test_cpu.py b/tests/models/test_cpu.py
--- a/tests/models/test_cpu.py
+++ b/tests/models/test_cpu.py
@@ -1,3 +1,4 @@
+from collections import namedtuple
 import platform
 
 import pytest
@@ -221,6 +222,13 @@ def test_single_gpu_batch_parse():
     assert batch[1][0]['b'].device.index == 0
     assert batch[1][0]['b'].type() == 'torch.cuda.FloatTensor'
 
+    # namedtuple of tensor
+    BatchType = namedtuple('BatchType', ['a', 'b'])
+    batch = [BatchType(a=torch.rand(2, 3), b=torch.rand(2, 3)) for _ in range(2)]
+    batch = trainer.transfer_batch_to_gpu(batch, 0)
+    assert batch[0].a.device.index == 0
+    assert batch[0].a.type() == 'torch.cuda.FloatTensor'
+
 
 def test_simple_cpu(tmpdir):
     """Verify continue training session on CPU."""