Repository: pytorch/ignite

Issue:
auto_dataloader can auto set pin_memory if appropriate
## 🚀 Feature

Idea is to automatically set `pin_memory=True` if `idist.backend()` is `nccl` or default device is `cuda`.

In the code we can omit this param instead of 
https://github.com/pytorch/ignite/blob/6b88eb89cf40e3c4ebe05e0fe67f3c8013ecc5c2/examples/contrib/cifar10/main.py#L199



Discussion:
Hey, me again :)

So there is only one line that should be changed? Did I understand that right?
Hey @InCogNiTo124 :)

1) It is about adding some code on L91
https://github.com/pytorch/ignite/blob/6b88eb89cf40e3c4ebe05e0fe67f3c8013ecc5c2/ignite/distributed/auto.py#L90-L92

check if `kwargs` contains `pin_memory` and if not set to True if `idist.device()` is `cuda`.
Probably, more than one line.

2) update examples
- cifar10


Sounds doable. And what about docs or tests?
> Sounds doable. And what about docs or tests?

Just say here : https://github.com/pytorch/ignite/blob/6b88eb89cf40e3c4ebe05e0fe67f3c8013ecc5c2/ignite/distributed/auto.py#L26-L28
that we auto set pin_memory=True for cuda devices if user does not provide pin_memory arg.

For tests, we can check `pin_memory` attribute of DataLoader here : 
https://github.com/pytorch/ignite/blob/6b88eb89cf40e3c4ebe05e0fe67f3c8013ecc5c2/tests/ignite/distributed/test_auto.py#L15
if provided instance is DataLoader (can be MpDeviceLoader for XLA)...

Resolving patch:
diff --git a/examples/contrib/cifar10/main.py b/examples/contrib/cifar10/main.py
--- a/examples/contrib/cifar10/main.py
+++ b/examples/contrib/cifar10/main.py
@@ -192,12 +192,7 @@ def get_dataflow(config):
 
     # Setup data loader also adapted to distributed config: nccl, gloo, xla-tpu
     train_loader = idist.auto_dataloader(
-        train_dataset,
-        batch_size=config["batch_size"],
-        num_workers=config["num_workers"],
-        shuffle=True,
-        pin_memory="cuda" in idist.device().type,
-        drop_last=True,
+        train_dataset, batch_size=config["batch_size"], num_workers=config["num_workers"], shuffle=True, drop_last=True,
     )
 
     test_loader = idist.auto_dataloader(
diff --git a/ignite/distributed/auto.py b/ignite/distributed/auto.py
--- a/ignite/distributed/auto.py
+++ b/ignite/distributed/auto.py
@@ -25,6 +25,7 @@ def auto_dataloader(dataset, **kwargs):
     - number of workers is scaled by number of local processes: ``num_workers / nprocs``.
     - if no sampler provided by user, `torch DistributedSampler` is setup.
     - if a sampler is provided by user, it is wrapped by :class:`~ignite.distributed.auto.DistributedProxySampler`.
+    - if the default device is 'cuda', `pin_memory` is automatically set to `True`.
 
     .. warning::
 
@@ -96,6 +97,8 @@ def auto_dataloader(dataset, **kwargs):
             "Argument `pin_memory=False` will be used to construct data loader."
         )
         kwargs["pin_memory"] = False
+    else:
+        kwargs["pin_memory"] = kwargs.get("pin_memory", idist.device() == "cuda")
 
     logger.info("Use data loader kwargs for dataset '{}': \n\t{}".format(repr(dataset)[:20].strip(), kwargs))
     dataloader = DataLoader(dataset, **kwargs)


Test patch:
diff --git a/tests/ignite/distributed/test_auto.py b/tests/ignite/distributed/test_auto.py
--- a/tests/ignite/distributed/test_auto.py
+++ b/tests/ignite/distributed/test_auto.py
@@ -39,6 +39,8 @@ def _test_auto_dataloader(ws, nproc, sampler_name=None, dl_type=DataLoader):
     else:
         sampler_type = DistributedSampler if sampler is None else DistributedProxySampler
         assert isinstance(dataloader.sampler, sampler_type)
+    if isinstance(dataloader, DataLoader):
+        assert dataloader.pin_memory == (data.device == "cuda")
 
 
 def _test_auto_model_optimizer(ws, device):