Dataloader pickle torchio
I am trying to get multi-gpu training working, on single gpu it is al working fine. However, when I increase the number of GPUs I get a pickling error, and I don't know what to do about it. For the dataloader I am using the patch-based approach from TorchIO, which creates a Queue, maybe that is the cause? Does anyone has experience with TorchIO Queue and Lightning multi-gpu maybe? Or is something else going on?
The error i am getting is as follows:
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3,4]

Traceback (most recent call last):
  File "/filepath/SRGAN-patch_tio.py", line 94, in <module>
    main()
  File "/filepath/SRGAN-patch_tio.py", line 91, in main
    trainer.fit(model, train_dataloaders=training_loader)
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 737, in fit
    self._call_and_handle_interrupt(
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 772, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run
    self._dispatch()
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1274, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 173, in start_training
    self.spawn(self.new_process, trainer, self.mp_queue, return_result=False)
  File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 201, in spawn
    mp.spawn(self._wrapped_function, args=(function, args, kwargs, return_queue), nprocs=self.num_processes)
  File "/mnt/beta/name/miniconda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/mnt/beta/name/miniconda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes
    process.start()
  File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/mnt/beta/name/miniconda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 547, in __getstate__
    raise NotImplementedError("{} cannot be pickled", self.__class__.__name__)
NotImplementedError: ('{} cannot be pickled', '_SingleProcessDataLoaderIter')
How are launching your training? Can you try to set strategy='ddp' instead of the default 'ddp_spawn' for multiple GPUs. For me that works and in opposite to ddp_spawn it does not pickle your dataloader.