question: Dataloader pickle torchio
context: I am trying to get multi-gpu training working, on single gpu it is al working fine. However, when I increase the number of GPUs I get a pickling error, and I don't know what to do about it. For the dataloader I am using the patch-based approach from TorchIO, which creates a Queue, maybe that is the cause? Does anyone has experience with TorchIO Queue and Lightning multi-gpu maybe? Or is something else going on?
  The error i am getting is as follows:
  GPU available: True, used: True
  TPU available: False, using: 0 TPU cores
  IPU available: False, using: 0 IPUs
  LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3,4]
  
  Traceback (most recent call last):
    File "/filepath/SRGAN-patch_tio.py", line 94, in <module>
      main()
    File "/filepath/SRGAN-patch_tio.py", line 91, in main
      trainer.fit(model, train_dataloaders=training_loader)
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 737, in fit
      self._call_and_handle_interrupt(
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
      return trainer_fn(*args, **kwargs)
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 772, in _fit_impl
      self._run(model, ckpt_path=ckpt_path)
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run
      self._dispatch()
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1274, in _dispatch
      self.training_type_plugin.start_training(self)
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 173, in start_training
      self.spawn(self.new_process, trainer, self.mp_queue, return_result=False)
    File "/home/name/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 201, in spawn
      mp.spawn(self._wrapped_function, args=(function, args, kwargs, return_queue), nprocs=self.num_processes)
    File "/mnt/beta/name/miniconda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
      return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
    File "/mnt/beta/name/miniconda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes
      process.start()
    File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/process.py", line 121, in start
      self._popen = self._Popen(self)
    File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
      return Popen(process_obj)
    File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
      super().__init__(process_obj)
    File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
      self._launch(process_obj)
    File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
      reduction.dump(process_obj, fp)
    File "/mnt/beta/name/miniconda/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
      ForkingPickler(file, protocol).dump(obj)
    File "/mnt/beta/name/miniconda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 547, in __getstate__
      raise NotImplementedError("{} cannot be pickled", self.__class__.__name__)
  NotImplementedError: ('{} cannot be pickled', '_SingleProcessDataLoaderIter')
answer: How are launching your training? Can you try to set strategy='ddp' instead of the default 'ddp_spawn' for multiple GPUs. For me that works and in opposite to ddp_spawn it does not pickle your dataloader.