Repository: deepset-ai/haystack

Issue:
Error while Training Your Own "Dense Passage Retrieval" Model
**Describe the bug**
Encounter an error while trying out the tutorial 9 for running a pre-existing DPR model on a custom dataset. 

**Error message**
File "/projects/mrqa_jars/emanual/emanual/code/run_dpr.py", line 30, in <module>
    retriever.train(
  File "/usr0/home/rdutt/anaconda3/envs/dpr/lib/python3.9/site-packages/haystack/nodes/retriever/dense.py", line 412, in train
    trainer.train()
  File "/usr0/home/rdutt/anaconda3/envs/dpr/lib/python3.9/site-packages/haystack/modeling/training/base.py", line 233, in train
    self.model.module.connect_heads_with_processor(self.data_silo.processor.tasks, require_labels=True)
  File "/usr0/home/rdutt/anaconda3/envs/dpr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'DataParallel' object has no attribute 'connect_heads_with_processor'

**Expected behavior**
Does not follow the fix as encountered in https://github.com/AlonEirew/haystack/commit/81e7488328b2f5f4d01042fbe5ebf913b422b465

**To Reproduce**
Follow the steps mentioned in the tutorial in a separate conda environment

https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial9_DPR_training.ipynb#scrollTo=Obmjagb70xiR

**FAQ Check**
- [✅] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: VERSION="18.04.6 LTS (Bionic Beaver)"
 - GPU/CPU: CUDA Version: 11.2, 5 GPUs GeoForce GTX 1080
 - Haystack version (commit or version number): 1.0.0rc1
 - DocumentStore: InMemoryDocumentStore
 - Reader: Not applicable
 - Retriever: Dense Passage Retriever



Discussion:
Hi @ShoRit Thank you for reporting this. I will try to reproduce the error. In the meantime, I think this issue is related to https://github.com/deepset-ai/haystack/issues/1687 so maybe you could have a look there? I think the problem should have been fixed in https://github.com/deepset-ai/haystack/commit/861522b6b11997326cc01ec38940fb04cb16be3c I am pretty sure it's related to using more than one GPU. 
@bogdankostic @brandenchan any thoughts on this?

Resolving patch:
diff --git a/haystack/modeling/model/optimization.py b/haystack/modeling/model/optimization.py
--- a/haystack/modeling/model/optimization.py
+++ b/haystack/modeling/model/optimization.py
@@ -303,7 +303,7 @@ def optimize_model(model, device, local_rank, optimizer=None, distributed=False,
                            find_unused_parameters=True)
 
     elif torch.cuda.device_count() > 1 and device.type == "cuda":
-        model = WrappedDataParallel(model)
+        model = WrappedDataParallel(model) if not isinstance(model, DataParallel) else WrappedDataParallel(model.module)
         logger.info("Multi-GPU Training via DataParallel")
 
     return model, optimizer
diff --git a/haystack/nodes/retriever/dense.py b/haystack/nodes/retriever/dense.py
--- a/haystack/nodes/retriever/dense.py
+++ b/haystack/nodes/retriever/dense.py
@@ -413,7 +413,8 @@ def train(self,
         self.query_tokenizer.save_pretrained(f"{save_dir}/{query_encoder_save_dir}")
         self.passage_tokenizer.save_pretrained(f"{save_dir}/{passage_encoder_save_dir}")
 
-        self.model = DataParallel(self.model, device_ids=self.devices)
+        if len(self.devices) > 1 and not isinstance(self.model, DataParallel):
+            self.model = DataParallel(self.model, device_ids=self.devices)
 
     def save(self, save_dir: Union[Path, str], query_encoder_dir: str = "query_encoder",
              passage_encoder_dir: str = "passage_encoder"):


Test patch: