Repository: deepset-ai/haystack

Issue:
Remove quick fix to avoid applying DataParallel twice?
**Is your feature request related to a problem? Please describe.**
Issue #224 reported problems with a multi-GPU setup and #234 introduced a quick fix. However, commenting out the fix, I cannot reproduce the earlier problems anymore.

**Describe the solution you'd like**
I wonder if the quick fix to avoid applying DataParallel twice can be removed by deleting the following [three lines of code](https://github.com/deepset-ai/haystack/blob/b76ed4c5a4cac54b92e6cad2dc8ce9da07935bc6/haystack/reader/farm.py#L224) in reader and ranker nodes:
```
self.inferencer.model.save("tmp_model")
model = BaseAdaptiveModel.load(load_dir="tmp_model", device=device, strict=True)
shutil.rmtree('tmp_model')
```
Throughout the rest of the code, `model` should be replaced with `self.inferencer.model` if these lines are removed.

**Additional context**
Having apex installed on a machine with 4 GPUs and running tutorial 5 with `python -m torch.distributed.launch` , I couldn't find any difference in the logging output with or without the quick fix. It seems to run fine but I have never used apex before so I might have overlooked something. I could not find a check in FARM's optimize_model() that avoids applying DataParallel there if it was already done before: https://github.com/deepset-ai/FARM/blob/816b4e3e65c142f8a31a63833058b75fe0419ed4/farm/modeling/optimization.py#L272



Discussion:
With Timo I just saw that the optimize_model method is not called anymore in the Inferencer in FARM but the import is still there. @tholor your commit here removed the call of optimize_model in the Inferencer. That's why I could not reproduce the error, I guess. https://github.com/deepset-ai/FARM/commit/94c6b8d88486359a9aae694d094783e7baff07b1#diff-8a98ef73f1fa756ee7fdc0cd3a2f8bdc07f4e2eb490111356337ddc8c170066cR278 
@julian-risch  Ok, so let's remove the lines in Haystack then :) 

Resolving patch:
diff --git a/haystack/ranker/farm.py b/haystack/ranker/farm.py
--- a/haystack/ranker/farm.py
+++ b/haystack/ranker/farm.py
@@ -7,10 +7,8 @@
 from farm.data_handler.processor import TextPairClassificationProcessor
 from farm.infer import Inferencer
 from farm.modeling.optimization import initialize_optimizer
-from farm.modeling.adaptive_model import BaseAdaptiveModel
 from farm.train import Trainer
 from farm.utils import set_all_seeds, initialize_device_settings
-import shutil
 
 from haystack import Document
 from haystack.ranker.base import BaseRanker
@@ -172,16 +170,9 @@ def train(
         # and calculates a few descriptive statistics of our datasets
         data_silo = DataSilo(processor=processor, batch_size=batch_size, distributed=False, max_processes=num_processes)
 
-        # Quick-fix until this is fixed upstream in FARM:
-        # We must avoid applying DataParallel twice (once when loading the inferencer,
-        # once when calling initalize_optimizer)
-        self.inferencer.model.save("tmp_model")
-        model = BaseAdaptiveModel.load(load_dir="tmp_model", device=device, strict=True)
-        shutil.rmtree('tmp_model')
-
         # 3. Create an optimizer and pass the already initialized model
         model, optimizer, lr_schedule = initialize_optimizer(
-            model=model,
+            model=self.inferencer.model,
             learning_rate=learning_rate,
             schedule_opts={"name": "LinearWarmup", "warmup_proportion": warmup_proportion},
             n_batches=len(data_silo.loaders["train"]),
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -13,12 +13,11 @@
 from farm.infer import QAInferencer
 from farm.modeling.optimization import initialize_optimizer
 from farm.modeling.predictions import QAPred, QACandidate
-from farm.modeling.adaptive_model import BaseAdaptiveModel, AdaptiveModel
+from farm.modeling.adaptive_model import AdaptiveModel
 from farm.train import Trainer
 from farm.eval import Evaluator
 from farm.utils import set_all_seeds, initialize_device_settings
 from scipy.special import expit
-import shutil
 
 from haystack import Document
 from haystack.document_store.base import BaseDocumentStore
@@ -221,16 +220,9 @@ def train(
         # and calculates a few descriptive statistics of our datasets
         data_silo = DataSilo(processor=processor, batch_size=batch_size, distributed=False, max_processes=num_processes)
 
-        # Quick-fix until this is fixed upstream in FARM:
-        # We must avoid applying DataParallel twice (once when loading the inferencer,
-        # once when calling initalize_optimizer)
-        self.inferencer.model.save("tmp_model")
-        model = BaseAdaptiveModel.load(load_dir="tmp_model", device=device, strict=True)
-        shutil.rmtree('tmp_model')
-
         # 3. Create an optimizer and pass the already initialized model
         model, optimizer, lr_schedule = initialize_optimizer(
-            model=model,
+            model=self.inferencer.model,
             # model=self.inferencer.model,
             learning_rate=learning_rate,
             schedule_opts={"name": "LinearWarmup", "warmup_proportion": warmup_proportion},


Test patch: