Repository: deepset-ai/haystack

Issue:
Use AMP training with FARMReader
**Is your feature request related to a problem? Please describe.**
Speed up reader training.

**Describe the solution you'd like**
I have seen that FARM `Trainer` already has [use_amp parameter](https://github.com/deepset-ai/FARM/blob/master/farm/train.py#L201) to enable automatic mixed precision training. However, when I try to feed this parameter from `Trainer` initialization in `FARMReader` `train` function (i.e. [here](https://github.com/deepset-ai/haystack/blob/master/haystack/reader/farm.py#L208)), training stopped with an error:


```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-c881150ab43c> in <module>()
----> 1 reader.train(data_dir=train_data, train_filename=train_filename, use_gpu=True, n_epochs=1, save_dir="my_model")

4 frames
/usr/local/lib/python3.6/dist-packages/apex/amp/handle.py in scale_loss(loss, optimizers, loss_id, model, delay_unscale, delay_overflow_check)
     80     """
     81     if not hasattr(_amp_state, "opt_properties"):
---> 82         raise RuntimeError("Invoked 'with amp.scale_loss`, but internal Amp state has not been initialized.  "
     83                            "model, optimizer = amp.initialize(model, optimizer, opt_level=...) must be called "
     84                            "before `with amp.scale_loss`.")

RuntimeError: Invoked 'with amp.scale_loss`, but internal Amp state has not been initialized.  model, optimizer = amp.initialize(model, optimizer, opt_level=...) must be called before `with amp.scale_loss`.
```

**Additional context**
I think in `FARM.utils.initialize_device_setting` something is missing, even considering that the `use_amp` parameter is used only to be printed. Probably is required some AMP initialization there?


Discussion:
Yes, you would need to pass `use_amp` in three locations: 
- init devices (just for logging): https://github.com/deepset-ai/haystack/blob/669c72d538c6dd04e14b101f91724668822ef9d7/haystack/reader/farm.py#L166
- Optimizer: https://github.com/deepset-ai/haystack/blob/669c72d538c6dd04e14b101f91724668822ef9d7/haystack/reader/farm.py#L198
- Trainer:https://github.com/deepset-ai/haystack/blob/669c72d538c6dd04e14b101f91724668822ef9d7/haystack/reader/farm.py#L208

You can also find an example for amp usage in FARM:

As of now, we don't have AMP in haystack, but if you are interested, I'll be happy to review a PR here.  

Resolving patch:
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -118,7 +118,8 @@ def train(
         dev_split: float = 0,
         evaluate_every: int = 300,
         save_dir: Optional[str] = None,
-        num_processes: Optional[int] = None
+        num_processes: Optional[int] = None,
+        use_amp: str = None,
     ):
         """
         Fine-tune a model on a QA dataset. Options:
@@ -145,6 +146,14 @@ def train(
         :param num_processes: The number of processes for `multiprocessing.Pool` during preprocessing.
                               Set to value of 1 to disable multiprocessing. When set to 1, you cannot split away a dev set from train set.
                               Set to None to use all CPU cores minus one.
+        :param use_amp: Optimization level of NVIDIA's automatic mixed precision (AMP). The higher the level, the faster the model.
+                        Available options:
+                        None (Don't use AMP)
+                        "O0" (Normal FP32 training)
+                        "O1" (Mixed Precision => Recommended)
+                        "O2" (Almost FP16)
+                        "O3" (Pure FP16).
+                        See details on: https://nvidia.github.io/apex/amp.html
         :return: None
         """
 
@@ -163,7 +172,7 @@ def train(
         if max_seq_len is None:
             max_seq_len = self.max_seq_len
 
-        device, n_gpu = initialize_device_settings(use_cuda=use_gpu)
+        device, n_gpu = initialize_device_settings(use_cuda=use_gpu,use_amp=use_amp)
 
         if not save_dir:
             save_dir = f"../../saved_models/{self.inferencer.model.language_model.name}"
@@ -202,7 +211,8 @@ def train(
             schedule_opts={"name": "LinearWarmup", "warmup_proportion": warmup_proportion},
             n_batches=len(data_silo.loaders["train"]),
             n_epochs=n_epochs,
-            device=device
+            device=device,
+            use_amp=use_amp,
         )
         # 4. Feed everything to the Trainer, which keeps care of growing our model and evaluates it from time to time
         trainer = Trainer(
@@ -214,6 +224,7 @@ def train(
             lr_schedule=lr_schedule,
             evaluate_every=evaluate_every,
             device=device,
+            use_amp=use_amp,
         )
 
 


Test patch: