Repository: deepset-ai/haystack

Issue:
Cannot load FARMReader model from local disk
**Describe the bug**

I would like to load the large FARMReader model from local disk, not from huggingface.co, in order to cut down on traffic and setup time when docker compose is invoked.

It does not work.

**Error message**
Error that was thrown (if available)

```
dev-haystack-api-1           | INFO:haystack.modeling.model.language_model:Auto-detected model language: german
dev-haystack-api-1           | INFO:haystack.modeling.model.language_model:Loaded '/data/gelectra-large-germanquad' (Electra model) from model hub.
dev-haystack-api-1           | INFO:haystack.modeling.model.adaptive_model:Found files for loading 0 prediction heads
dev-haystack-api-1           | [2022-11-23 15:34:21 +0000] [8] [ERROR] Exception in worker process
dev-haystack-api-1           | Traceback (most recent call last):
dev-haystack-api-1           |   File "/home/user/haystack/pipelines/base.py", line 1958, in _load_or_get_component
dev-haystack-api-1           |     component_instance = BaseComponent._create_instance(
dev-haystack-api-1           |   File "/home/user/haystack/nodes/base.py", line 158, in _create_instance
dev-haystack-api-1           |     instance = subclass(**component_params)
dev-haystack-api-1           |   File "/home/user/haystack/nodes/base.py", line 48, in wrapper_exportable_to_yaml
dev-haystack-api-1           |     init_func(self, *args, **kwargs)
dev-haystack-api-1           |   File "/home/user/haystack/nodes/reader/farm.py", line 138, in __init__
dev-haystack-api-1           |     self.inferencer = QAInferencer.load(
dev-haystack-api-1           |   File "/home/user/haystack/modeling/infer.py", line 200, in load
dev-haystack-api-1           |     processor = Processor.load_from_dir(model_name_or_path)
dev-haystack-api-1           |   File "/home/user/haystack/modeling/data_handler/processor.py", line 174, in load_from_dir
dev-haystack-api-1           |     config = json.load(open(processor_config_file))
dev-haystack-api-1           | FileNotFoundError: [Errno 2] No such file or directory: '/data/gelectra-large-germanquad/processor_config.json'
```

**Expected behavior**

Load the model from local disk without error message.

**Additional context**

I am aware of the workaround posted here: https://github.com/deepset-ai/FARM/issues/299#issuecomment-735745722
(Basically load Transformer model with FARM and write the FARM model to a directory.)

However, I cannot `pip install farm` because of some Rust compilation issues, so I was unable to install the required dependencies and execute the script.

Using "deepset/gelectra-large-germanquad" works as expected.

**To Reproduce**

1. Git clone the model with lfs per instructions: https://huggingface.co/deepset/gelectra-large-germanquad/tree/main
2. Change `model_name_or_path` to a local directory in `pipelines.haystack-pipeline.yml`:

```yaml
  - name: Reader       # custom-name for the component; helpful for visualization & debugging
    type: FARMReader    # Haystack Class name for the component
    params:
      model_name_or_path: /data/gelectra-large-germanquad # deepset/gelectra-large-germanquad
      context_window_size: 500
      return_no_answer: true
      use_gpu: true
```

3. `docker compose up`


**FAQ Check**
- [X] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Ubuntu 22.04
 - GPU/CPU: GPU
 - Haystack version (commit or version number): 1.10.0
 - DocumentStore: OpenSearch
 - Reader: FarmReader
 - Retriever: BM25Retriever



Discussion:
@kaumanns Thanks for pointing out this issue. I am able to reproduce this issue as well. I will dig deeper to find the cause of the issue and get back to you. 

In case you already have a solution and would like to work on it, please let me know here :) 

Hey @mayankjobanputra the PR https://github.com/deepset-ai/haystack/pull/2685 from @mathislucka may be related
Hey @sjrl thanks for pointing it out! I will check it out and see if we can make progress here :)

Resolving patch:
diff --git a/haystack/modeling/infer.py b/haystack/modeling/infer.py
--- a/haystack/modeling/infer.py
+++ b/haystack/modeling/infer.py
@@ -191,15 +191,17 @@ def load(
 
         name = os.path.basename(model_name_or_path)
 
-        # a) either from local dir
-        if os.path.exists(model_name_or_path):
+        # a) non-hf models (i.e. FARM, ONNX) from local dir
+        farm_model_bin = os.path.join(model_name_or_path, "language_model.bin")
+        onnx_model = os.path.join(model_name_or_path, "model.onnx")
+        if os.path.isfile(farm_model_bin) or os.path.isfile(onnx_model):
             model = BaseAdaptiveModel.load(load_dir=model_name_or_path, device=devices[0], strict=strict)
             if task_type == "embeddings":
                 processor = InferenceProcessor.load_from_dir(model_name_or_path)
             else:
                 processor = Processor.load_from_dir(model_name_or_path)
 
-        # b) or from remote transformers model hub
+        # b) transformers models from hub or from local
         else:
             if not task_type:
                 raise ValueError(


Test patch:
diff --git a/test/nodes/test_reader.py b/test/nodes/test_reader.py
--- a/test/nodes/test_reader.py
+++ b/test/nodes/test_reader.py
@@ -1,8 +1,11 @@
 import math
 import os
 from pathlib import Path
+from shutil import rmtree
 
 import pytest
+
+from huggingface_hub import snapshot_download
 from haystack.modeling.data_handler.inputs import QAInput, Question
 
 from haystack.schema import Document, Answer
@@ -250,6 +253,44 @@ def test_farm_reader_update_params(docs):
         reader.predict(query="Who lives in Berlin?", documents=docs, top_k=3)
 
 
+# There are 5 different ways to load a FARMReader model.
+# 1. HuggingFace Hub (online load)
+# 2. HuggingFace downloaded (local load)
+# 3. HF Model saved as FARM Model (same works for trained FARM model) (local load)
+# 4. FARM Model converted to transformers (same as hf local model) (local load)
+# 5. ONNX Model load (covered by test_farm_reader_onnx_conversion_and_inference)
+@pytest.mark.integration
+def test_farm_reader_load_hf_online():
+    # Test Case: 1. HuggingFace Hub (online load)
+
+    hf_model = "hf-internal-testing/tiny-random-RobertaForQuestionAnswering"
+    _ = FARMReader(model_name_or_path=hf_model, use_gpu=False, no_ans_boost=0, num_processes=0)
+
+
+@pytest.mark.integration
+def test_farm_reader_load_hf_local(tmp_path):
+    # Test Case: 2. HuggingFace downloaded (local load)
+
+    hf_model = "hf-internal-testing/tiny-random-RobertaForQuestionAnswering"
+    # TODO: change the /tmp to proper tmp_path and get rid of rmtree
+    # local_model_path = str(Path.joinpath(tmp_path, "locally_saved_hf"))
+    local_model_path = "/tmp/locally_saved_hf"
+    model_path = snapshot_download(repo_id=hf_model, revision="main", cache_dir=local_model_path)
+    _ = FARMReader(model_name_or_path=model_path, use_gpu=False, no_ans_boost=0, num_processes=0)
+    rmtree(local_model_path)
+
+
+@pytest.mark.integration
+def test_farm_reader_load_farm_local(tmp_path):
+    # Test Case: 3. HF Model saved as FARM Model (same works for trained FARM model) (local load)
+
+    hf_model = "hf-internal-testing/tiny-random-RobertaForQuestionAnswering"
+    local_model_path = f"{tmp_path}/locally_saved_farm"
+    reader = FARMReader(model_name_or_path=hf_model, use_gpu=False, no_ans_boost=0, num_processes=0)
+    reader.save(Path(local_model_path))
+    _ = FARMReader(model_name_or_path=local_model_path, use_gpu=False, no_ans_boost=0, num_processes=0)
+
+
 @pytest.mark.parametrize("use_confidence_scores", [True, False])
 def test_farm_reader_uses_same_sorting_as_QAPredictionHead(use_confidence_scores):
     reader = FARMReader(