Repository: deepset-ai/haystack

Issue:
RuntimeError when loading data using data_silo
**Describe the bug**
Getting RuntimeError when loading data using data_silo, this error seems related to the multiprocessing sharing strategy, which opens many file descriptors. trying to increase the ulimit on my machine to 2048 does not help (cannot increase further). 

A solution to the issue might be to further increase file descriptors limit (following this https://github.com/fastai/fastai/issues/23). 
Unfortunately the hard limit in my machine is set to 2048

See file descriptors (open files) limit on machine:
`#>ulimit -n`

Increase file descriptors:
`#>ulimit -n 2048`

See hard limits:
`#>ulimit -H -a`

**Error message**
```Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 289, in rebuild_storage_fd
    fd = df.detach()
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/reduction.py", line 164, in recvfds
    raise RuntimeError('received %d items of ancdata' %
RuntimeError: received 0 items of ancdata
```
**Expected behavior**
No Error

**Additional context**
Running on a machine with 88 CPU's

**To Reproduce**
```
document_store = FAISSDocumentStore.load(index_path=faiss_index_path, config_path=faiss_config_path)

retriever = DensePassageRetriever(document_store=document_store,
                                      query_embedding_model='bert-base-uncased',
                                      passage_embedding_model='bert-base-uncased',
                                      infer_tokenizer_classes=True,
                                      max_seq_len_query=64,
                                      max_seq_len_passage=180
                                      )

retriever.train(data_dir=doc_dir,
                    train_filename=train_filename,
                    dev_filename=dev_filename,
                    test_filename=dev_filename,
                    n_epochs=n_epochs,
                    batch_size=16,
                    grad_acc_steps=8,
                    save_dir=save_dir,
                    evaluate_every=20,
                    embed_title=False,
                    num_positives=1,
                    num_hard_negatives=1
                    )
```

**FAQ Check**

**System:**
 - OS: 16.04.1-Ubuntu
 - GPU/CPU: 4 * TITAN Xp / 87 * Intel(R) Xeon(R) CPU E5-2699A v4 @ 2.40GHz
 - Haystack version (commit or version number): v1.0.0
 - DocumentStore: FAISSDocumentStore
 - Reader: NA
 - Retriever: DensePassageRetriever

RuntimeError when loading data using data_silo
**Describe the bug**
Getting RuntimeError when loading data using data_silo, this error seems related to the multiprocessing sharing strategy, which opens many file descriptors. trying to increase the ulimit on my machine to 2048 does not help (cannot increase further). 

A solution to the issue might be to further increase file descriptors limit (following this https://github.com/fastai/fastai/issues/23). 
Unfortunately the hard limit in my machine is set to 2048

See file descriptors (open files) limit on machine:
`#>ulimit -n`

Increase file descriptors:
`#>ulimit -n 2048`

See hard limits:
`#>ulimit -H -a`

**Error message**
```Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 289, in rebuild_storage_fd
    fd = df.detach()
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/alon_nlp/miniconda3/envs/wec-es/lib/python3.9/multiprocessing/reduction.py", line 164, in recvfds
    raise RuntimeError('received %d items of ancdata' %
RuntimeError: received 0 items of ancdata
```
**Expected behavior**
No Error

**Additional context**
Running on a machine with 88 CPU's

**To Reproduce**
```
document_store = FAISSDocumentStore.load(index_path=faiss_index_path, config_path=faiss_config_path)

retriever = DensePassageRetriever(document_store=document_store,
                                      query_embedding_model='bert-base-uncased',
                                      passage_embedding_model='bert-base-uncased',
                                      infer_tokenizer_classes=True,
                                      max_seq_len_query=64,
                                      max_seq_len_passage=180
                                      )

retriever.train(data_dir=doc_dir,
                    train_filename=train_filename,
                    dev_filename=dev_filename,
                    test_filename=dev_filename,
                    n_epochs=n_epochs,
                    batch_size=16,
                    grad_acc_steps=8,
                    save_dir=save_dir,
                    evaluate_every=20,
                    embed_title=False,
                    num_positives=1,
                    num_hard_negatives=1
                    )
```

**FAQ Check**

**System:**
 - OS: 16.04.1-Ubuntu
 - GPU/CPU: 4 * TITAN Xp / 87 * Intel(R) Xeon(R) CPU E5-2699A v4 @ 2.40GHz
 - Haystack version (commit or version number): v1.0.0
 - DocumentStore: FAISSDocumentStore
 - Reader: NA
 - Retriever: DensePassageRetriever

DPR training fails on multiple GPU's
**Describe the bug**
Following the DPR training [guide](https://haystack.deepset.ai/tutorials/train-dpr), in case trying to run on multiple GPU's, training will fail with a "AttributeError". Trying to call a `BiAdaptiveModel` method on a `DataParallel` object instance. 

**Error message**
AttributeError: 'DataParallel' object has no attribute 'connect_heads_with_processor'

**Expected behavior**
No error

**System:**
 - OS: Ubuntu
 - GPU/CPU: TITAN Xp
 - Haystack version (commit or version number): v0.10.0
 - DocumentStore: InMemoryDocumentStore
 - Reader: 'NA'
 - Retriever: DensePassageRetriever



Discussion:

Resolving patch:
diff --git a/haystack/modeling/data_handler/data_silo.py b/haystack/modeling/data_handler/data_silo.py
--- a/haystack/modeling/data_handler/data_silo.py
+++ b/haystack/modeling/data_handler/data_silo.py
@@ -44,6 +44,7 @@ def __init__(
         automatic_loading: bool = True,
         max_multiprocessing_chunksize: int = 2000,
         max_processes: int = 128,
+        multiprocessing_strategy: str = 'file_descriptor',
         caching: bool = False,
         cache_path: Path = Path("cache/data_silo"),
     ):
@@ -59,6 +60,9 @@ def __init__(
             values are rather large that might cause memory issues.
         :param max_processes: the maximum number of processes to spawn in the multiprocessing.Pool used in DataSilo.
                               It can be set to 1 to disable the use of multiprocessing or make debugging easier.
+        :multiprocessing_strategy: Set the multiprocessing sharing strategy, this can be one of file_descriptor/file_system.
+                                   If your system has low limits for the number of open file descriptors, and you can’t raise them,
+                                   you should use the file_system strategy.
         :param caching: save the processed datasets on disk to save time/compute if the same train data is used to run
                         multiple experiments. Each cache has a checksum based on the train_filename of the Processor
                         and the batch size.
@@ -70,6 +74,7 @@ def __init__(
         self.batch_size = batch_size
         self.class_weights = None
         self.max_processes = max_processes
+        self.multiprocessing_strategy = multiprocessing_strategy
         self.max_multiprocessing_chunksize = max_multiprocessing_chunksize
         self.caching = caching
         self.cache_path = cache_path
@@ -138,6 +143,14 @@ def _get_dataset(self, filename: Optional[Union[str, Path]], dicts: Optional[Lis
 
         with ExitStack() as stack:
             if self.max_processes > 1:  # use multiprocessing only when max_processes > 1
+                if self.multiprocessing_strategy and self.multiprocessing_strategy in ['file_descriptor', 'file_system']:
+                    mp.set_sharing_strategy(self.multiprocessing_strategy)
+                else:
+                    logger.warning(
+                        f"{self.multiprocessing_strategy} is an invalid strategy, "
+                        f"falling back to default (file_descriptor) strategy."
+                    )
+
                 p = stack.enter_context(mp.Pool(processes=num_cpus_used))
 
                 logger.info(
diff --git a/haystack/nodes/retriever/dense.py b/haystack/nodes/retriever/dense.py
--- a/haystack/nodes/retriever/dense.py
+++ b/haystack/nodes/retriever/dense.py
@@ -308,6 +308,7 @@ def train(self,
               test_filename: str = None,
               max_samples: int = None,
               max_processes: int = 128,
+              multiprocessing_strategy: str = 'file_descriptor',
               dev_split: float = 0,
               batch_size: int = 2,
               embed_title: bool = True,
@@ -337,6 +338,9 @@ def train(self,
         :param max_samples: maximum number of input samples to convert. Can be used for debugging a smaller dataset.
         :param max_processes: the maximum number of processes to spawn in the multiprocessing.Pool used in DataSilo.
                               It can be set to 1 to disable the use of multiprocessing or make debugging easier.
+        :param multiprocessing_strategy: Set the multiprocessing sharing strategy, this can be one of file_descriptor/file_system.
+                                         If your system has low limits for the number of open file descriptors, and you can’t raise them,
+                                         you should use the file_system strategy.
         :param dev_split: The proportion of the train set that will sliced. Only works if dev_filename is set to None
         :param batch_size: total number of samples in 1 batch of data
         :param embed_title: whether to concatenate passage title with each passage. The default setting in official DPR embeds passage title with the corresponding passage
@@ -377,7 +381,13 @@ def train(self,
         else:
             self.model.connect_heads_with_processor(self.processor.tasks, require_labels=True)
 
-        data_silo = DataSilo(processor=self.processor, batch_size=batch_size, distributed=False, max_processes=max_processes)
+        data_silo = DataSilo(
+            processor=self.processor,
+            batch_size=batch_size,
+            distributed=False,
+            max_processes=max_processes,
+            multiprocessing_strategy=multiprocessing_strategy
+        )
 
         # 5. Create an optimizer
         self.model, optimizer, lr_schedule = initialize_optimizer(


Test patch: