internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
Crawler node doesn't close Webdriver connection
**Describe the bug**
Crawler is not closing the Webdriver. Result after N different instances, are N chrome zombie processes.

**Error message**
`ps -aux` check zombie processes

**Expected behavior**
Crawler should correctly close Webdriver and clean unused resources

**Additional context**
This leads to exhaustion of resources on container.

Possible solution is implement `__del__()` to close Webdriver and clean resources

**To Reproduce**
1. Create a new instance of Crawler.
2. Crawl any page
3. Repeat process

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Linux
 - GPU/CPU: i7
 - Haystack version (commit or version number): 1.7.0


Discussion:

Resolving patch:
diff --git a/haystack/nodes/connector/crawler.py b/haystack/nodes/connector/crawler.py
--- a/haystack/nodes/connector/crawler.py
+++ b/haystack/nodes/connector/crawler.py
@@ -117,6 +117,9 @@ def __init__(
         self.loading_wait_time = loading_wait_time
         self.crawler_naming_function = crawler_naming_function
 
+    def __del__(self):
+        self.driver.quit()
+
     def crawl(
         self,
         output_dir: Union[str, Path, None] = None,


Test patch:
Repository: deepset-ai/haystack

Issue:
Pinecone: warn users about indexing metadata if filters return no documents
**Is your feature request related to a problem? Please describe.**
In `PineconeDocumentStore`, no error is raised if no metadata is indexed and the user tries to filter with it. This aligns with Pinecone's behavior, and at the moment, there is no way to check which fields are indexed. 

**Describe the solution you'd like**
We should add a warning if a user queries with a filter and nothing is returned. This way, we can point out that this may be a problem for users unaware of this behavior.

**Additional context**: https://github.com/deepset-ai/haystack/pull/3021#pullrequestreview-1070235487


Discussion:

Resolving patch:
diff --git a/haystack/document_stores/pinecone.py b/haystack/document_stores/pinecone.py
--- a/haystack/document_stores/pinecone.py
+++ b/haystack/document_stores/pinecone.py
@@ -636,6 +636,13 @@ def get_all_documents_generator(
                 namespace = self.document_namespace
 
         ids = self._get_all_document_ids(index=index, namespace=namespace, filters=filters, batch_size=batch_size)
+
+        if filters is not None and len(ids) == 0:
+            logger.warning(
+                "This query might have been done without metadata indexed and thus no DOCUMENTS were retrieved. "
+                "Make sure the desired metadata you want to filter with is indexed."
+            )
+
         for i in range(0, len(ids), batch_size):
             i_end = min(len(ids), i + batch_size)
             documents = self.get_documents_by_id(
@@ -1127,6 +1134,12 @@ def query_by_embedding(
             vector_id_matrix, meta_matrix, values=values, index=index, return_embedding=return_embedding
         )
 
+        if filters is not None and len(documents) == 0:
+            logger.warning(
+                "This query might have been done without metadata indexed and thus no results were retrieved. "
+                "Make sure the desired metadata you want to filter with is indexed."
+            )
+
         # assign query score to each document
         scores_for_vector_ids: Dict[str, float] = {str(v_id): s for v_id, s in zip(vector_id_matrix, score_matrix)}
         for doc in documents:


Test patch:
Repository: deepset-ai/haystack

Issue:
Tutorial 8, preprocessing: inconsistent suggested word count when splitting
### Discussed in https://github.com/deepset-ai/haystack/discussions/3103

<div type='discussions-op-text'>

<sup>Originally posted by **stevenhaley** August 25, 2022</sup>
The [Jupyter notebook](https://github.com/deepset-ai/haystack/blob/main/tutorials/Tutorial8_Preprocessing.ipynb) and [Python file](https://github.com/deepset-ai/haystack/blob/main/tutorials/Tutorial8_Preprocessing.py) for Tutorial 8, Preprocessing/splitting are not consistent about what document length is ideal. This is in the section with header "PreProcessor". They sometimes use 100, and sometimes 1_000.

The notebook markdown documentation recommends "100 words for dense retrieval methods", and uses that in the code (`split_by="word", split_length=100`). However, the in-code comments suggests 1_000. On the other hand, the Python code uses 1_000 in the comments and in the code itself (`split_by="word", split_length=1000`).

I assume that's meant to be 100 everywhere, as sentence transformers such as multi-qa-mpnet-base-dot-v1 have a maximum of 512 word pieces.

Also, if you're updating the code file anyway, you could fix the broken link `https://haystack.deepset.ai/docs/latest/preprocessingmd` in:
```
## PreProcessor
    
The PreProcessor class is designed to help you clean text and split text into sensible units.
File splitting can have a very significant impact on the system's performance.
Have a look at the [Preprocessing](https://haystack.deepset.ai/docs/latest/preprocessingmd)
and [Optimization](https://haystack.deepset.ai/docs/latest/optimizationmd) pages on our website for more details.
"""
```

PS: I wasn't sure if this was technically a bug report a discussion. I figure it's low priority, so a discussion is better.</div>


Discussion:

Resolving patch:
diff --git a/tutorials/Tutorial8_Preprocessing.py b/tutorials/Tutorial8_Preprocessing.py
--- a/tutorials/Tutorial8_Preprocessing.py
+++ b/tutorials/Tutorial8_Preprocessing.py
@@ -77,7 +77,7 @@ def tutorial8_preprocessing():
     # This is a default usage of the PreProcessor.
     # Here, it performs cleaning of consecutive whitespaces
     # and splits a single large document into smaller documents.
-    # Each document is up to 1000 words long and document breaks cannot fall in the middle of sentences
+    # Each document is up to 100 words long and document breaks cannot fall in the middle of sentences
     # Note how the single document passed into the document gets split into 5 smaller documents
 
     preprocessor = PreProcessor(
@@ -85,7 +85,7 @@ def tutorial8_preprocessing():
         clean_whitespace=True,
         clean_header_footer=False,
         split_by="word",
-        split_length=1000,
+        split_length=100,
         split_respect_sentence_boundary=True,
     )
     docs_default = preprocessor.process([doc_txt])


Test patch:
Repository: deepset-ai/haystack

Issue:
`ElasticSearchDocumentStore` does not use `batch_size` in `get_documents_by_id`
**Describe the bug**
`ElasticSearchDocumentStore.get_documents_by_id` takes a parameter called `batch_size`, but makes no use of it.

**Expected behavior**
`batch_size` should be used to retrieve documents in batches.

**Additional context**
https://github.com/deepset-ai/haystack/blob/b07fcb718541c9764b4400970f97c96a48f3254f/haystack/document_stores/elasticsearch.py#L267-L282

**FAQ Check**
- [X] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?




Discussion:

Resolving patch:
diff --git a/haystack/document_stores/elasticsearch.py b/haystack/document_stores/elasticsearch.py
--- a/haystack/document_stores/elasticsearch.py
+++ b/haystack/document_stores/elasticsearch.py
@@ -336,13 +336,27 @@ def get_documents_by_id(
         headers: Optional[Dict[str, str]] = None,
     ) -> List[Document]:
         """
-        Fetch documents by specifying a list of text id strings. Be aware that passing a large number of ids might lead
-        to performance issues. Note that Elasticsearch limits the number of results to 10,000 documents by default.
+        Fetch documents by specifying a list of text id strings.
+
+        :param ids: List of document IDs. Be aware that passing a large number of ids might lead to performance issues.
+        :param index: Elasticsearch index where the documents are stored. If not supplied,
+                      self.index will be used.
+        :param batch_size: Maximum number of results for each query.
+                           By default, Elasticsearch limits the number of results to 10,000 documents.
+                           To reduce the pressure on the Elasticsearch cluster, you can lower this limit, at the expense
+                           of longer retrieval times.
+        :param headers: Custom HTTP headers to pass to Elasticsearch client (e.g. {'Authorization': 'Basic YWRtaW46cm9vdA=='})
+                        Check out https://www.elastic.co/guide/en/elasticsearch/reference/current/http-clients.html for more information.
         """
         index = index or self.index
-        query = {"size": len(ids), "query": {"ids": {"values": ids}}}
-        result = self.client.search(index=index, body=query, headers=headers)["hits"]["hits"]
-        documents = [self._convert_es_hit_to_document(hit, return_embedding=self.return_embedding) for hit in result]
+        documents = []
+        for i in range(0, len(ids), batch_size):
+            ids_for_batch = ids[i : i + batch_size]
+            query = {"size": len(ids_for_batch), "query": {"ids": {"values": ids_for_batch}}}
+            result = self.client.search(index=index, body=query, headers=headers)["hits"]["hits"]
+            documents.extend(
+                [self._convert_es_hit_to_document(hit, return_embedding=self.return_embedding) for hit in result]
+            )
         return documents
 
     def get_metadata_values_by_key(


Test patch:
Repository: deepset-ai/haystack

Issue:
Small indentation error in FAISS
There's an if statement [here]( https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/faiss.py#L210) that will never hold true. It was reported by a community member on Discord [here](https://discord.com/channels/993534733298450452/1021883835971018812)


**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?




Discussion:
So, should we simply delete that `if`  block? 
Yes, I think so and deletion is also recommended in the Discord discussion. @avijit9 Would you be willing to open a PR for this? 

Resolving patch:
diff --git a/haystack/document_stores/faiss.py b/haystack/document_stores/faiss.py
--- a/haystack/document_stores/faiss.py
+++ b/haystack/document_stores/faiss.py
@@ -204,8 +204,6 @@ def _create_new_index(
             index = faiss.IndexHNSWFlat(embedding_dim, n_links, metric_type)
             index.hnsw.efSearch = ef_search
             index.hnsw.efConstruction = ef_construction
-            if "ivf" in index_factory.lower():  # enable reconstruction of vectors for inverted index
-                self.faiss_indexes[index].set_direct_map_type(faiss.DirectMap.Hashtable)
 
             logger.info(
                 f"HNSW params: n_links: {n_links}, efSearch: {index.hnsw.efSearch}, efConstruction: {index.hnsw.efConstruction}"


Test patch:
Repository: deepset-ai/haystack

Issue:
schema generation seems to depend on dependencies/environment
**Describe the bug**
Depending on which python env one's running `python .github/utils/generate_json_schema.py` this might produce schemas with arg types in different orders.

See run https://github.com/deepset-ai/haystack/actions/runs/3195756209/jobs/5216810534
of [this PR](https://github.com/deepset-ai/haystack/pull/3321). There was nothing changed between the failing run and the previously working run that should influence the schema.

**Error message**
It just complains about the order of types of args:
```
--- a/haystack/json-schemas/haystack-pipeline-1.10.0rc0.schema.json
+++ b/haystack/json-schemas/haystack-pipeline-1.10.0rc0.schema.json
@@ -3044,10 +3044,10 @@
               "title": "Use Auth Token",
               "anyOf": [
                 {
-                  "type": "boolean"
+                  "type": "string"
                 },
                 {
-                  "type": "string"
+                  "type": "boolean"
                 },
                 {
                   "type": "null"
```

**Expected behavior**
schema generation should be deterministic

**Additional context**
This is nasty as you always need to update your dependencies/environment before commiting/pushing

**To Reproduce**
See the PR above

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?



Discussion:
I have also seen this exact change in another PR: https://github.com/deepset-ai/haystack/pull/3318

Resolving patch:
diff --git a/haystack/nodes/_json_schema.py b/haystack/nodes/_json_schema.py
--- a/haystack/nodes/_json_schema.py
+++ b/haystack/nodes/_json_schema.py
@@ -380,6 +380,9 @@ def get_json_schema(filename: str, version: str, modules: List[str] = ["haystack
         ],
         "definitions": schema_definitions,
     }
+
+    # Leveraging an implementation detail of dict: keys stay in the order they are inserted.
+    pipeline_schema = dict(sorted(pipeline_schema.items(), key=lambda pair: pair[0]))
     return pipeline_schema
 
 
@@ -435,4 +438,4 @@ def update_json_schema(destination_path: Path = JSON_SCHEMAS_PATH):
         if new_entry not in index["oneOf"]:
             index["oneOf"].append(new_entry)
     with open(destination_path / index_name, "w") as json_file:
-        json.dump(index, json_file, indent=2)
+        json.dump(obj=index, fp=json_file, indent=2)


Test patch:
Repository: deepset-ai/haystack

Issue:
Telemetry can't be fully disabled
### Discussed in https://github.com/deepset-ai/haystack/discussions/3394

**Problem**
It seems like telemetry can't be completely disabled by setting `HAYSTACK_TELEMETRY_ENABLED` to `False`. See the discussion for more details and a preliminary analysis by @anakin87 @aschatzHM and @flawaetz


Discussion:

Resolving patch:
diff --git a/haystack/telemetry.py b/haystack/telemetry.py
--- a/haystack/telemetry.py
+++ b/haystack/telemetry.py
@@ -207,15 +207,16 @@ def send_tutorial_event(url: str):
     send_custom_event(event=f"tutorial {dataset_url_to_tutorial.get(url, '?')} executed")
 
 
-def _get_or_create_user_id() -> str:
+def _get_or_create_user_id() -> Optional[str]:
     """
     Randomly generates a user id or loads the id defined in the config file and returns it.
+    Returns None if no id has been set previously and a new one cannot be stored because telemetry is disabled
     """
     global user_id  # pylint: disable=global-statement
     if user_id is None:
         # if user_id is not set, read it from config file
         _read_telemetry_config()
-        if user_id is None:
+        if user_id is None and is_telemetry_enabled():
             # if user_id cannot be read from config file, create new user_id and write it to config file
             user_id = str(uuid.uuid4())
             _write_telemetry_config()


Test patch:
Repository: deepset-ai/haystack

Issue:
`PDFToTextOCRConverter.convert()` write temp files into the package folder
**Describe the bug**
- `PDFToTextOCRConverter.convert()` write temp files into the package folder: https://github.com/deepset-ai/haystack/blob/5ca96357ff526bf11aefa7fe4aa24fd11135cd0c/haystack/nodes/file_converter/pdf.py#L255
- This breaks the node in some contexts, i.e. conda on Windows, non-editable install.
- See https://discord.com/channels/993534733298450452/1034328904829247568/1034740296128409670 

**Error message**
```
ERROR:haystack.nodes.file_converter.pdf:File Machine_Learning.pdf has an error:
[Errno 13] Permission denied: 'C:\Users\bhati\anaconda3\envs\env_3\Lib\site-packages\haystack\nodes\file_converter\tmp4os8nw11.jpeg'
```

**Expected behavior**
- The file should be generated into a writable temp directory. `tempfile` does that by default. 
- Forcing the path to be in the package brings no benefit

**Additional context**
n/a

**To Reproduce**
- Install Haystack in non-editable mode with conda on Windows.
- Try to convert a file with PDFToTextOCRConverter

**FAQ Check**
- [X] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?



Discussion:

Resolving patch:
diff --git a/haystack/nodes/file_converter/pdf.py b/haystack/nodes/file_converter/pdf.py
--- a/haystack/nodes/file_converter/pdf.py
+++ b/haystack/nodes/file_converter/pdf.py
@@ -1,6 +1,5 @@
 from typing import List, Optional, Dict, Any
 
-import os
 import logging
 import tempfile
 import subprocess
@@ -252,7 +251,7 @@ def convert(
         try:
             images = convert_from_path(file_path)
             for image in images:
-                temp_img = tempfile.NamedTemporaryFile(dir=os.path.dirname(os.path.realpath(__file__)), suffix=".jpeg")
+                temp_img = tempfile.NamedTemporaryFile(suffix=".jpeg")
                 image.save(temp_img.name)
                 pages.append(self.image_2_text.convert(file_path=temp_img.name)[0].content)
         except Exception as exception:


Test patch:
Repository: deepset-ai/haystack

Issue:
Add tutorial for `MultiModalRetriever`



Discussion:
https://github.com/deepset-ai/haystack-tutorials/pull/52

Pull request created in Tutorials repo.

Resolving patch:
diff --git a/haystack/telemetry.py b/haystack/telemetry.py
--- a/haystack/telemetry.py
+++ b/haystack/telemetry.py
@@ -203,6 +203,7 @@ def send_tutorial_event(url: str):
         # "https://nlp.stanford.edu/data/glove.6B.zip": "16",
         "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/preprocessing_tutorial16.zip": "16",
         "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt17.zip": "17",
+        "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/spirit-animals.zip": "19",
     }
     send_custom_event(event=f"tutorial {dataset_url_to_tutorial.get(url, '?')} executed")
 


Test patch:
Repository: deepset-ai/haystack

Issue:
ParsrConverter invalid if statement
**Describe the bug**
I _believe_ that there's a bug right here: https://github.com/deepset-ai/haystack/blob/dc26e6d43ef23c1746722c613347e28f05982eee/haystack/nodes/file_converter/parsr.py#L189

This if statement might just end up throwing an error because in the case that the pdf is only a table, text will remain text = ""

**Error message**
```
File ~/opt/anaconda3/envs/haystack-tutorials/lib/python3.10/site-packages/haystack/nodes/file_converter/parsr.py:169, in ParsrConverter.convert(self, file_path, meta, remove_numeric_tables, valid_languages, encoding, id_hash_keys)
    165                 table = self._convert_table_element(
    166                     element, parsr_output["pages"], page_idx, elem_idx, meta, id_hash_keys
    167                 )
    168                 tables.append(table)
--> 169         if text[-1] != "\f":
    170             text += "\f"
    172 if valid_languages:
IndexError: string index out of range
```


**Expected behavior**
Check that text is not empty before checking the last char.

**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**
Use the pdf provided by community member [here](https://discord.com/channels/993534733298450452/1041694986321788970/1042049831960596632) which only has a table in it. And try to convert it to document using ParsrConverter

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:
I agree that this is a bug.
I reproduced it with the following code:

```
# docker run -d -p 3001:3001 axarev/parsr

from haystack.nodes import ParsrConverter
converter = ParsrConverter()
converter.convert("pdf-document.pdf")
```

I used this document: https://github.com/camelot-dev/camelot/blob/master/tests/files/medicine.pdf

About the PDF provided by the user, it should be said that it is image-based.
So, even if I remove those 2 lines of code, I have an empty output:
`[<Document: {'content': '', 'content_type': 'text', 'score': None, 'meta': {'headlines': []}, 'embedding': None, 'id': '00'}>]`.

I will submit a PR to fix this bug...

Resolving patch:
diff --git a/haystack/nodes/file_converter/parsr.py b/haystack/nodes/file_converter/parsr.py
--- a/haystack/nodes/file_converter/parsr.py
+++ b/haystack/nodes/file_converter/parsr.py
@@ -186,7 +186,7 @@ def convert(
                             id_hash_keys,
                         )
                         tables.append(table)
-                if text[-1] != "\f":
+                if len(text) == 0 or text[-1] != "\f":
                     text += "\f"
 
         if valid_languages:


Test patch:
Repository: deepset-ai/haystack

Issue:
Weaviate: enable BM25 queries with filters
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
When BM25 support was firstly introduced in 1.14, Weaviate didn't support passing filters along with the query and Haystack raises an exception when you call `query()` with both a query text and some filters.

**Describe the solution you'd like**
This is now supported in 1.16 and we should amend this part of the code https://github.com/deepset-ai/haystack/blob/005025bd146112970c5462dd7066aaf8a8a3e75c/haystack/document_stores/weaviate.py#L989-L997




Discussion:

Resolving patch:
diff --git a/haystack/document_stores/weaviate.py b/haystack/document_stores/weaviate.py
--- a/haystack/document_stores/weaviate.py
+++ b/haystack/document_stores/weaviate.py
@@ -982,19 +982,18 @@ def query(
 
             # Retrieval with BM25 AND filtering
             if filters:
-                raise NotImplementedError(
-                    "Weaviate currently (v1.14.1) does not support filters WITH inverted index text query (eg BM25)!"
-                )
 
                 # Once Weaviate starts supporting filters with BM25:
-                # filter_dict = LogicalFilterClause.parse(filters).convert_to_weaviate()
-                # gql_query = weaviate.gql.get.GetBuilder(class_name=index,
-                #                                         properties=properties,
-                #                                         connection=self.weaviate_client) \
-                #     .with_near_vector({'vector': [0, 0]}) \
-                #     .with_where(filter_dict) \
-                #     .with_limit(top_k) \
-                #     .build()
+                filter_dict = LogicalFilterClause.parse(filters).convert_to_weaviate()
+                gql_query = (
+                    weaviate.gql.get.GetBuilder(
+                        class_name=index, properties=properties, connection=self.weaviate_client
+                    )
+                    .with_near_vector({"vector": [0, 0]})
+                    .with_where(filter_dict)
+                    .with_limit(top_k)
+                    .build()
+                )
 
             # BM25 retrieval without filtering
             gql_query = (


Test patch:
diff --git a/test/document_stores/test_weaviate.py b/test/document_stores/test_weaviate.py
--- a/test/document_stores/test_weaviate.py
+++ b/test/document_stores/test_weaviate.py
@@ -155,8 +155,8 @@ def test_query(self, ds, documents):
         assert len(docs) == 3
 
         # BM25 retrieval WITH filters is not yet supported as of Weaviate v1.14.1
-        with pytest.raises(Exception):
-            docs = ds.query(query_text, filters={"name": ["filename2"]})
+        # with pytest.raises(Exception):
+        docs = ds.query(query_text, filters={"name": ["filename2"]})
 
         docs = ds.query(filters={"name": ["name_0"]})
         assert len(docs) == 3