internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
TCP port in `launch_opensearch()` is different from default value in `OpenSearchDocumentStore`
In `launch_opensearch()` we are starting an OpenSearch container using the port `9201`. The default port for `OpenSearchDocumentStore` is currently `9200`. I think we should align those two values.



Discussion:
I add my two cents: my impression is that port 9201 is used to allow the particular use case where a user calls both `launch_es` (default port: 9200) and `launch_opensearch`. Otherwise, there would be a conflict.

But in the document store code, the default port is 9200:
https://github.com/deepset-ai/haystack/blob/b84a6b17165dbf10665794b5decdefa113300749/haystack/document_stores/opensearch.py#L45

If we prefer to not change `launch_opensearch`, we should at least warn the user that the container is exposing port 9201.
WDYT?

Resolving patch:
diff --git a/haystack/utils/doc_store.py b/haystack/utils/doc_store.py
--- a/haystack/utils/doc_store.py
+++ b/haystack/utils/doc_store.py
@@ -37,7 +37,7 @@ def launch_es(sleep=15, delete_existing=False):
         time.sleep(sleep)
 
 
-def launch_opensearch(sleep=15, delete_existing=False):
+def launch_opensearch(sleep=15, delete_existing=False, local_port=9200):
     """
     Start an OpenSearch server via Docker.
     """
@@ -48,7 +48,7 @@ def launch_opensearch(sleep=15, delete_existing=False):
         _ = subprocess.run([f"docker rm --force {OPENSEARCH_CONTAINER_NAME}"], shell=True, stdout=subprocess.DEVNULL)
     status = subprocess.run(
         [
-            f'docker start {OPENSEARCH_CONTAINER_NAME} > /dev/null 2>&1 || docker run -d -p 9201:9200 -p 9600:9600 -e "discovery.type=single-node" --name {OPENSEARCH_CONTAINER_NAME} opensearchproject/opensearch:1.3.5'
+            f'docker start {OPENSEARCH_CONTAINER_NAME} > /dev/null 2>&1 || docker run -d -p {local_port}:9200 -p 9600:9600 -e "discovery.type=single-node" --name {OPENSEARCH_CONTAINER_NAME} opensearchproject/opensearch:1.3.5'
         ],
         shell=True,
     )


Test patch:
diff --git a/test/benchmarks/utils.py b/test/benchmarks/utils.py
--- a/test/benchmarks/utils.py
+++ b/test/benchmarks/utils.py
@@ -93,7 +93,7 @@ def get_document_store(document_store_type, similarity="dot_product", index="doc
         )
         assert document_store.get_document_count() == 0
     elif document_store_type in ("opensearch_flat", "opensearch_hnsw"):
-        launch_opensearch()
+        launch_opensearch(local_port=9201)
         if document_store_type == "opensearch_flat":
             index_type = "flat"
         elif document_store_type == "opensearch_hnsw":
Repository: deepset-ai/haystack

Issue:
Bad Semaphore initialization in RequestLimiter
**Describe the bug**
RequestLimiter takes a number as parameter and use it to set up a Semaphore. The issue is that the environment variable indicates the concurrent allowed requests per worker. When the semaphore is created (https://github.com/deepset-ai/haystack/blob/6790eaf7d8be05c5674d97a75cc5783e00a66875/rest_api/rest_api/controller/utils.py#L13), this value is set down by 1. This is clearly not what the project tried to achieve (at least per naming). 

**Error message**
REST API will always return it's busy, error 503 when CONCURRENT_REQUEST_PER_WORKER is equal to CONCURRENT_REQUEST_PER_WORKER -1. When user set the concurrency to 1, it will never be able to call the API, since the Semaphore declaration will be Semaphore(0)

**Expected behavior**
Being able to set the request limits using the env variable CONCURRENT_REQUEST_PER_WORKER

**Additional context**


**To Reproduce**


**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Ubuntu
 - GPU/CPU: i7/ Nvidia
 - Haystack version (commit or version number): 1.9
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/rest_api/rest_api/controller/utils.py b/rest_api/rest_api/controller/utils.py
--- a/rest_api/rest_api/controller/utils.py
+++ b/rest_api/rest_api/controller/utils.py
@@ -10,7 +10,7 @@
 
 class RequestLimiter:
     def __init__(self, limit):
-        self.semaphore = Semaphore(limit - 1)
+        self.semaphore = Semaphore(limit)
 
     @contextmanager
     def run(self):


Test patch:
Repository: deepset-ai/haystack

Issue:
Code error in the deepsetcloud document store
**Describe the bug**
Take a look at this line https://github.com/deepset-ai/haystack/blob/600dc2d61199dc1cc87186d8703fbb22f3c6dac7/haystack/document_stores/deepsetcloud.py#L556

It is over-indented. That `for` loop should NOT be under the `else:`, but in its own block.
The `if-else` statement is to handle `filters` being a list of filters vs a single filter value, it should have NO influence on when to run the `for` loop.

**Error message**
No error, this is a logical error in the code resulting in an empty document list returned by the `query_batch()` method when the `filters` is a list.

**Expected behavior**
I guess its obvious...

**Additional context**
N/A

**To Reproduce**
Just run the `query_batch()` for the deepsetcloud document store with a filter list and you will never get any results back, as it never actually runs the query.

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore: deepsetcloud
 - Reader:
 - Retriever: sparse



Discussion:

Resolving patch:
diff --git a/haystack/document_stores/deepsetcloud.py b/haystack/document_stores/deepsetcloud.py
--- a/haystack/document_stores/deepsetcloud.py
+++ b/haystack/document_stores/deepsetcloud.py
@@ -553,18 +553,19 @@ def query_batch(
                 )
         else:
             filters = [filters] * len(queries) if filters is not None else [{}] * len(queries)
-            for query, cur_filters in zip(queries, filters):
-                cur_docs = self.query(
-                    query=query,
-                    filters=cur_filters,
-                    top_k=top_k,
-                    custom_query=custom_query,
-                    index=index,
-                    headers=headers,
-                    all_terms_must_match=all_terms_must_match,
-                    scale_score=scale_score,
-                )
-                documents.append(cur_docs)
+
+        for query, cur_filters in zip(queries, filters):
+            cur_docs = self.query(
+                query=query,
+                filters=cur_filters,
+                top_k=top_k,
+                custom_query=custom_query,
+                index=index,
+                headers=headers,
+                all_terms_must_match=all_terms_must_match,
+                scale_score=scale_score,
+            )
+            documents.append(cur_docs)
 
         return documents
 


Test patch:
Repository: deepset-ai/haystack

Issue:
launch_milvus() fails as milvus_dir != '/home/$USER/milvus/'
**Describe the bug**
launch_milvus() fails when `Path.home() ` is not same as `/home/$USER/milvus/`

milvus docker-compose.yml is stored inside Path.home()

```
  milvus_dir = Path.home() / "milvus"
 ......
  with open(milvus_dir / "docker-compose.yml", "wb") as f:
        f.write(request.content)
```

While docker compose is initiated using
```
status = subprocess.run(["cd /home/$USER/milvus/ && docker-compose up -d"], shell=True)
```

**Error message**
Error that was thrown (if available)
```
/bin/sh: line 0: cd: /home/bhoumis/milvus/: No such file or directory
WARNING - __main__ -  Tried to start Milvus through Docker but this failed. It is likely that there is already an existing Milvus instance running. 
```
**Expected behavior**
A clear and concise description of what you expected to happen.
```
status = subprocess.run([f"cd {milvus_dir} && docker-compose up -d"], shell=True)
```
**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**
Steps to reproduce the behavior
`
from haystack.utils import launch_milvus
launch_milvus()
`

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: macOS Monterey v12.6.1
 - GPU/CPU: M1-Pro
 - Haystack version (commit or version number):  v1.12.2
 - DocumentStore: MilvusDocumentStore(
 - Reader: NA
 - Retriever: NA



Discussion:

Resolving patch:
diff --git a/haystack/utils/doc_store.py b/haystack/utils/doc_store.py
--- a/haystack/utils/doc_store.py
+++ b/haystack/utils/doc_store.py
@@ -133,7 +133,7 @@ def launch_milvus(sleep=15, delete_existing=False):
     with open(milvus_dir / "docker-compose.yml", "wb") as f:
         f.write(request.content)
 
-    status = subprocess.run(["cd /home/$USER/milvus/ && docker-compose up -d"], shell=True)
+    status = subprocess.run([f"cd {milvus_dir} && docker-compose up -d"], shell=True)
 
     if status.returncode:
         logger.warning(


Test patch:
Repository: deepset-ai/haystack

Issue:
Better error messages for OCR requirements
**Describe the bug**
Some error messages are not clear enough for users to take necessary actions.

e.g. `haystack.errors.PipelineSchemaError: Haystack component with the name 'PDFToTextConverter' not found.` gives a message that `PDFToTextConverter` is missing. It's suitable if `PDFToTextConverter` class is deleted or removed but the same error raises when either `pdf2image` or `pytesseract` packages are not installed which come with `ocr` dependency option. Then, the message is misleading.  

**Error message**
Currently only: `haystack.errors.PipelineSchemaError: Haystack component with the name 'PDFToTextConverter' not found.` Check #3201 for the full error log.

**Expected behavior**
The error message should direct users to install OCR dependencies. Something like: 
```
...
"Install all OCR tools to use `PDFToTextConverter` using `pip install 'farm-haystack[ocr]'`"
```

**To Reproduce**
```bash
git clone https://github.com/deepset-ai/haystack.git
cd haystack
pip install haystack/ #or pip install farm-haystack
pip install rest_api/
gunicorn rest_api.application:app -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker -t 300
```
You'll get the error. To fix the error, run `pip install 'farm-haystack[ocr]'` and run `gunicorn rest_api.application:app -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker -t 300`

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: macOS Monterey
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 1.12



Discussion:

Resolving patch:
diff --git a/haystack/nodes/base.py b/haystack/nodes/base.py
--- a/haystack/nodes/base.py
+++ b/haystack/nodes/base.py
@@ -121,7 +121,12 @@ def get_params(self, return_defaults: bool = False) -> Dict[str, Any]:
     @classmethod
     def get_subclass(cls, component_type: str) -> Type[BaseComponent]:
         if component_type not in cls._subclasses.keys():
-            raise PipelineSchemaError(f"Haystack component with the name '{component_type}' not found.")
+            raise PipelineSchemaError(
+                f"Haystack component with the name '{component_type}' not found. "
+                "Check the class name of your component for spelling mistakes and make sure you installed "
+                "Haystack with the proper extras: https://docs.haystack.deepset.ai/docs/installation#custom-installation"
+            )
+
         subclass = cls._subclasses[component_type]
         return subclass
 


Test patch:
Repository: deepset-ai/haystack

Issue:
bug: MarkdownConverter not removing code blocks
The `MarkdownConverter` does not remove code blocks in the scenario I have at hand. The first thing that seems to happen is that the markdown gets converter to html and then the code looks for `<pre>` or `<code>` blocks. However, the html produced from our tutorials for example has code blocks as `<p>` ``` ... ``` `</p>`

I am able to fix this by adding the following line [here](https://github.com/deepset-ai/haystack/blob/d962bc0bc95ad1870e37b59f5aef4b6842b2df58/haystack/nodes/file_converter/markdown.py#L90):
`html = re.sub(r"<p>```(.*?)```</p>", " ", html, flags=re.DOTALL)
` 

If this is an ok fix, I'm happy to provide a PR


Discussion:

Resolving patch:
diff --git a/haystack/nodes/file_converter/markdown.py b/haystack/nodes/file_converter/markdown.py
--- a/haystack/nodes/file_converter/markdown.py
+++ b/haystack/nodes/file_converter/markdown.py
@@ -93,7 +93,7 @@ def convert(
             metadata, markdown_text = frontmatter.parse(f.read())
 
         # md -> html -> text since BeautifulSoup can extract text cleanly
-        html = markdown(markdown_text)
+        html = markdown(markdown_text, extensions=["fenced_code"])
 
         # remove code snippets
         if remove_code_snippets:


Test patch:
diff --git a/test/nodes/test_file_converter.py b/test/nodes/test_file_converter.py
--- a/test/nodes/test_file_converter.py
+++ b/test/nodes/test_file_converter.py
@@ -143,7 +143,8 @@ def test_docx_converter():
 def test_markdown_converter():
     converter = MarkdownConverter()
     document = converter.convert(file_path=SAMPLES_PATH / "markdown" / "sample.md")[0]
-    assert document.content.startswith("What to build with Haystack")
+    assert document.content.startswith("\nWhat to build with Haystack")
+    assert "# git clone https://github.com/deepset-ai/haystack.git" not in document.content
 
 
 def test_markdown_converter_headline_extraction():
@@ -178,6 +179,12 @@ def test_markdown_converter_frontmatter_to_meta():
     assert document.meta["date"] == "1.1.2023"
 
 
+def test_markdown_converter_remove_code_snippets():
+    converter = MarkdownConverter(remove_code_snippets=False)
+    document = converter.convert(file_path=SAMPLES_PATH / "markdown" / "sample.md")[0]
+    assert document.content.startswith("pip install farm-haystack")
+
+
 def test_azure_converter():
     # Check if Form Recognizer endpoint and credential key in environment variables
     if "AZURE_FORMRECOGNIZER_ENDPOINT" in os.environ and "AZURE_FORMRECOGNIZER_KEY" in os.environ:
diff --git a/test/samples/markdown/sample.md b/test/samples/markdown/sample.md
--- a/test/samples/markdown/sample.md
+++ b/test/samples/markdown/sample.md
@@ -2,6 +2,9 @@
 type: intro
 date: 1.1.2023
 ---
+```bash
+pip install farm-haystack
+```
 ## What to build with Haystack
 
 - **Ask questions in natural language** and find granular answers in your own documents.
Repository: deepset-ai/haystack

Issue:
PromptNode has no `run_batch` implementation
**Describe the bug**
The `run_batch` method for the [PromptNode](https://github.com/deepset-ai/haystack/blob/9e457db2e96248e107c39ab78e5f8d6dbd75d755/haystack/nodes/prompt/prompt_node.py#L893) says just `pass`.

**Expected behavior**
The method should either be properly implemented or raise a meaningful error - if the latter, I would expect some docstrings explaining the situation.

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/haystack/nodes/prompt/prompt_node.py b/haystack/nodes/prompt/prompt_node.py
--- a/haystack/nodes/prompt/prompt_node.py
+++ b/haystack/nodes/prompt/prompt_node.py
@@ -931,7 +931,7 @@ def run_batch(
         params: Optional[dict] = None,
         debug: Optional[bool] = None,
     ):
-        pass
+        raise NotImplementedError("run_batch is not implemented for PromptNode.")
 
     def _prepare_model_kwargs(self):
         # these are the parameters from PromptNode level


Test patch:
Repository: deepset-ai/haystack

Issue:
RayPipeline with a decision node fails
**Describe the bug**
RayPipeline with a decision node fails.

**Error message**
```
File ~/.local/share/virtualenvs/llm_demo-hW1oTPAF/src/farm-haystack/haystack/pipelines/ray.py:255, in RayPipeline._add_ray_deployment_in_graph(self, handle, name, outgoing_edges, inputs)
    253     assert "output_" in input_edge_name, f"'{input_edge_name}' is not a valid edge name."
    254     outgoing_edges_input_node = self.graph.nodes[input_node_name]["component"].outgoing_edges
--> 255     assert int(input_edge_name.split("_")[1]) <= outgoing_edges_input_node, (
    256         f"Cannot connect '{input_edge_name}' from '{input_node_name}' as it only has "
    257         f"{outgoing_edges_input_node} outgoing edge(s)."
    258     )
    259 else:
    260     outgoing_edges_input_node = self.graph.nodes[i]["outgoing_edges"]

TypeError: '<=' not supported between instances of 'int' and 'RayServeSyncHandle'
```
**Expected behavior**
It should handle `.output_1` as input, just like the normal pipeline does

**Additional context**
None

**To Reproduce**
Any decision node in RayPipeline

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Any 
 - GPU/CPU: Any
 - Haystack version (commit or version number): latest
 - DocumentStore: Any
 - Reader: Any
 - Retriever: Any



Discussion:

Resolving patch:
diff --git a/haystack/pipelines/ray.py b/haystack/pipelines/ray.py
--- a/haystack/pipelines/ray.py
+++ b/haystack/pipelines/ray.py
@@ -251,7 +251,7 @@ def _add_ray_deployment_in_graph(self, handle, name: str, outgoing_edges: int, i
             if "." in i:
                 [input_node_name, input_edge_name] = i.split(".")
                 assert "output_" in input_edge_name, f"'{input_edge_name}' is not a valid edge name."
-                outgoing_edges_input_node = self.graph.nodes[input_node_name]["component"].outgoing_edges
+                outgoing_edges_input_node = self.graph.nodes[input_node_name]["outgoing_edges"]
                 assert int(input_edge_name.split("_")[1]) <= outgoing_edges_input_node, (
                     f"Cannot connect '{input_edge_name}' from '{input_node_name}' as it only has "
                     f"{outgoing_edges_input_node} outgoing edge(s)."


Test patch:
Repository: deepset-ai/haystack

Issue:
feat: Support the use of multiple `RayPipelines` from the same app
**Is your feature request related to a problem? Please describe.**
Currently you can only run one `RayPipeline` within one app, because the `RayPipeline` always tries to initialize a new connection - if one is already initialized then it will simply fail.
See the Ray initialization at https://github.com/deepset-ai/haystack/blob/main/haystack/pipelines/ray.py#L80

Due to that, the app will fail when it starts to initialize the second `RayPipeline`, as the first one has already initialized the Ray connection and an error will be thrown:
```
  File "/app/raypipelines/qa_pipeline.py", line 87, in __init__
    self.raypipeline = RayPipeline.load_from_yaml(
  File "/hW1oTPAF/src/farm-haystack/haystack/pipelines/ray.py", line 193, in load_from_yaml
    return cls.load_from_config(
  File "/hW1oTPAF/src/farm-haystack/haystack/pipelines/ray.py", line 101, in load_from_config
    pipeline = cls(address=address, ray_args=ray_args or {}, serve_args=serve_args or {})
  File "/hW1oTPAF/src/farm-haystack/haystack/pipelines/ray.py", line 80, in __init__
    ray.init(address=address, **ray_args)
  File "/hW1oTPAF/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/hW1oTPAF/lib/python3.10/site-packages/ray/_private/worker.py", line 1255, in init
    ctx = builder.connect()
  File "/hW1oTPAF/lib/python3.10/site-packages/ray/client_builder.py", line 182, in connect
    client_info_dict = ray.util.client_connect.connect(
  File "/hW1oTPAF/lib/python3.10/site-packages/ray/util/client_connect.py", line 44, in connect
    raise RuntimeError(
RuntimeError: Ray Client is already connected. Maybe you called ray.init("ray://<address>") twice by accident?
```

**Describe the solution you'd like**
This could be fixed relatively easily just by checking if Ray has already been initialized and if yes, then skip initialization.
Obviously this also means that the second RayPipeline's parameters will NOT be applied to the Ray connection, as you can only have one of those - which the first RayPipeline has already opened, so throwing a warning log is warranted.

**Describe alternatives you've considered**
There isn't one.

**Additional context**
I will make a PR for this.



Discussion:

Resolving patch:
diff --git a/haystack/pipelines/ray.py b/haystack/pipelines/ray.py
--- a/haystack/pipelines/ray.py
+++ b/haystack/pipelines/ray.py
@@ -77,7 +77,10 @@ def __init__(
         :param serve_args: Optional parameters for initializing Ray Serve.
         """
         ray_args = ray_args or {}
-        ray.init(address=address, **ray_args)
+        if not ray.is_initialized():
+            ray.init(address=address, **ray_args)
+        else:
+            logger.warning("Ray was already initialized, so reusing that for this RayPipeline.")
         self._serve_controller_client = serve.start(**serve_args)
         super().__init__()
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Update PR template
**Describe the bug**
Can you please update the PR template so that one of the checks is:

If this change requires documentation or docstrings updates, I added the documentation team as reviewers. 

**Error message**
na

**Expected behavior**
There's a check ensuring that the documentation team is notified of any changes.

**Additional context**
The automatic notifications don't work any longer and the docs team is often not notified about changes to doc strings or PRs that may need documentation changes.

**To Reproduce**
na

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/.github/utils/docstrings_checksum.py b/.github/utils/docstrings_checksum.py
new file mode 100644
--- /dev/null
+++ b/.github/utils/docstrings_checksum.py
@@ -0,0 +1,43 @@
+from pathlib import Path
+from typing import Iterator
+
+import ast
+import hashlib
+
+
+def docstrings_checksum(python_files: Iterator[Path]):
+    files_content = (f.read_text() for f in python_files)
+    trees = (ast.parse(c) for c in files_content)
+
+    # Get all docstrings from async functions, functions,
+    # classes and modules definitions
+    docstrings = []
+    for tree in trees:
+        for node in ast.walk(tree):
+            if not isinstance(node, (ast.AsyncFunctionDef, ast.FunctionDef, ast.ClassDef, ast.Module)):
+                # Skip all node types that can't have docstrings to prevent failures
+                continue
+            docstring = ast.get_docstring(node)
+            if docstring:
+                docstrings.append(docstring)
+
+    # Sort them to be safe, since ast.walk() returns
+    # nodes in no specified order.
+    # See https://docs.python.org/3/library/ast.html#ast.walk
+    docstrings.sort()
+
+    return hashlib.md5(str(docstrings).encode("utf-8")).hexdigest()
+
+
+if __name__ == "__main__":
+    # Get all Haystack and rest_api python files
+    root = Path(__file__).parent.parent.parent
+    haystack_files = root.glob("haystack/**/*.py")
+    rest_api_files = root.glob("rest_api/**/*.py")
+
+    import itertools
+
+    python_files = itertools.chain(haystack_files, rest_api_files)
+
+    md5 = docstrings_checksum(python_files)
+    print(md5)


Test patch: