internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
Create minimal indexing pipeline for Haystack 2.0
**Is your feature request related to a problem? Please describe.**
We want to showcase indexing pipeline in v2.

**Describe the solution you'd like**
We need example code and/or end2end tests showing: 
- an indexing pipeline taking as input: either a folder containing multiple files, or a single file
- convert files of various types (PDF, docx, txt)
- usage of the preprocessor
- creation of embeddings
- indexing to a docstore for both sparse and dense retrieval 





Discussion:

Resolving patch:
diff --git a/examples/pipelines/indexing_pipeline.py b/examples/pipelines/indexing_pipeline.py
new file mode 100644
--- /dev/null
+++ b/examples/pipelines/indexing_pipeline.py
@@ -0,0 +1,37 @@
+from pathlib import Path
+
+from haystack import Pipeline
+from haystack.components.embedders import SentenceTransformersDocumentEmbedder
+from haystack.components.converters import PyPDFToDocument, TextFileToDocument
+from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
+from haystack.components.routers import FileTypeRouter, DocumentJoiner
+from haystack.components.writers import DocumentWriter
+from haystack.document_stores import InMemoryDocumentStore
+
+
+# Create components and an indexing pipeline that converts txt and pdf files to documents, cleans and splits them, and
+# indexes them for sparse and dense retrieval.
+p = Pipeline()
+p.add_component(instance=FileTypeRouter(mime_types=["text/plain", "application/pdf"]), name="file_type_router")
+p.add_component(instance=TextFileToDocument(), name="text_file_converter")
+p.add_component(instance=PyPDFToDocument(), name="pdf_file_converter")
+p.add_component(instance=DocumentJoiner(), name="joiner")
+p.add_component(instance=DocumentCleaner(), name="cleaner")
+p.add_component(instance=DocumentSplitter(split_by="sentence", split_length=250, split_overlap=30), name="splitter")
+p.add_component(
+    instance=SentenceTransformersDocumentEmbedder(model_name_or_path="sentence-transformers/all-MiniLM-L6-v2"),
+    name="embedder",
+)
+p.add_component(instance=DocumentWriter(document_store=InMemoryDocumentStore()), name="writer")
+
+p.connect("file_type_router.text/plain", "text_file_converter.sources")
+p.connect("file_type_router.application/pdf", "pdf_file_converter.sources")
+p.connect("text_file_converter.documents", "joiner.documents")
+p.connect("pdf_file_converter.documents", "joiner.documents")
+p.connect("joiner.documents", "cleaner.documents")
+p.connect("cleaner.documents", "splitter.documents")
+p.connect("splitter.documents", "embedder.documents")
+p.connect("embedder.documents", "writer.documents")
+
+# Take the current directory as input and run the pipeline
+result = p.run({"file_type_router": {"sources": list(Path(".").iterdir())}})


Test patch:
Repository: deepset-ai/haystack

Issue:
`debug=True` in pipeline.run creates errors with `LinkContentFetcher`

To reproduce: use this [colab](https://colab.research.google.com/drive/1nuC_PsZjArnVZhkhy7LQPPMUUh73ctp8?usp=sharing)

At the end, adding `debug=True` will produce the following error:

```
TypeError: descriptor '__dict__' for 'LinkContentFetcher' objects doesn't apply to a 'LinkContentFetcher' object
```


Discussion:

Resolving patch:
diff --git a/haystack/core/pipeline/draw/draw.py b/haystack/core/pipeline/draw/draw.py
--- a/haystack/core/pipeline/draw/draw.py
+++ b/haystack/core/pipeline/draw/draw.py
@@ -45,14 +45,6 @@ def _draw(
     logger.debug("Pipeline diagram saved at %s", path)
 
 
-def _convert_for_debug(graph: networkx.MultiDiGraph) -> Any:
-    """
-    Renders the pipeline graph with additional debug information into a text file that Mermaid can later render.
-    """
-    graph = _prepare_for_drawing(graph=graph, style_map={})
-    return _to_mermaid_text(graph=graph)
-
-
 def _convert(
     graph: networkx.MultiDiGraph, engine: RenderingEngines = "mermaid-image", style_map: Optional[Dict[str, str]] = None
 ) -> Any:
diff --git a/haystack/core/pipeline/pipeline.py b/haystack/core/pipeline/pipeline.py
--- a/haystack/core/pipeline/pipeline.py
+++ b/haystack/core/pipeline/pipeline.py
@@ -23,7 +23,7 @@
     PipelineValidationError,
 )
 from haystack.core.pipeline.descriptions import find_pipeline_outputs
-from haystack.core.pipeline.draw.draw import _draw, _convert_for_debug, RenderingEngines
+from haystack.core.pipeline.draw.draw import _draw, RenderingEngines
 from haystack.core.pipeline.validation import validate_pipeline_input, find_pipeline_inputs
 from haystack.core.component.connection import Connection, parse_connect_string
 from haystack.core.type_utils import _type_name
@@ -519,14 +519,12 @@ def _record_pipeline_step(
         """
         Stores a snapshot of this step into the self.debug dictionary of the pipeline.
         """
-        mermaid_graph = _convert_for_debug(deepcopy(self.graph))
         self._debug[step] = {
             "time": datetime.datetime.now(),
             "components_queue": components_queue,
             "mandatory_values_buffer": mandatory_values_buffer,
             "optional_values_buffer": optional_values_buffer,
             "pipeline_output": pipeline_output,
-            "diagram": mermaid_graph,
         }
 
     def _clear_visits_count(self):


Test patch:
Repository: deepset-ai/haystack

Issue:
Add `to_dict` and `from_dict` methods in `HTMLToDocument`
`HTMLToDocument` Component is missing serialization methods.

We need to add them so it can be properly serialized.


Discussion:

Resolving patch:
diff --git a/haystack/components/converters/html.py b/haystack/components/converters/html.py
--- a/haystack/components/converters/html.py
+++ b/haystack/components/converters/html.py
@@ -3,7 +3,7 @@
 from typing import Any, Dict, List, Optional, Union, Literal
 from boilerpy3 import extractors
 
-from haystack import Document, component
+from haystack import Document, component, default_from_dict, default_to_dict
 from haystack.dataclasses import ByteStream
 from haystack.components.converters.utils import get_bytestream_from_source, normalize_metadata
 
@@ -49,6 +49,13 @@ def __init__(
         """
         self.extractor_type = extractor_type
 
+    def to_dict(self) -> Dict[str, Any]:
+        return default_to_dict(self, extractor_type=self.extractor_type)
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "HTMLToDocument":
+        return default_from_dict(cls, data)
+
     @component.output_types(documents=List[Document])
     def run(
         self,


Test patch:
diff --git a/test/components/converters/test_html_to_document.py b/test/components/converters/test_html_to_document.py
--- a/test/components/converters/test_html_to_document.py
+++ b/test/components/converters/test_html_to_document.py
@@ -160,3 +160,12 @@ def test_mixed_sources_run(self, test_files_path):
         assert len(docs) == 3
         for doc in docs:
             assert "Haystack" in doc.content
+
+    def test_serde(self):
+        """
+        Test if the component runs correctly gets serialized and deserialized.
+        """
+        converter = HTMLToDocument("ArticleExtractor")
+        serde_data = converter.to_dict()
+        new_converter = HTMLToDocument.from_dict(serde_data)
+        assert new_converter.extractor_type == converter.extractor_type
Repository: deepset-ai/haystack

Issue:
PromptNode does't support 16,385 tokens for new gpt-3.5-turbo-1106
**Describe the bug**
Prompt Node is setting a limit of 4096 total tokens on the new gpt-3.5-turbo-1106 model. 

**Error message**
If the max_length is set to 4096, this truncates the prompt to 0 allowed tokens. 
i.e If max_length: 4096
\nException while running node 'PromptNode': No prompt provided. Model gpt-3.5-turbo-1106 requires prompt.Make sure to provide prompt in kwargs.\nEnable debug logging to see the data that was passed when the pipeline failed."

**Expected behavior**
gpt-3.5-turbo-1106 should permit a total context window of 16,385 tokens, and a maximum of 4096 tokens as output. In other words when max_length: 4096, the prompt should allow a maximum of '12289 tokens' as opposed to '0' as it is at the moment. 


**To Reproduce**
Using model gpt-3.5-turbo-1106 in Prompt node set max_length to 4096 and observe a 0 token allowance for the prompt. 


**System:**
 - Haystack version (commit or version number): 1.23



Discussion:

Resolving patch:
diff --git a/haystack/utils/openai_utils.py b/haystack/utils/openai_utils.py
--- a/haystack/utils/openai_utils.py
+++ b/haystack/utils/openai_utils.py
@@ -82,6 +82,9 @@ def _openai_text_completion_tokenization_details(model_name: str):
         elif model_name.startswith("gpt-3.5-turbo-16k") or model_name.startswith("gpt-35-turbo-16k"):
             max_tokens_limit = 16384
             tokenizer_name = model_tokenizer
+        elif model_name.startswith("gpt-3.5-turbo-1106") or model_name.startswith("gpt-35-turbo-1106"):
+            max_tokens_limit = 16384
+            tokenizer_name = model_tokenizer
         elif model_name.startswith("gpt-3"):
             max_tokens_limit = 4096
             tokenizer_name = model_tokenizer


Test patch:
diff --git a/test/utils/test_openai_utils.py b/test/utils/test_openai_utils.py
--- a/test/utils/test_openai_utils.py
+++ b/test/utils/test_openai_utils.py
@@ -41,6 +41,13 @@ def test_openai_text_completion_tokenization_details_gpt3_5():
     assert max_tokens_limit == 4096
 
 
+@pytest.mark.unit
+def test_openai_text_completion_tokenization_details_gpt3_5_1106():
+    tokenizer_name, max_tokens_limit = _openai_text_completion_tokenization_details(model_name="gpt-3.5-turbo-1106")
+    assert tokenizer_name == "cl100k_base"
+    assert max_tokens_limit == 16384
+
+
 @pytest.mark.unit
 def test_openai_text_completion_tokenization_details_gpt_4():
     tokenizer_name, max_tokens_limit = _openai_text_completion_tokenization_details(model_name="gpt-4")
Repository: deepset-ai/haystack

Issue:
`Pipeline.connect()` should return the `Pipeline` instance
When calling `connect()` it would be useful if the method returns the instance of the `Pipeline` too. 

This way one could chain multiple connect together with ease.

```
pipe = Pipeline()

pipe.connect("first", "second").connect("second", "third")
```

This could also be useful when creating a `Pipeline` in a notebook, as usually `connect()` is the last statement of the cell we can also show a representation of the `Pipeline`.


Discussion:

Resolving patch:
diff --git a/haystack/core/pipeline/pipeline.py b/haystack/core/pipeline/pipeline.py
--- a/haystack/core/pipeline/pipeline.py
+++ b/haystack/core/pipeline/pipeline.py
@@ -210,7 +210,7 @@ def add_component(self, name: str, instance: Component) -> None:
             visits=0,
         )
 
-    def connect(self, connect_from: str, connect_to: str) -> None:
+    def connect(self, connect_from: str, connect_to: str) -> "Pipeline":
         """
         Connects two components together. All components to connect must exist in the pipeline.
         If connecting to an component that has several output connections, specify the inputs and output names as
@@ -223,7 +223,7 @@ def connect(self, connect_from: str, connect_to: str) -> None:
                 in the format `component_name.connection_name` if the component has multiple inputs.
 
         Returns:
-            None
+            The Pipeline instance
 
         Raises:
             PipelineConnectError: if the two components cannot be connected (for example if one of the components is
@@ -339,7 +339,7 @@ def connect(self, connect_from: str, connect_to: str) -> None:
 
         if receiver in sender_socket.receivers and sender in receiver_socket.senders:
             # This is already connected, nothing to do
-            return
+            return self
 
         if receiver_socket.senders and not receiver_socket.is_variadic:
             # Only variadic input sockets can receive from multiple senders
@@ -363,6 +363,7 @@ def connect(self, connect_from: str, connect_to: str) -> None:
             to_socket=receiver_socket,
             mandatory=receiver_socket.is_mandatory,
         )
+        return self
 
     def get_component(self, name: str) -> Component:
         """


Test patch:
diff --git a/test/core/pipeline/test_connect.py b/test/core/pipeline/test_connect.py
--- a/test/core/pipeline/test_connect.py
+++ b/test/core/pipeline/test_connect.py
@@ -18,7 +18,7 @@ def test_connect():
     pipe = Pipeline()
     pipe.add_component("comp1", comp1)
     pipe.add_component("comp2", comp2)
-    pipe.connect("comp1.value", "comp2.value")
+    assert pipe.connect("comp1.value", "comp2.value") is pipe
 
     assert comp1.__haystack_output__.value.receivers == ["comp2"]
     assert comp2.__haystack_input__.value.senders == ["comp1"]
Repository: deepset-ai/haystack

Issue:
Pipeline drawings in Colab have black background
**Describe the bug**
Since Haystack 2.0-beta8, Pipeline drawings in Colab and other environments (VS Code/Pycharm) randomly
have a black background.

![image](https://github.com/deepset-ai/haystack/assets/44616784/0869599a-14b5-4aed-9e96-a8b09d83658a)

These images are not nice and less readable than the previous ones **with transparent background**:
![image](https://github.com/deepset-ai/haystack/assets/44616784/ec5a2bae-6f8a-49b6-b819-d6834c719bed)


**To Reproduce**
Run the [first 2.0 tutorial](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline)


**System:**
 - Haystack version (commit or version number): 2.0-beta8



Discussion:

Resolving patch:
diff --git a/haystack/core/pipeline/draw.py b/haystack/core/pipeline/draw.py
--- a/haystack/core/pipeline/draw.py
+++ b/haystack/core/pipeline/draw.py
@@ -70,7 +70,7 @@ def _to_mermaid_image(graph: networkx.MultiDiGraph):
     graphbytes = graph_styled.encode("ascii")
     base64_bytes = base64.b64encode(graphbytes)
     base64_string = base64_bytes.decode("ascii")
-    url = "https://mermaid.ink/img/" + base64_string
+    url = f"https://mermaid.ink/img/{base64_string}?type=png"
 
     logging.debug("Rendeding graph at %s", url)
     try:


Test patch:
Repository: deepset-ai/haystack

Issue:
Docstrings - `haystack.components.samplers`



Discussion:

Resolving patch:
diff --git a/haystack/components/samplers/top_p.py b/haystack/components/samplers/top_p.py
--- a/haystack/components/samplers/top_p.py
+++ b/haystack/components/samplers/top_p.py
@@ -16,8 +16,8 @@ class TopPSampler:
     """
     Implements top-p (nucleus) sampling for document filtering based on cumulative probability scores.
 
-    This class provides functionality to filter a list of documents by selecting those whose scores fall
-    within the top 'p' percent of the cumulative distribution. The method is useful for focusing on high-probability
+    This component provides functionality to filter a list of documents by selecting those whose scores fall
+    within the top 'p' percent of the cumulative distribution. It is useful for focusing on high-probability
     documents while filtering out less relevant ones based on their assigned scores.
 
     Usage example:
@@ -44,9 +44,9 @@ def __init__(self, top_p: float = 1.0, score_field: Optional[str] = None):
         Creates an instance of TopPSampler.
 
         :param top_p: Float between 0 and 1 representing the cumulative probability threshold for document selection.
-        Defaults to 1.0, indicating no filtering (all documents are retained).
+            A value of 1.0 indicates no filtering (all documents are retained).
         :param score_field: Name of the field in each document's metadata that contains the score. If None, the default
-        document score field is used.
+            document score field is used.
         """
         torch_import.check()
 
@@ -57,17 +57,14 @@ def __init__(self, top_p: float = 1.0, score_field: Optional[str] = None):
     def run(self, documents: List[Document], top_p: Optional[float] = None):
         """
         Filters documents using top-p sampling based on their scores.
+        If the specified top_p results in no documents being selected (especially in cases of a low top_p value), the
+        method returns the document with the highest similarity score.
 
         :param documents: List of Document objects to be filtered.
         :param top_p: Optional. A float to override the cumulative probability threshold set during initialization.
-                 If None, the class's top_p value is used.
-        :return: A dictionary with a key 'documents' containing the list of filtered Document objects.
-
-        This method applies top-p sampling to filter out documents. It selects those documents whose similarity scores
-        are within the top 'p' percent of the cumulative distribution, based on the specified or default top_p value.
 
-        If the specified top_p results in no documents being selected (especially in cases of a low top_p value), the
-        method defaults to returning the document with the highest similarity score.
+        :returns: A dictionary with the following key:
+            - `documents`: List of Document objects that have been selected based on the top-p sampling.
 
         :raises ValueError: If the top_p value is not within the range [0, 1].
         """


Test patch:
Repository: deepset-ai/haystack

Issue:
Rework workflow that updates documentation to delete pages that don't exist anymore
The `readme_sync.yml` workflow doesn't delete pages from Readme if those don't exist anymore. 
When syncing documentation we should if all the hosted pages have local counterparts, and delete all those that don't.

The same should be done for the `haystack-core-integrations` repo.


Discussion:

Resolving patch:
diff --git a/.github/utils/delete_outdated_docs.py b/.github/utils/delete_outdated_docs.py
new file mode 100644
--- /dev/null
+++ b/.github/utils/delete_outdated_docs.py
@@ -0,0 +1,72 @@
+import argparse
+import base64
+import os
+import re
+from pathlib import Path
+from typing import List
+
+import requests
+import yaml
+
+VERSION_VALIDATOR = re.compile(r"^[0-9]+\.[0-9]+$")
+
+
+def readme_token():
+    api_key = os.getenv("RDME_API_KEY", None)
+    if not api_key:
+        raise Exception("RDME_API_KEY env var is not set")
+
+    api_key = f"{api_key}:"
+    return base64.b64encode(api_key.encode("utf-8")).decode("utf-8")
+
+
+def create_headers(version: str):
+    return {"authorization": f"Basic {readme_token()}", "x-readme-version": version}
+
+
+def get_docs_in_category(category_slug: str, version: str) -> List[str]:
+    """
+    Returns the slugs of all documents in a category for the specific version.
+    """
+    url = f"https://dash.readme.com/api/v1/categories/{category_slug}/docs"
+    headers = create_headers(version)
+    res = requests.get(url, headers=headers, timeout=10)
+    return [doc["slug"] for doc in res.json()]
+
+
+def delete_doc(slug: str, version: str):
+    url = f"https://dash.readme.com/api/v1/docs/{slug}"
+    headers = create_headers(version)
+    res = requests.delete(url, headers=headers, timeout=10)
+    res.raise_for_status()
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Delete outdated documentation from Readme.io. "
+        "It will delete all documents that are not present in the current config files."
+    )
+    parser.add_argument(
+        "-c", "--config-path", help="Path to folder containing YAML documentation configs", required=True, type=Path
+    )
+    parser.add_argument("-v", "--version", help="The version that will have its documents deleted", required=True)
+    args = parser.parse_args()
+
+    configs = [yaml.safe_load(c.read_text()) for c in args.config_path.glob("*.yml")]
+
+    remote_docs = {}
+    for config in configs:
+        category_slug = config["renderer"]["category_slug"]
+        if category_slug in remote_docs:
+            continue
+        docs = get_docs_in_category(category_slug, args.version)
+
+        remote_docs[category_slug] = docs
+
+    for config in configs:
+        doc_slug = config["renderer"]["slug"]
+        category_slug = config["renderer"]["category_slug"]
+        if doc_slug in remote_docs[category_slug]:
+            continue
+
+        delete_doc(doc_slug, args.version)


Test patch:
Repository: deepset-ai/haystack

Issue:
API Docs - `haystack.dataclasses`



Discussion:

Resolving patch:
diff --git a/haystack/dataclasses/document.py b/haystack/dataclasses/document.py
--- a/haystack/dataclasses/document.py
+++ b/haystack/dataclasses/document.py
@@ -143,7 +143,7 @@ def from_dict(cls, data: Dict[str, Any]) -> "Document":
         """
         Creates a new Document object from a dictionary.
 
-        NOTE: `dataframe` and `blob` fields are converted to their original types.
+        The `dataframe` and `blob` fields are converted to their original types.
         """
         if (dataframe := data.get("dataframe")) is not None:
             data["dataframe"] = read_json(io.StringIO(dataframe))


Test patch:
Repository: deepset-ai/haystack

Issue:
latest openai embedding support is not working and still assuming the model name is from hugging face
**Describe the bug**
I saw the recent update 956bb6398e8e2f66a710f09a4edc71fc475ca9a5 on v1.25.0-rc1 says it is supporting latest OpenAI embedding model text-embedding-3-large and text-embedding-3-small, But when i use EmbeddingRetriever, it still try to find the huggingface model of text-embedding-3-large, instead of using OpenAI embedding.

**Error message**
```
 text-embedding-3-large is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
```

**Expected behavior**
OpenAI instead of searching Huggingface for the OpenAI embedding model

**Additional context**
Seems like the new model name is not included in _infer_model_format of EmbeddingRetriever.

**To Reproduce**
initialize a retriever like the follow:
```py
retriever = EmbeddingRetriever(
                    document_store=document_store,
                    embedding_model='text-embedding-3-large',
                    model_format=None,
                    use_gpu=False,
                    batch_size=8,
                    api_key=api_key,
                    max_seq_len=8192,
)
```

**FAQ Check**
- [X] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Debian GNU/Linux 10 (buster)
 - GPU/CPU: Tesla T4 /Intel(R) Xeon(R) CPU @ 2.20GHz (GCP n1-standard-4)
 - Haystack version (commit or version number): v1.25.0-rc1
 - DocumentStore: elastic search
 - Reader: N/A
 - Retriever: text-embedding-3-large



Discussion:

Resolving patch:
diff --git a/haystack/nodes/retriever/dense.py b/haystack/nodes/retriever/dense.py
--- a/haystack/nodes/retriever/dense.py
+++ b/haystack/nodes/retriever/dense.py
@@ -1895,9 +1895,14 @@ def _preprocess_documents(self, docs: List[Document]) -> List[Document]:
     @staticmethod
     def _infer_model_format(model_name_or_path: str, use_auth_token: Optional[Union[str, bool]]) -> str:
         # pylint: disable=too-many-return-statements
-        valid_openai_model_name = model_name_or_path in ["ada", "babbage", "davinci", "curie"] or any(
-            m in model_name_or_path for m in ["-ada-", "-babbage-", "-davinci-", "-curie-"]
-        )
+        valid_openai_model_name = model_name_or_path in [
+            "ada",
+            "babbage",
+            "davinci",
+            "curie",
+            "text-embedding-3-small",
+            "text-embedding-3-large",
+        ] or any(m in model_name_or_path for m in ["-ada-", "-babbage-", "-davinci-", "-curie-"])
         if valid_openai_model_name:
             return "openai"
         if model_name_or_path in COHERE_EMBEDDING_MODELS:


Test patch: