internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
[Docs] run and run batch methods are missing from API



Discussion:
FYI @TuanaCelik @bilgeyucel 
They're the same as the base component's run and run_batch methods. Either point people to base component or copy it over to each node. 

Resolving patch:
diff --git a/docs/pydoc/loaders.py b/docs/pydoc/loaders.py
new file mode 100644
--- /dev/null
+++ b/docs/pydoc/loaders.py
@@ -0,0 +1,63 @@
+import copy
+import typing as t
+
+import docspec
+from pydoc_markdown.interfaces import Context
+from pydoc_markdown.contrib.loaders.python import PythonLoader
+
+
+class CustomPythonLoader(PythonLoader):
+    def load(self) -> t.Iterable[docspec.Module]:
+        """
+        Load the modules, but include inherited methods in the classes.
+        """
+        # Load all haystack modules
+        temp_loader = PythonLoader(search_path=["../../../haystack"])
+        temp_loader.init(Context(directory="."))
+        all_modules = list(temp_loader.load())
+
+        # Collect all classes
+        classes = {}
+        for module in all_modules:
+            for member in module.members:
+                if isinstance(member, docspec.Class):
+                    classes[member.name] = member
+
+        # Load the modules specified in the search path
+        modules = super().load()
+
+        # Add inherited methods to the classes
+        modules = self.include_inherited_methods(modules, classes)
+
+        return modules
+
+    def include_inherited_methods(
+        self, modules: t.Iterable[docspec.Module], classes: t.Dict[str, docspec.Class]
+    ) -> t.Iterable[docspec.Module]:
+        """
+        Recursively include inherited methods from the base classes.
+        """
+        modules = list(modules)
+        for module in modules:
+            for cls in module.members:
+                if isinstance(cls, docspec.Class):
+                    self.include_methods_for_class(cls, classes)
+
+        return modules
+
+    def include_methods_for_class(self, cls: docspec.Class, classes: t.Dict[str, docspec.Class]):
+        """
+        Include all methods inherited from base classes to the class.
+        """
+        if cls.bases is None:
+            return
+        for base in cls.bases:
+            if base in classes:
+                base_cls = classes[base]
+                self.include_methods_for_class(base_cls, classes)
+
+                for member in base_cls.members:
+                    if isinstance(member, docspec.Function) and not any(m.name == member.name for m in cls.members):
+                        new_member = copy.deepcopy(member)
+                        new_member.parent = cls
+                        cls.members.append(new_member)


Test patch:
Repository: deepset-ai/haystack

Issue:
Deprecate `SklearnQueryClassifier`
### Rationale behind deprecation
- `SklearnQueryClassifier` is almost unused
- You can do the same query classification using `TransformersQueryClassifier` or other strategies
- Every new release of `scikit-learn` forces us to retrain our models and change Haystack dependencies
- (in future, we might remove `scikit-learn` from the core dependencies) 


Discussion:

Resolving patch:
diff --git a/haystack/nodes/query_classifier/sklearn.py b/haystack/nodes/query_classifier/sklearn.py
--- a/haystack/nodes/query_classifier/sklearn.py
+++ b/haystack/nodes/query_classifier/sklearn.py
@@ -3,6 +3,7 @@
 from typing import Union, Any, List, Optional, Iterator, Dict
 import pickle
 import urllib
+import warnings
 
 from tqdm import tqdm
 from sklearn.ensemble._gb_losses import BinomialDeviance
@@ -16,6 +17,8 @@
 
 class SklearnQueryClassifier(BaseQueryClassifier):
     """
+    This component is now deprecated and will be removed in future versions. Use `TransformersQueryClassifier` instead of `SklearnQueryClassifier`.
+
     A node to classify an incoming query into one of two categories using a lightweight sklearn model. Depending on the result, the query flows to a different branch in your pipeline
     and the further processing can be customized. You can define this by connecting the further pipeline to either `output_1` or `output_2` from this node.
 
@@ -75,6 +78,11 @@ def __init__(
         :param batch_size: Number of queries to process at a time.
         :param progress_bar: Whether to show a progress bar.
         """
+        warnings.warn(
+            "`SklearnQueryClassifier` component is deprecated and will be removed in future versions. Use `TransformersQueryClassifier` "
+            "instead of `SklearnQueryClassifier`.",
+            category=DeprecationWarning,
+        )
         if ((not isinstance(model_name_or_path, Path)) and (not isinstance(model_name_or_path, str))) or (
             (not isinstance(vectorizer_name_or_path, Path)) and (not isinstance(vectorizer_name_or_path, str))
         ):


Test patch:
diff --git a/test/nodes/test_query_classifier.py b/test/nodes/test_query_classifier.py
--- a/test/nodes/test_query_classifier.py
+++ b/test/nodes/test_query_classifier.py
@@ -1,5 +1,18 @@
 import pytest
-from haystack.nodes.query_classifier.transformers import TransformersQueryClassifier
+from pathlib import Path
+from urllib.error import URLError
+from haystack.nodes.query_classifier import TransformersQueryClassifier, SklearnQueryClassifier
+from ..conftest import fail_at_version
+
+
+@pytest.mark.unit
+@fail_at_version(1, 21)
+def test_sklearnqueryclassifier_deprecation():
+    with pytest.warns(DeprecationWarning):
+        try:
+            SklearnQueryClassifier(Path("fake_model"), Path("fake_vectorizer"))
+        except URLError:
+            pass
 
 
 @pytest.fixture
Repository: deepset-ai/haystack

Issue:
[Docs]: Update WebRetriever docstrings
**Describe the bug**
* Parameters like `search_engine_provider`, `top_search_results` are missing in the docstrings
* Default value for `mode` is not correct





Discussion:

Resolving patch:
diff --git a/haystack/nodes/retriever/web.py b/haystack/nodes/retriever/web.py
--- a/haystack/nodes/retriever/web.py
+++ b/haystack/nodes/retriever/web.py
@@ -67,8 +67,11 @@ def __init__(
         cache_time: int = 1 * 24 * 60 * 60,
     ):
         """
+        :param api_key: API key for the search engine provider.
+        :param search_engine_provider: Name of the search engine provider class, see `providers.py` for a list of supported providers.
+        :param top_search_results: Number of top search results to be retrieved.
         :param top_k: Top k documents to be returned by the retriever.
-        :param mode: Whether to return snippets, raw documents, or preprocessed documents. Preprocessed documents are the default.
+        :param mode: Whether to return snippets, raw documents, or preprocessed documents. Snippets are the default.
         :param preprocessor: Optional PreProcessor to be used to split documents into paragraphs. If not provided, the default PreProcessor is used.
         :param cache_document_store: DocumentStore to be used to cache search results.
         :param cache_index: Index name to be used to cache search results.


Test patch:
Repository: deepset-ai/haystack

Issue:
`TransformersImageToText.generate_captions` accepts "str"
**Describe the bug**
`generate_captions()` func of  `TransformersImageToText` doesn't raise an error when `str` is passed instead of `List[str]` as `image_file_paths` . Instead, `transformers` raises a misleading error which is kind of expected from their end.

**Error message**
```bash
...
  File "/Users/bilgeyucel/opt/miniconda3/lib/python3.9/site-packages/transformers/pipelines/image_to_text.py", line 109, in preprocess
    image = load_image(image)
  File "/Users/bilgeyucel/opt/miniconda3/lib/python3.9/site-packages/transformers/image_utils.py", line 276, in load_image
    raise ValueError(
ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and / is not a valid path
```

**Expected behavior**
A check would be nice from Haystack side. Options:

1. Warn the Haystack user as "image_file_paths expects List[str], not a string"
2. Handle the case where `str` is passed

**To Reproduce**
```python
from haystack.nodes import TransformersImageToText

image_to_text = TransformersImageToText(
  model_name_or_path="nlpconnect/vit-gpt2-image-captioning",
  use_gpu=True,
  batch_size=16,
  progress_bar=True
)

caption = image_to_text.generate_captions(image_file_paths=<FILE_PATH>)
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Mac M2 Air macOS 13.4
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 1.19



Discussion:

Resolving patch:
diff --git a/haystack/nodes/image_to_text/transformers.py b/haystack/nodes/image_to_text/transformers.py
--- a/haystack/nodes/image_to_text/transformers.py
+++ b/haystack/nodes/image_to_text/transformers.py
@@ -158,6 +158,11 @@ def generate_captions(
         if len(image_file_paths) == 0:
             raise ImageToTextError("ImageToText needs at least one file path to produce a caption.")
 
+        if type(image_file_paths) is not list:
+            raise ImageToTextError(
+                "Expected List[str] for image_file_paths, got %s instead" % str(type(image_file_paths))
+            )
+
         images_dataset = ListDataset(image_file_paths)
 
         captions: List[str] = []


Test patch:
Repository: deepset-ai/haystack

Issue:
leading whitespace is missing in the generated text when using `stop_wods`
**Describe the bug**
When running a text-generation task using the gpt2 model, I noticed that my output is missing a whitespace when using `stop_words`. As can be seen below, the generated text without using a stop word is `[" and I'm not a programmer."]` and the generated text using a stop word is `["and I'm not a"]`. Since the first token the model produces is a whitespace and it should stop at the word `programmer`, the correct answer should be `[" and I'm not a"]`.

```python
from transformers import AutoTokenizer
from haystack.nodes.prompt.invocation_layer import HFLocalInvocationLayer
model='gpt2'
tokenizer = AutoTokenizer.from_pretrained(model, add_prefix_space=True)
tokenizer.pad_token = tokenizer.eos_token

stop_words = "programmer"
layer = HFLocalInvocationLayer(model, max_length=7, tokenizer=tokenizer)
result1 = layer.invoke(prompt="Hello, I'm a language model,", do_sample=False)
print(result1)

result2 = layer.invoke(prompt="Hello, I'm a language model,", do_sample=False, stop_words=[stop_words])
print(result2)

```
Output:
```bash 
[" and I'm not a programmer."]
["and I'm not a"]
```

After digging into the code, I notice that this is caused by the `strip()` method at [this line](https://github.com/deepset-ai/haystack/blob/d96c963bc4ab611d327661ee748cb8683ded9f39/haystack/nodes/prompt/invocation_layer/hugging_face.py#L283). This is not correct, because the stop word always appears at the end of the generated text. We should use `rstrip()` instead of `strip()`. 


**To Reproduce**
Use my script above

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/haystack/nodes/prompt/invocation_layer/hugging_face.py b/haystack/nodes/prompt/invocation_layer/hugging_face.py
--- a/haystack/nodes/prompt/invocation_layer/hugging_face.py
+++ b/haystack/nodes/prompt/invocation_layer/hugging_face.py
@@ -280,7 +280,7 @@ def invoke(self, *args, **kwargs):
             # We want to exclude it to be consistent with other invocation layers
             for idx, _ in enumerate(generated_texts):
                 for stop_word in stop_words:
-                    generated_texts[idx] = generated_texts[idx].replace(stop_word, "").strip()
+                    generated_texts[idx] = generated_texts[idx].replace(stop_word, "").rstrip()
         return generated_texts
 
     def _ensure_token_limit(self, prompt: Union[str, List[Dict[str, str]]]) -> Union[str, List[Dict[str, str]]]:


Test patch:
Repository: deepset-ai/haystack

Issue:
Make `DocumentWriter` return the number of documents it wrote
`DocumentWriter` currently returns no output. In today's Office Hours we received the suggestion of making `DocumentWriter` output this information, to help the user make sure the Pipeline actually worked as expected.


Discussion:
@bilgeyucel could you please assign this issue to me

Resolving patch:
diff --git a/haystack/preview/components/writers/document_writer.py b/haystack/preview/components/writers/document_writer.py
--- a/haystack/preview/components/writers/document_writer.py
+++ b/haystack/preview/components/writers/document_writer.py
@@ -44,13 +44,14 @@ def from_dict(cls, data: Dict[str, Any]) -> "DocumentWriter":
         data["init_parameters"]["policy"] = DuplicatePolicy[data["init_parameters"]["policy"]]
         return default_from_dict(cls, data)
 
+    @component.output_types(documents_written=int)
     def run(self, documents: List[Document], policy: Optional[DuplicatePolicy] = None):
         """
         Run DocumentWriter on the given input data.
 
         :param documents: A list of documents to write to the store.
         :param policy: The policy to use when encountering duplicate documents.
-        :return: None
+        :return: Number of documents written
 
         :raises ValueError: If the specified document store is not found.
         """
@@ -58,4 +59,4 @@ def run(self, documents: List[Document], policy: Optional[DuplicatePolicy] = Non
             policy = self.policy
 
         self.document_store.write_documents(documents=documents, policy=policy)
-        return {}
+        return {"documents_written": len(documents)}


Test patch:
Repository: deepset-ai/haystack

Issue:
`AnalyzeResult` Type Error in `AzureOCRDocumentConverter
### Description:

The static method `_convert_azure_result_to_document` in the `AzureOCRDocumentConverter` class has a type hint for `AnalyzeResult` in its method signature. The issue is that `AnalyzeResult` comes from the `azure.ai.formrecognizer` package, which is an optional dependency for our project. This leads to a `NameError` when the package isn't installed.

### Stack Trace:
```
NameError                                 Traceback (most recent call last)
<ipython-input-4-808600d3e9ab> in <cell line: 6>()
      4 from haystack.preview.components.builders.prompt_builder import PromptBuilder
      5 from haystack.preview.components.fetchers import LinkContentFetcher
----> 6 from haystack.preview.components.file_converters import HTMLToDocument
      7 from haystack.preview.components.generators.openai.gpt import GPTGenerator, default_streaming_callback
      8 from haystack.preview.components.routers.file_type_router import FileTypeRouter

2 frames
/usr/local/lib/python3.10/dist-packages/haystack/preview/components/file_converters/azure.py in AzureOCRDocumentConverter()
     93 
     94     @staticmethod
---> 95     def _convert_azure_result_to_document(result: AnalyzeResult, id_hash_keys: List[str], file_suffix: str) -> Document:
     96         """
     97         Convert the result of Azure OCR to a Haystack text Document.

NameError: name 'AnalyzeResult' is not defined
```



Discussion:

Resolving patch:
diff --git a/haystack/preview/components/file_converters/azure.py b/haystack/preview/components/file_converters/azure.py
--- a/haystack/preview/components/file_converters/azure.py
+++ b/haystack/preview/components/file_converters/azure.py
@@ -92,7 +92,9 @@ def from_dict(cls, data: Dict[str, Any]) -> "AzureOCRDocumentConverter":
         return default_from_dict(cls, data)
 
     @staticmethod
-    def _convert_azure_result_to_document(result: AnalyzeResult, id_hash_keys: List[str], file_suffix: str) -> Document:
+    def _convert_azure_result_to_document(
+        result: "AnalyzeResult", id_hash_keys: List[str], file_suffix: str
+    ) -> Document:
         """
         Convert the result of Azure OCR to a Haystack text Document.
         """


Test patch:
Repository: deepset-ai/haystack

Issue:
Create a script for 2.0 API Reference docs



Discussion:

Resolving patch:
diff --git a/docs/pydoc/renderers.py b/docs/pydoc/renderers.py
--- a/docs/pydoc/renderers.py
+++ b/docs/pydoc/renderers.py
@@ -133,3 +133,16 @@ def _frontmatter(self) -> str:
             slug=self.slug,
             order=self.order,
         )
+
+
+@dataclasses.dataclass
+class ReadmePreviewRenderer(ReadmeRenderer):
+    """
+    This custom Renderer behaves just like the ReadmeRenderer but renders docs with the hardcoded version 2.0 to generate correct category ids.
+    """
+
+    def _doc_version(self) -> str:
+        """
+        Returns the hardcoded docs version 2.0.
+        """
+        return "v2.0"


Test patch:
Repository: deepset-ai/haystack

Issue:
2.0 - improve Document representation (`__repr__` and `__str__`)
When doing experiments, printing Documents or getting their representation via `__repr__` is helpful.

**Current representations**
```python
from haystack.preview.dataclasses import Document

doc = Document(content="dfd", meta={"name": "test"}, score=0.9, embedding=[1, 2, 3])

doc
# Document(id='0e2e032e0f638d798e971143e0a4439f606a46e304ed6c50a01501e820940e19', content='dfd',
# dataframe=None, blob=None, meta={'name': 'test'}, score=0.9)

print(doc)
# Document(id=0e2e032e0f638d798e971143e0a4439f606a46e304ed6c50a01501e820940e19, content: 'dfd')
```

As a user, I feel the **representation can be improved**, including:
- metadata
- score
- embedding size



Discussion:

Resolving patch:
diff --git a/haystack/preview/dataclasses/document.py b/haystack/preview/dataclasses/document.py
--- a/haystack/preview/dataclasses/document.py
+++ b/haystack/preview/dataclasses/document.py
@@ -65,9 +65,9 @@ class Document(metaclass=_BackwardCompatible):
     blob: Optional[ByteStream] = field(default=None)
     meta: Dict[str, Any] = field(default_factory=dict)
     score: Optional[float] = field(default=None)
-    embedding: Optional[List[float]] = field(default=None, repr=False)
+    embedding: Optional[List[float]] = field(default=None)
 
-    def __str__(self):
+    def __repr__(self):
         fields = []
         if self.content is not None:
             fields.append(
@@ -77,6 +77,12 @@ def __str__(self):
             fields.append(f"dataframe: {self.dataframe.shape}")
         if self.blob is not None:
             fields.append(f"blob: {len(self.blob.data)} bytes")
+        if len(self.meta) > 0:
+            fields.append(f"meta: {self.meta}")
+        if self.score is not None:
+            fields.append(f"score: {self.score}")
+        if self.embedding is not None:
+            fields.append(f"embedding: vector of size {len(self.embedding)}")
         fields_str = ", ".join(fields)
         return f"{self.__class__.__name__}(id={self.id}, {fields_str})"
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Documents with metadata raises an error in InMemoryDocumentStore
**Describe the bug**
When I have a "language" field in Document meta, InMemoryDocumentStore raises an error in retrieval. Document indexing works without any problem.

**Error message**
```bash
...
PipelineRuntimeError: retriever raised 'TypeError: Document.__init__() got an unexpected keyword argument 'language''
```
More detailed error message is in the colab that I share below

**Expected behavior**
No errors

**To Reproduce**
Here's a colab: https://colab.research.google.com/drive/1ZwCE746GE94KNfJnD7_QPdNWG32bRMHE?usp=sharing

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Colab
 - GPU/CPU: GPU
 - Haystack version (commit or version number): haystack-ai 0.139.0
 - DocumentStore: InMemoryDocumentStore
 - Retriever: InMemoryEmbeddingRetriever



Discussion:

Resolving patch:
diff --git a/haystack/preview/document_stores/in_memory/document_store.py b/haystack/preview/document_stores/in_memory/document_store.py
--- a/haystack/preview/document_stores/in_memory/document_store.py
+++ b/haystack/preview/document_stores/in_memory/document_store.py
@@ -326,7 +326,7 @@ def embedding_retrieval(
             doc_fields["score"] = score
             if return_embedding is False:
                 doc_fields["embedding"] = None
-            top_documents.append(Document(**doc_fields))
+            top_documents.append(Document.from_dict(doc_fields))
 
         return top_documents
 


Test patch:
Repository: deepset-ai/haystack

Issue:
[Installation] Import of GPTGenerator failing without torch being installed
**Describe the bug**
Installed haystack from latest main in a fresh env via
```
pip install -e '.[dev]'
pip install -U -e .[preview]
```
Import of GPTGenerator throws error:
```
>>> from haystack.preview.components.generators.openai import GPTGenerator
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maltepietsch/deepset/dev/haystack/haystack/preview/components/generators/__init__.py", line 1, in <module>
    from haystack.preview.components.generators.hugging_face_local import HuggingFaceLocalGenerator
  File "/Users/maltepietsch/deepset/dev/haystack/haystack/preview/components/generators/hugging_face_local.py", line 27, in <module>
    class StopWordsCriteria(StoppingCriteria):
  File "/Users/maltepietsch/deepset/dev/haystack/haystack/preview/components/generators/hugging_face_local.py", line 43, in StopWordsCriteria
    device: Union[str, torch.device] = "cpu",
                       ^^^^^
NameError: name 'torch' is not defined
```
**Expected behavior**
I can use GPTGenerator without any torch installation 🤔 


**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: MacOSX
 - Haystack version (commit or version number): e905066



Discussion:
This happens because in HF Local Generator, we have 2 different lazy import blocks for torch and transformers and that code is not contained in the torch lazy import.
I'm preparing a fix...

Resolving patch:
diff --git a/haystack/preview/components/generators/hugging_face_local.py b/haystack/preview/components/generators/hugging_face_local.py
--- a/haystack/preview/components/generators/hugging_face_local.py
+++ b/haystack/preview/components/generators/hugging_face_local.py
@@ -9,12 +9,8 @@
 
 SUPPORTED_TASKS = ["text-generation", "text2text-generation"]
 
-with LazyImport(
-    message="PyTorch is needed to run this component. Please install it by following the instructions at https://pytorch.org/"
-) as torch_import:
+with LazyImport(message="Run 'pip install transformers[torch]'") as torch_and_transformers_import:
     import torch
-
-with LazyImport(message="Run 'pip install transformers'") as transformers_import:
     from huggingface_hub import model_info
     from transformers import (
         pipeline,
@@ -127,8 +123,7 @@ def __init__(
             For some chat models, the output includes both the new text and the original prompt.
             In these cases, it's important to make sure your prompt has no stop words.
         """
-        transformers_import.check()
-        torch_import.check()
+        torch_and_transformers_import.check()
 
         pipeline_kwargs = pipeline_kwargs or {}
         generation_kwargs = generation_kwargs or {}


Test patch: