internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
`langdetect` is not optional
@ZanSara I was thinking: is this behavior correct/intended?

1. install haystack from main (`pip install git+https://github.com/deepset-ai/haystack.git`)
2. `import haystack`
3. `ModuleNotFoundError: No module named 'langdetect'`

_Originally posted by @anakin87 in https://github.com/deepset-ai/haystack/discussions/4679#discussioncomment-5633860_


Discussion:

Resolving patch:
diff --git a/haystack/nodes/doc_language_classifier/langdetect.py b/haystack/nodes/doc_language_classifier/langdetect.py
--- a/haystack/nodes/doc_language_classifier/langdetect.py
+++ b/haystack/nodes/doc_language_classifier/langdetect.py
@@ -10,10 +10,10 @@
 
 try:
     import langdetect
-except ImportError as exc:
+except (ImportError, ModuleNotFoundError) as exc:
     logger.debug(
         "langdetect could not be imported. "
-        "Run 'pip install farm-haystack[file-conversion]' or 'pip install langdetect' to fix this issue."
+        "Run 'pip install farm-haystack[preprocessing]' or 'pip install langdetect' to fix this issue."
     )
     langdetect = None
 
diff --git a/haystack/nodes/file_converter/base.py b/haystack/nodes/file_converter/base.py
--- a/haystack/nodes/file_converter/base.py
+++ b/haystack/nodes/file_converter/base.py
@@ -1,12 +1,24 @@
 from typing import List, Optional, Dict, Union, Any
 
+import logging
 from abc import abstractmethod
 from pathlib import Path
-import langdetect
+
 from tqdm.auto import tqdm
 from haystack.nodes.base import BaseComponent
 from haystack.schema import Document
 
+logger = logging.getLogger(__name__)
+
+try:
+    import langdetect
+except (ImportError, ModuleNotFoundError) as exc:
+    logger.debug(
+        "langdetect could not be imported. "
+        "Run 'pip install farm-haystack[preprocessing]' or 'pip install langdetect' to fix this issue."
+    )
+    langdetect = None
+
 
 # https://en.wikipedia.org/wiki/Ligature_(writing)
 KNOWN_LIGATURES = {
@@ -125,10 +137,14 @@ def validate_language(self, text: str, valid_languages: Optional[List[str]] = No
         if not valid_languages:
             return True
 
-        try:
-            lang = langdetect.detect(text)
-        except langdetect.lang_detect_exception.LangDetectException:
-            lang = None
+        lang = None
+        if not langdetect:
+            logger.debug("langdetect could not be imported. Haystack won't try to guess the document language.")
+        else:
+            try:
+                lang = langdetect.detect(text)
+            except langdetect.lang_detect_exception.LangDetectException:
+                pass
 
         return lang in valid_languages
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Weaviate BM25 query builder bug
**Describe the bug**
The new Weaviate client (v3.15.6) has brought out a Haystack bug, see https://github.com/weaviate/weaviate-python-client/pull/290#issuecomment-1513744295

Basically we are calling the `.with_bm25()` with a dict, instead of actual params. 

**Error message**
```
  File "/home/user/.local/share/virtualenvs/question-answering-zG86D2Ya/src/farm-haystack/haystack/document_stores/weaviate.py", line 1018, in query
    .build()

  File "/home/user/.local/share/virtualenvs/question-answering-zG86D2Ya/lib/python3.10/site-packages/weaviate/gql/get.py", line 1040, in build
    query += str(self._bm25)
    │            │    └ BM25(query={'query': "What is Zoltan Fedor's phone number?", 'properties': 'content'}, properties=None)
    │            └ <weaviate.gql.get.GetBuilder object at 0x7fd618c815d0>
    └ '{Get{WorkerDir20230418194901(limit: 2 '
  File "/home/user/.local/share/virtualenvs/question-answering-zG86D2Ya/lib/python3.10/site-packages/weaviate/gql/get.py", line 32, in __str__
    ret = f'query: "{util.strip_newlines(self.query)}"'
  File "/home/user/.local/share/virtualenvs/question-answering-zG86D2Ya/lib/python3.10/site-packages/weaviate/util.py", line 559, in strip_newlines
    return s.replace("\n", " ")
           └ {'query': "What is Zoltan Fedor's phone number?", 'properties': 'content'}
```

**Expected behavior**
We should call the `.with_bm25()` of the Weaviate client with the right params.

**Additional context**
I will make a quick PR for this.

**To Reproduce**
Upgrade to the latest Weaviate client v3.15.6 and run any of the test using BM25 with the Weaviate doc store.

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Ubuntu
 - GPU/CPU: not relevant
 - Haystack version (commit or version number): latest
 - DocumentStore: Weaviate
 - Reader: not relevant
 - Retriever: not relevant



Discussion:

Resolving patch:
diff --git a/haystack/document_stores/weaviate.py b/haystack/document_stores/weaviate.py
--- a/haystack/document_stores/weaviate.py
+++ b/haystack/document_stores/weaviate.py
@@ -1028,7 +1028,7 @@ def query(
                 gql_query = (
                     gql.get.GetBuilder(class_name=index, properties=properties, connection=self.weaviate_client)
                     .with_limit(top_k)
-                    .with_bm25({"query": query, "properties": self.content_field})
+                    .with_bm25(query=query, properties=[self.content_field])
                     .with_where(filter_dict)
                     .build()
                 )
@@ -1037,7 +1037,7 @@ def query(
                 gql_query = (
                     gql.get.GetBuilder(class_name=index, properties=properties, connection=self.weaviate_client)
                     .with_limit(top_k)
-                    .with_bm25({"query": query, "properties": self.content_field})
+                    .with_bm25(query=query, properties=[self.content_field])
                     .build()
                 )
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Azure's gpt-35-turbo gets wrong max_token_limit
**Describe the bug**
Due to Tiktoken's lack of support for Azure's `gpt-35-turbo`, the open_ai_utils.py script returns the wrong value for the max token length.

**Error message**
Returned token length is default of 2049

**Expected behavior**
Token length should be 4096



Discussion:

Resolving patch:
diff --git a/haystack/utils/openai_utils.py b/haystack/utils/openai_utils.py
--- a/haystack/utils/openai_utils.py
+++ b/haystack/utils/openai_utils.py
@@ -99,6 +99,10 @@ def _openai_text_completion_tokenization_details(model_name: str):
     max_tokens_limit = 2049  # Based on this ref: https://platform.openai.com/docs/models/gpt-3
     model_tokenizer = MODEL_TO_ENCODING.get(model_name) if USE_TIKTOKEN else None
 
+    # covering the lack of support in Tiktoken. https://github.com/openai/tiktoken/pull/72
+    if model_name == "gpt-35-turbo" and USE_TIKTOKEN:
+        model_tokenizer = "cl100k_base"
+
     if model_tokenizer:
         # Based on OpenAI models page, 'davinci' considers have 2049 tokens,
         ## therefore, it is better to add `text-davinci` instead to the condition.


Test patch:
Repository: deepset-ai/haystack

Issue:
Shaper raises exception when retriever get 0 documents from document store.
**Describe the bug**
If implementing a pipeline as following(retriever->ranker->shaper->promptNode), sometimes the retriever retrieves 0 documents from document store. The shaper raises a exeception.  Actually LLM can answer the questions without the context documents.

**Error message**

Traceback (most recent call last):
  File "/home/user/haystack/haystack/nodes/other/shaper.py", line 777, in run
    output_values = self.function(**input_values)
TypeError: join_documents() missing 1 required positional argument: 'documents'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/haystack/haystack/pipelines/base.py", line 552, in run
    node_output, stream_id = self._run_node(node_id, node_input)
  File "/home/user/haystack/haystack/pipelines/base.py", line 467, in _run_node
    return self.graph.nodes[node_id]["component"]._dispatch_run(**node_input)
  File "/home/user/haystack/haystack/nodes/base.py", line 201, in _dispatch_run
    return self._dispatch_run_general(self.run, **kwargs)
  File "/home/user/haystack/haystack/nodes/base.py", line 245, in _dispatch_run_general
    output, stream = run_method(**run_inputs, **run_params)
  File "/home/user/haystack/haystack/nodes/other/shaper.py", line 781, in run
    raise ValueError(
ValueError: Shaper couldn't apply the function to your inputs and parameters. Check the above stacktrace and make sure you provided all the correct inputs, parameters, and parameter types.

The above exception was the direct cause of the following exception:



**Expected behavior**
no exception.

**Additional context**
None

**To Reproduce**
None

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/haystack/nodes/other/shaper.py b/haystack/nodes/other/shaper.py
--- a/haystack/nodes/other/shaper.py
+++ b/haystack/nodes/other/shaper.py
@@ -740,7 +740,7 @@ def run(  # type: ignore
         if labels and "labels" not in invocation_context.keys():
             invocation_context["labels"] = labels
 
-        if documents and "documents" not in invocation_context.keys():
+        if documents != None and "documents" not in invocation_context.keys():
             invocation_context["documents"] = documents
 
         if meta and "meta" not in invocation_context.keys():


Test patch:
Repository: deepset-ai/haystack

Issue:
Wrong extras recommended when importing MarkdownConverter
**Describe the bug**
When I try to import MarkdownConverter, I get an error about having to install `farm-haystack[preprocessing]`. However, the necessary dependencies are actually in `farm-haystack[file-conversion]`

**Error message**
```
ImportError: Failed to import 'MarkdownConverter', which is an optional component in Haystack.
Run 'pip install 'farm-haystack[preprocessing]'' to install the required dependencies and make this component available

```
**Expected behavior**
Output should be

`Run 'pip install 'farm-haystack[file-conversion]'' to install the required dependencies and make this component available`

**Additional context**

**To Reproduce**
1. Remove required dependencies (`pip uninstall python-frontmatter`)
2. Import `haystack.nodes.MarkdownConverter`


**FAQ Check**
- [ X ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS:  Linux
 - GPU/CPU: n/a
 - Haystack version (commit or version number): 1.16.0rc0
 - DocumentStore: n/a
 - Reader: n/a
 - Retriever: n/a



Discussion:

Resolving patch:
diff --git a/haystack/nodes/file_converter/markdown.py b/haystack/nodes/file_converter/markdown.py
--- a/haystack/nodes/file_converter/markdown.py
+++ b/haystack/nodes/file_converter/markdown.py
@@ -10,7 +10,7 @@
 except (ImportError, ModuleNotFoundError) as ie:
     from haystack.utils.import_utils import _optional_component_not_installed
 
-    _optional_component_not_installed(__name__, "preprocessing", ie)
+    _optional_component_not_installed(__name__, "file-conversion", ie)
 
 from haystack.nodes.file_converter.base import BaseConverter
 from haystack.schema import Document


Test patch:
Repository: deepset-ai/haystack

Issue:
bug: gpt-3.5-turbo-0301 model_tokenizer is not defined for answer generation
**Describe the bug**
When setting another model_name as "gpt-2" or "3.5 turbo". This can be the case, e.g. for 'gpt-3.5-turbo-0301'
https://github.com/deepset-ai/haystack/blob/5022abb546c8065915f95b756a9b06f4c9e2d320/haystack/utils/openai_utils.py#L60

**Error message**

Error that was thrown (if available)

**Expected behavior**
Its not funto encode any model name, also in the future, similar to `gpt-3.5-turbo-0301`
Either a verbose error is raised or, in my opinion gpt-2 is used as fallback.

**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**
Steps to reproduce the behavior

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/haystack/utils/openai_utils.py b/haystack/utils/openai_utils.py
--- a/haystack/utils/openai_utils.py
+++ b/haystack/utils/openai_utils.py
@@ -64,6 +64,7 @@ def _openai_text_completion_tokenization_details(model_name: str):
     """
     tokenizer_name = "gpt2"
     max_tokens_limit = 2049  # Based on this ref: https://platform.openai.com/docs/models/gpt-3
+    model_tokenizer = None
 
     if model_name == "gpt-35-turbo":
         # covering the lack of support in Tiktoken. https://github.com/openai/tiktoken/pull/72


Test patch:
diff --git a/test/utils/test_openai_utils.py b/test/utils/test_openai_utils.py
--- a/test/utils/test_openai_utils.py
+++ b/test/utils/test_openai_utils.py
@@ -10,8 +10,8 @@
 
 @pytest.mark.unit
 def test_openai_text_completion_tokenization_details_gpt_default():
-    tokenizer_name, max_tokens_limit = _openai_text_completion_tokenization_details(model_name="text-ada-001")
-    assert tokenizer_name == "r50k_base"
+    tokenizer_name, max_tokens_limit = _openai_text_completion_tokenization_details(model_name="not-recognized-name")
+    assert tokenizer_name == "gpt2"
     assert max_tokens_limit == 2049
Repository: deepset-ai/haystack

Issue:
Wrong info message about clipped samples with `FARMReader.train`
**Describe the bug**
This came up in this [Discord message](https://discord.com/channels/993534733298450452/993539071815200889/1109450961098903592).

When `FARMReader.train` is called with using a dataset where most of the Documents exceed `max_seq_len`, we print the following info message:
```
INFO:haystack.modeling.data_handler.data_silo:[Haystack Tip] xy% of your samples got cut down to 256 tokens. 
Consider increasing max_seq_len (the maximum value allowed with the current model is max_seq_len=512, 
if this is not enough consider splitting the document in smaller units or changing the model). 
This will lead to higher memory consumption but is likely to improve your model performance
```
This might be misleading, as the samples are not cut down but split into chunks. if a chunk doesn't contain the answer to the question, it is treated as a no-answer sample.

**Expected behavior**
I think we shouldn't print this message when `DataSilo` is used with `SquadProcessor`.




Discussion:

Resolving patch:
diff --git a/haystack/modeling/data_handler/data_silo.py b/haystack/modeling/data_handler/data_silo.py
--- a/haystack/modeling/data_handler/data_silo.py
+++ b/haystack/modeling/data_handler/data_silo.py
@@ -14,7 +14,7 @@
 from torch.utils.data.sampler import RandomSampler, SequentialSampler
 
 from haystack.modeling.data_handler.dataloader import NamedDataLoader
-from haystack.modeling.data_handler.processor import Processor
+from haystack.modeling.data_handler.processor import Processor, SquadProcessor
 from haystack.utils.experiment_tracking import Tracker as tracker
 from haystack.modeling.visual import TRACTOR_SMALL
 
@@ -412,7 +412,8 @@ def _calculate_statistics(self):
         logger.info("Total examples   : %s", self.counts["train"] + self.counts["dev"] + self.counts["test"])
         logger.info("")
         if self.data["train"]:
-            if "input_ids" in self.tensor_names:
+            # SquadProcessor does not clip sequences, but splits them into multiple samples
+            if "input_ids" in self.tensor_names and not isinstance(self.processor, SquadProcessor):
                 logger.info("Longest sequence length observed after clipping:     %s", max(seq_lens))
                 logger.info("Average sequence length after clipping: %s", ave_len)
                 logger.info("Proportion clipped:      %s", clipped)


Test patch:
Repository: deepset-ai/haystack

Issue:
[Docs] Update Haystack CLI documentation
We need to update the documentation of [Haystack CLI](https://github.com/deepset-ai/haystack/tree/main/haystack/cli) that will be ready with https://github.com/deepset-ai/haystack/pull/5055
* We can add more explanations to commands
* Update the readme and printed `--help` message
* Add `version` command to the list



Discussion:

Resolving patch:
diff --git a/haystack/cli/prompt/fetch.py b/haystack/cli/prompt/fetch.py
--- a/haystack/cli/prompt/fetch.py
+++ b/haystack/cli/prompt/fetch.py
@@ -6,18 +6,21 @@
 @click.command(
     short_help="Downloads and saves prompts from Haystack PromptHub",
     help="""
-    Downloads a prompt from the official Haystack PromptHub and saves it locally
-    to ease use in environments with no network.
+    Downloads a prompt from the official [Haystack PromptHub](https://prompthub.deepset.ai/)
+    and saves it locally for easier use in environments with no network.
 
-    PROMPT_NAME can be specified multiple times.
+    You can specify multiple prompts to fetch at the same time.
 
     PROMPTHUB_CACHE_PATH environment variable can be set to change the default
-    folder in which the prompts will be saved in.
+    folder in which the prompts will be saved in. You can find the default cache path on your machine by running the following code:
 
-    If a custom PROMPTHUB_CACHE_PATH is used, remember to also use it for
-    Haystack invocations.
+    ```
+    from haystack.nodes.prompt.prompt_template import PROMPTHUB_CACHE_PATH
+    print(PROMPTHUB_CACHE_PATH)
+    ```
 
-    The Haystack PromptHub is https://prompthub.deepset.ai/
+    If you set a custom PROMPTHUB_CACHE_PATH environment variable, remember to
+    set it to the same value in your console before running Haystack.
     """,
 )
 @click.argument("prompt_name", nargs=-1)


Test patch:
Repository: deepset-ai/haystack

Issue:
Add support for gpt-3.5-turbo-16k and other new OpenAI models.
**Is your feature request related to a problem? Please describe.**
New OpenAI models are not yet supported in the prompt node. 

**Describe the solution you'd like**
Add support for gpt-3.5-turbo-16k and others. 

**Describe alternatives you've considered**
None.

**Additional context**
Some new OpenAI models came out.
https://openai.com/blog/function-calling-and-other-api-updates


Discussion:

Resolving patch:
diff --git a/haystack/nodes/prompt/invocation_layer/chatgpt.py b/haystack/nodes/prompt/invocation_layer/chatgpt.py
--- a/haystack/nodes/prompt/invocation_layer/chatgpt.py
+++ b/haystack/nodes/prompt/invocation_layer/chatgpt.py
@@ -144,4 +144,4 @@ def url(self) -> str:
 
     @classmethod
     def supports(cls, model_name_or_path: str, **kwargs) -> bool:
-        return model_name_or_path in ["gpt-3.5-turbo", "gpt-4", "gpt-4-32k"]
+        return any(m for m in ["gpt-3.5-turbo", "gpt-4"] if m in model_name_or_path)


Test patch:
diff --git a/test/prompt/invocation_layer/test_chatgpt.py b/test/prompt/invocation_layer/test_chatgpt.py
--- a/test/prompt/invocation_layer/test_chatgpt.py
+++ b/test/prompt/invocation_layer/test_chatgpt.py
@@ -27,3 +27,15 @@ def test_custom_api_base(mock_request):
 
     invocation_layer.invoke(prompt="dummy_prompt")
     assert mock_request.call_args.kwargs["url"] == "https://fake_api_base.com/chat/completions"
+
+
+@pytest.mark.unit
+def test_supports_correct_model_names():
+    for model_name in ["gpt-3.5-turbo", "gpt-4", "gpt-4-32k", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613"]:
+        assert ChatGPTInvocationLayer.supports(model_name)
+
+
+@pytest.mark.unit
+def test_does_not_support_wrong_model_names():
+    for model_name in ["got-3.5-turbo", "wrong_model_name"]:
+        assert not ChatGPTInvocationLayer.supports(model_name)
Repository: deepset-ai/haystack

Issue:
PromptNode does't support 16k tokens.
**Describe the bug**
While gpt-3.5-turbo-16k support was added, PromptNode is still setting a token limit of 4096.

**Error message**
The prompt or the messages are too long (447 tokens). The length of the prompt or messages and the answer (30000 tokens) should be within the max token limit (16384 tokens). Reduce the length of the prompt or messages.

**Expected behavior**
PromptNode recognizes 16k models and adjust the token limit.

**To Reproduce**
Set the model of PromptNode gpt-3.5-turbo-16k and attempt to send a prompt larger than 5k tokens.

**FAQ Check**
- [ x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: MacOS Ventura
 - GPU/CPU: CPU
 - Haystack version (commit or version number): latest



Discussion:

Resolving patch:
diff --git a/haystack/utils/openai_utils.py b/haystack/utils/openai_utils.py
--- a/haystack/utils/openai_utils.py
+++ b/haystack/utils/openai_utils.py
@@ -84,6 +84,9 @@ def _openai_text_completion_tokenization_details(model_name: str):
         if "text-davinci" in model_name:
             max_tokens_limit = 4097
             tokenizer_name = model_tokenizer
+        elif model_name.startswith("gpt-3.5-turbo-16k"):
+            max_tokens_limit = 16384
+            tokenizer_name = model_tokenizer
         elif model_name.startswith("gpt-3"):
             max_tokens_limit = 4096
             tokenizer_name = model_tokenizer


Test patch:
Repository: deepset-ai/haystack

Issue:
Shrink CPU Docker image
**Is your feature request related to a problem? Please describe.**
As part of https://github.com/deepset-ai/haystack/issues/4417 we recently introduced a few changes that might help us shrinking the CPU image, or at least add a new flavor optimized for generative AI use cases. In particular:
- After https://github.com/deepset-ai/haystack/issues/4847 we could derive from the alpine Python image
- After https://github.com/deepset-ai/haystack/issues/4233 we could imagine a lightweight `cpu-remote-inference` image to be used with PromptNode and its siblings

**Describe the solution you'd like**
Shrink the CPU docker image, currently 3Gb for AMD64 architectures (the ARM version is already fairly small thanks to the `torch-cpu` inherent usage)



Discussion:
Note: pay attention to the fact that `beir` depends on `torch` directly, so installing it brings in most of the dependencies what we also install with the `inference` extra. This is to say, remember not to install `beir` into lightweight images. https://deepset-ai.slack.com/archives/C04F0D0FYKV/p1687512413160519?thread_ts=1687510736.424199&cid=C04F0D0FYKV

Resolving patch:
diff --git a/haystack/nodes/retriever/web.py b/haystack/nodes/retriever/web.py
--- a/haystack/nodes/retriever/web.py
+++ b/haystack/nodes/retriever/web.py
@@ -81,6 +81,7 @@ def __init__(
         )
         self.mode = mode
         self.cache_document_store = cache_document_store
+        self.document_store = cache_document_store
         self.cache_index = cache_index
         self.cache_headers = cache_headers
         self.cache_time = cache_time


Test patch: