internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
cache NLTK punkt models for docker images
**Is your feature request related to a problem? Please describe.**
Haystack base images do not come with NLTK punkt models which are generally cached later while loading the pipeline. This can be an issue for building images on unusual machines that require proxies to be setup just for downloading these punkt models.

**Describe the solution you'd like**
create a docker utility to cache relevant NLTK models and call it during the docker build process.

**Describe alternatives you've considered**
Copying the models from local.

**Additional context**
cc: @ju-gu @masci 



Discussion:

Resolving patch:
diff --git a/haystack/utils/docker.py b/haystack/utils/docker.py
--- a/haystack/utils/docker.py
+++ b/haystack/utils/docker.py
@@ -3,6 +3,13 @@
 from haystack.nodes._json_schema import load_schema
 
 
+def cache_nltk_model(model: str = "punkt"):
+    logging.info("Caching %s model...", model)
+    import nltk
+
+    nltk.download(model)
+
+
 def cache_models(models: Optional[List[str]] = None, use_auth_token: Optional[Union[str, bool]] = None):
     """
     Small function that caches models and other data.
@@ -19,12 +26,6 @@ def cache_models(models: Optional[List[str]] = None, use_auth_token: Optional[Un
     if models is None:
         models = ["deepset/roberta-base-squad2"]
 
-    # download punkt tokenizer
-    logging.info("Caching punkt data")
-    import nltk
-
-    nltk.download("punkt")
-
     # Cache models
     import transformers
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Remove warning when explicitly setting document ID
Currently when creating a document and explicitly setting its ID, the message "Setting the ID manually. This might cause a mismatch with the ID that would be generated from the document content and id_hash_keys value" is being logged. This is inconvenient, especially when retrieving documents that are already in the document store and already have an ID, which means whenever a document is retrieved, the message above will be printed in the logs. 


Discussion:

Resolving patch:
diff --git a/haystack/schema.py b/haystack/schema.py
--- a/haystack/schema.py
+++ b/haystack/schema.py
@@ -83,10 +83,12 @@ def __init__(
                       If model's `scale_score` was set to True (default) score is in the unit interval (range of [0,1]), where 1 means extremely relevant.
         :param meta: Meta fields for a document like name, url, or author in the form of a custom dict (any keys and values allowed).
         :param embedding: Vector encoding of the text
-        :param id_hash_keys: Generate the document id from a custom list of strings that refere to the documents attributes.
-                             If you want ensure you don't have duplicate documents in your DocumentStore but texts are
-                             not unique, you can modify the metadata and pass e.g. "meta" to this field (e.g. ["content", "meta"]).
-                             In this case the id will be generated by using the content and the defined metadata.
+        :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's attributes.
+                             To ensure you don't have duplicate documents in your DocumentStore if texts are
+                             not unique, modify the metadata and pass, for example, "meta" to this field (example: ["content", "meta"]).
+                             In this case, the id is generated by using the content and the defined metadata.
+                             If you specify a custom ID for the `id` parameter, the `id_hash_keys` parameter is
+                             ignored and the custom ID is used.
         """
 
         if content is None:
@@ -115,10 +117,6 @@ def __init__(
 
         # Create a unique ID (either new one, or one from user input)
         if id is not None:
-            logger.info(
-                "Setting the ID manually. This might cause a mismatch with the ID "
-                "that would be generated from the document content and id_hash_keys value."
-            )
             self.id: str = str(id)
         else:
             self.id: str = self._get_id(id_hash_keys=id_hash_keys)


Test patch:
Repository: deepset-ai/haystack

Issue:
Broken non-cuda training caused by modeling/training/base.py:_save
While trying to train a DensePassageRetriever using only a CPU the training crashed with a cuda call.
For the file [modeling/training/base.py](https://github.com/deepset-ai/haystack/blob/main/haystack/modeling/training/base.py) the availability of torch.cuda is not evaluated before trying to call the function: `torch.cuda.get_rng_state()`.

lineno 508: `"cuda_rng_state": torch.cuda.get_rng_state(),`


Discussion:

Resolving patch:
diff --git a/haystack/modeling/training/base.py b/haystack/modeling/training/base.py
--- a/haystack/modeling/training/base.py
+++ b/haystack/modeling/training/base.py
@@ -505,7 +505,7 @@ def _save(self):
                 "scheduler_state": self.lr_schedule.state_dict(),
                 "numpy_rng_state": numpy.random.get_state(),
                 "rng_state": torch.get_rng_state(),
-                "cuda_rng_state": torch.cuda.get_rng_state(),
+                "cuda_rng_state": torch.cuda.get_rng_state() if torch.cuda.is_available() else None,
             },
             checkpoint_path / "trainer",
             pickle_module=dill,


Test patch:
Repository: deepset-ai/haystack

Issue:
Tiktoken not supported on Windows machines
**Describe the bug**
When using a Windows machine with a supported python version, `tiktoken` will not be used, although it is supported.

**Error message**
```
OpenAI tiktoken module is not available for Python < 3.8,Linux ARM64 and AARCH64. Falling back to GPT2TokenizerFast.
WARNING:haystack.nodes.answer_generator.openai:OpenAI tiktoken module is not available for Python < 3.8,Linux ARM64 and AARCH64. Falling back to GPT2TokenizerFast.
```

**Expected behavior**
Tiktoken should be used. 

**Additional context**
This check [here](https://github.com/deepset-ai/haystack/blob/24405f851c22b98ab61fea7a80f7d059683a4ffd/haystack/nodes/answer_generator/openai.py#L27) seems to fail on Windows. This seems to be related to `platform.machine` returning `AMD64` on Windows, as mentioned [here](https://bugs.python.org/issue7146).

```
>>> import platform
>>> platform.machine()
'AMD64'
>>> platform.machine() in ["amd64", "x86_64"]
False
>>> platform.machine().lower() in ["amd64", "x86_64"]
True
```

**To Reproduce**
Start any haystack project on a Windows machine which has `AMD64` as architecture.

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Windows 11
 - GPU/CPU: AMD Ryzen 7 5800X3D / Nvidia GTX 1070 Ti
 - Haystack version (commit or version number): 1.13.2
 - DocumentStore: None
 - Reader: None
 - Retriever: None



Discussion:
Hey @Xceron!
It seems a good catch...

Would you like to open a PR to fix this issue?
> Hey @Xceron! It seems a good catch...
> 
> Would you like to open a PR to fix this issue?

I've talked to @danielbichuetti on Discord about this as he proposed the original change, and he will create a PR soon. If not, I will create the PR and mention this issue as soon as tomorrow.

Resolving patch:
diff --git a/haystack/nodes/answer_generator/openai.py b/haystack/nodes/answer_generator/openai.py
--- a/haystack/nodes/answer_generator/openai.py
+++ b/haystack/nodes/answer_generator/openai.py
@@ -20,7 +20,7 @@
 
 logger = logging.getLogger(__name__)
 
-machine = platform.machine()
+machine = platform.machine().lower()
 system = platform.system()
 
 USE_TIKTOKEN = False
diff --git a/haystack/nodes/retriever/_openai_encoder.py b/haystack/nodes/retriever/_openai_encoder.py
--- a/haystack/nodes/retriever/_openai_encoder.py
+++ b/haystack/nodes/retriever/_openai_encoder.py
@@ -25,7 +25,7 @@
 
 logger = logging.getLogger(__name__)
 
-machine = platform.machine()
+machine = platform.machine().lower()
 system = platform.system()
 
 USE_TIKTOKEN = False


Test patch:
Repository: deepset-ai/haystack

Issue:
input_converter failed to initialize in Seq2SeqGenerator when bart_lfqa is manually downloaded and model folder path is given as input to class(variable model_name_or_path).
**Describe the bug**
When model like 'vblagoje/bart_lfqa' is manually downloaded using git clone and the folder path is given as an input when initializing Seq2SeqGenerator and predict method is called, the input_converter field is not set appropriately (in this case to _BartEli5Converter) because a dictionary is indexed using model_name_or_path (specifically cls._model_input_converters). This dictionary has only two keys ["yjernite/bart_eli5", "vblagoje/bart_lfqa"].

**Error message**
Exception: Exception while running node 'Generator': "Seq2SeqGenerator doesn't have input converter registered for /home/kshitij/lfqa/bart_lfqa. Provide custom converter for /home/kshitij/lfqa/bart_lfqa in Seq2SeqGenerator initialization"

**Expected behavior**
input_converter is initialized to _BartEli5Converter and predict function works without any error

**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**
[Tutorial Notebook](https://github.com/pinecone-io/examples/blob/master/integrations/haystack/haystack_lfqa.ipynb) for LFQA. Git clone the generator model and replace model_name_or_path with cloned folder path

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Ubuntu 18.04
 - GPU/CPU: Intel i5-11300H
 - Haystack version (commit or version number): latest
 - DocumentStore:FAISSDocumentStore or PineconeDocumentStore
 - Reader:
 - Retriever: EmbeddingRetriever



Discussion:

Resolving patch:
diff --git a/haystack/nodes/answer_generator/transformers.py b/haystack/nodes/answer_generator/transformers.py
--- a/haystack/nodes/answer_generator/transformers.py
+++ b/haystack/nodes/answer_generator/transformers.py
@@ -413,6 +413,11 @@ def _register_converters(cls, model_name_or_path: str, custom_converter: Optiona
 
     @classmethod
     def _get_converter(cls, model_name_or_path: str) -> Optional[Callable]:
+        # using dictionary key based on model_name_or_path endswith
+        if model_name_or_path.endswith("bart_lfqa"):
+            model_name_or_path = "vblagoje/bart_lfqa"
+        elif model_name_or_path.endswith("bart_eli5"):
+            model_name_or_path = "yjernite/bart_eli5"
         return cls._model_input_converters.get(model_name_or_path)
 
     def predict(self, query: str, documents: List[Document], top_k: Optional[int] = None) -> Dict:


Test patch:
Repository: deepset-ai/haystack

Issue:
Postgres metadata column width restriction (100)
**Describe the bug**
The width of the value field in meta_document table is set to 100 [here](https://github.com/deepset-ai/haystack/blob/d69e409cd55bb66071abb33facbc42563dd9fe08/haystack/document_stores/sql.py#L83) whereas for meta_label it's set to 1000 [here](https://github.com/deepset-ai/haystack/blob/d69e409cd55bb66071abb33facbc42563dd9fe08/haystack/document_stores/sql.py#L116)
Tagging @ZanSara as discussed in Discord

**Error message**
value too long for type character varying(100)

**Expected behavior**
I expect to be able to store longer texts in metadata (honestly, I would probably make it a Postgres text type if possible with sqlalchemy)


**To Reproduce**
Attempt to index any metadata value longer than 100 characters using Postgres as storage

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: MacOs
 - GPU/CPU: CPU
 - Haystack version (commit or version number): farm-haystack 1.13.2
 - DocumentStore: Milvus or Faiss




Discussion:
Thanks for submitting this @aaalexlit - Let's see what @ZanSara says about this 👍 
I agree this looks like a useless limitation. Should also be an easy fix, I'll take care of this soon

Resolving patch:
diff --git a/haystack/document_stores/sql.py b/haystack/document_stores/sql.py
--- a/haystack/document_stores/sql.py
+++ b/haystack/document_stores/sql.py
@@ -79,7 +79,7 @@ class MetaDocumentORM(ORMBase):
     __tablename__ = "meta_document"
 
     name = Column(String(100), index=True)
-    value = Column(ArrayType(100), index=True)
+    value = Column(ArrayType(1000), index=True)
     documents = relationship("DocumentORM", back_populates="meta")
 
     document_id = Column(String(100), nullable=False, index=True)


Test patch:
Repository: deepset-ai/haystack

Issue:
REST API: Allow additional parameters in file-upload endpoint
**Is your feature request related to a problem? Please describe.**
Currently the file-upload  endpoint does not allow to pass parameters to the pipeline, beyond the ones used for the File Converter and the Preprocessor. 

https://github.com/deepset-ai/haystack/blob/main/rest_api/rest_api/controller/file_upload.py#L82

I think it could be useful to allow to indicate parameters the same way the query endpoint does, for instance to be able to dynamically specify the name of the Elasticsearch index you want to use 

**Describe the solution you'd like**
Probably define the params object in a similar way the query implementation does would be enough:
```
params = request.params or {}
```




Discussion:

Resolving patch:
diff --git a/rest_api/rest_api/controller/file_upload.py b/rest_api/rest_api/controller/file_upload.py
--- a/rest_api/rest_api/controller/file_upload.py
+++ b/rest_api/rest_api/controller/file_upload.py
@@ -46,6 +46,7 @@ def upload_file(
     files: List[UploadFile] = File(...),
     # JSON serialized string
     meta: Optional[str] = Form("null"),  # type: ignore
+    additional_params: Optional[str] = Form("null"),  # type: ignore
     fileconverter_params: FileConverterParams = Depends(FileConverterParams.as_form),  # type: ignore
     preprocessor_params: PreprocessorParams = Depends(PreprocessorParams.as_form),  # type: ignore
 ):
@@ -75,11 +76,12 @@ def upload_file(
         finally:
             file.file.close()
 
+    params = json.loads(additional_params) or {}  # type: ignore
+
     # Find nodes names
     converters = indexing_pipeline.get_nodes_by_class(BaseConverter)
     preprocessors = indexing_pipeline.get_nodes_by_class(PreProcessor)
 
-    params = {}
     for converter in converters:
         params[converter.name] = fileconverter_params.dict()
     for preprocessor in preprocessors:


Test patch:
Repository: deepset-ai/haystack

Issue:
Change default `save_dir` for `FARMReader.train`
**Is your feature request related to a problem? Please describe.**
[This line](https://github.com/deepset-ai/haystack/blob/632cd1c141a8b485c6ef8695685d2d8eef3ca50f/haystack/nodes/reader/farm.py#L209) sets the parameter `save_dir` to a directory two layers above the current directory. This can be confusing and also lead to a crash in case the current user does not have write permissions to that directory.

**Describe the solution you'd like**
It should be set to a directory on the current level.

**Describe alternatives you've considered**
The error when you are missing the permissions to write could be caught. This wouldn't change the confusing behaviour, but would at least make sure it won't crash because of it.

**Additional context**
Add any other context or screenshots about the feature request here.



Discussion:
@tstadel can I take this up ? I wish to get a hand in. This just requires 

`save_dir = f"../../saved_models/{self.inferencer.model.language_model.name}"`
to be changed to
`save_dir = f"./saved_models/{self.inferencer.model.language_model.name}"`
in [farm.py](https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/reader/farm.py)


Resolving patch:
diff --git a/haystack/nodes/reader/farm.py b/haystack/nodes/reader/farm.py
--- a/haystack/nodes/reader/farm.py
+++ b/haystack/nodes/reader/farm.py
@@ -229,7 +229,7 @@ def _training_procedure(
         devices, n_gpu = initialize_device_settings(devices=devices, use_cuda=use_gpu, multi_gpu=False)  # type: ignore [arg-type]
 
         if not save_dir:
-            save_dir = f"../../saved_models/{self.inferencer.model.language_model.name}"
+            save_dir = f"./saved_models/{self.inferencer.model.language_model.name}"
             if tinybert:
                 save_dir += "_tinybert_stage_1"
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Duplicate tests for WhisperTranscriber
`test_whisper_api_transcribe` and `test_whisper_api_transcribe_with_params` in `test/nodes/test_whisper.py` have the exact same code.
We should remove `test_whisper_api_transcribe_with_params`.

**Alternatives considered**
Change `transcribe_test_helper(w)` to `transcribe_test_helper(w, language="en")` in `test_whisper_api_transcribe_with_params`. This is probably how it was intended. However, we're only testing the API then and don't check the output taking into account that parameter.



Discussion:

Resolving patch:
diff --git a/haystack/nodes/audio/whisper_transcriber.py b/haystack/nodes/audio/whisper_transcriber.py
--- a/haystack/nodes/audio/whisper_transcriber.py
+++ b/haystack/nodes/audio/whisper_transcriber.py
@@ -82,7 +82,8 @@ def transcribe(
 
         :param audio_file: Path to audio file or a binary file-like object.
         :param language: Language of the audio file. If None, the language is automatically detected.
-        :param return_segments: If True, returns the transcription for each segment of the audio file.
+        :param return_segments: If True, returns the transcription for each segment of the audio file. Supported with
+        local installation of whisper only.
         :param translate: If True, translates the transcription to English.
 
         """


Test patch:
diff --git a/test/nodes/test_whisper.py b/test/nodes/test_whisper.py
--- a/test/nodes/test_whisper.py
+++ b/test/nodes/test_whisper.py
@@ -16,14 +16,6 @@ def test_whisper_api_transcribe():
     assert "segments" not in audio_object_transcript and "segments" not in audio_path_transcript
 
 
-@pytest.mark.skipif(os.environ.get("OPENAI_API_KEY", "") == "", reason="OpenAI API key not found")
-@pytest.mark.integration
-def test_whisper_api_transcribe_with_params():
-    w = WhisperTranscriber(api_key=os.environ.get("OPENAI_API_KEY"))
-    audio_object_transcript, audio_path_transcript = transcribe_test_helper(w)
-    assert "segments" not in audio_object_transcript and "segments" not in audio_path_transcript
-
-
 @pytest.mark.skip("Fails on CI cause it fills up memory")
 @pytest.mark.integration
 @pytest.mark.skipif(not is_whisper_available(), reason="Whisper is not installed")
Repository: deepset-ai/haystack

Issue:
Error when delete_documents for new PineconeDocumentStore
**Describe the bug**
After initializing a PineconeDocumentStore, if we run delete_documents an error is raised (as it tries to access a non-existent index)

**Error message**
```
KeyError                                  Traceback (most recent call last)
[<ipython-input-5-825710cfbb3a>](https://localhost:8080/#) in <module>
      1 document_store.delete_all_documents()
      2 
----> 3 document_store.delete_documents()

[/usr/local/lib/python3.7/dist-packages/haystack/document_stores/pinecone.py](https://localhost:8080/#) in delete_documents(self, index, ids, filters, headers, drop_ids, namespace)
    960             # If no filters or IDs we delete everything
    961             self.pinecone_indexes[index].delete(delete_all=True, namespace=namespace)
--> 962             id_values = list(self.all_ids[index])
    963         else:
    964             if ids is None:

KeyError: 'haystack'
```

**Expected behavior**
Nothing, maybe a warning that no index exists. Otherwise this should function

**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**
```python
from haystack.document_stores.pinecone import PineconeDocumentStore

ENV ='us-west1-gcp'
KEY = '<<API_KEY>>'

document_store = PineconeDocumentStore(
    api_key=KEY, environment=ENV,
    index="haystack",
    similarity="dot_product"
)

document_store.delete_all_documents()
document_store.delete_documents()
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Colab
 - Haystack version (commit or version number): From main
 - DocumentStore: Pinecone


Discussion:
I will submit a fix soon
I come across this bug too. Please fix it ASAP.
I think the following check can be also introduced in `pinecone.py's delete_documents` avoiding the occurrence of the error :
```
        if index not in self.all_ids:
            self.all_ids[index] = set()
```
This way, there's no need to warn the user that the local(temp) dictionary does not have the index that the user just passed to PineconeDocumentStore's constructor, which in turn, creates it. (this warning would be confusing due to this from my perspective). Also, all `delete_documents` paths seem to be unaffected by this change.
I might be missing something, any thoughts about this?

Resolving patch:
diff --git a/haystack/document_stores/pinecone.py b/haystack/document_stores/pinecone.py
--- a/haystack/document_stores/pinecone.py
+++ b/haystack/document_stores/pinecone.py
@@ -997,6 +997,8 @@ def delete_documents(
 
         pinecone_syntax_filter = LogicalFilterClause.parse(filters).convert_to_pinecone() if filters else None
 
+        if index not in self.all_ids:
+            self.all_ids[index] = set()
         if ids is None and pinecone_syntax_filter is None:
             # If no filters or IDs we delete everything
             self.pinecone_indexes[index].delete(delete_all=True, namespace=namespace)


Test patch:
Repository: deepset-ai/haystack

Issue:
Docs: PromptNode "debug" param is missing a description
**Describe the bug**
There's no description for the PromptNode debug param.

**Error message**
Error that was thrown (if available)

**Expected behavior**
All params are documented.

**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**
Go to https://docs.haystack.deepset.ai/reference/prompt-node-api#promptnode__init__ and there's no docs for the debug param.

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/haystack/nodes/prompt/prompt_node.py b/haystack/nodes/prompt/prompt_node.py
--- a/haystack/nodes/prompt/prompt_node.py
+++ b/haystack/nodes/prompt/prompt_node.py
@@ -81,6 +81,7 @@ def __init__(
         :param top_k: The number of independently generated texts to return per prompt. For example, if you set top_k=3, the model will generate three answers to the query.
         :param stop_words: Stops text generation if any of the stop words is generated.
         :param model_kwargs: Additional keyword arguments passed when loading the model specified in `model_name_or_path`.
+        :param debug: Whether to include the used prompts as debug information in the output under the key _debug.
 
         Note that Azure OpenAI InstructGPT models require two additional parameters: azure_base_url (the URL for the
         Azure OpenAI API endpoint, usually in the form `https://<your-endpoint>.openai.azure.com') and


Test patch: