internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: deepset-ai/haystack

Issue:
Deprecate old pipeline eval nodes: EvalDocuments and EvalAnswers
With the new `pipeline.eval()` method, we now have two ways to evaluate a pipeline: either with the new method or with the 
older EvalDocuments and EvalAnswers nodes. As there is no reason to keep and maintain both ways, let's deprecate EvalDocuments and EvalAnswers. This issue is connected to https://github.com/deepset-ai/haystack/issues/1715

Related larger issue: https://github.com/deepset-ai/haystack/issues/1589



Discussion:

Resolving patch:
diff --git a/haystack/nodes/evaluator/evaluator.py b/haystack/nodes/evaluator/evaluator.py
--- a/haystack/nodes/evaluator/evaluator.py
+++ b/haystack/nodes/evaluator/evaluator.py
@@ -23,6 +23,9 @@ class EvalDocuments(BaseComponent):
     from this Node may differ from that when calling Retriever.eval() since that is a closed domain evaluation. Have
     a look at our evaluation tutorial for more info about open vs closed domain eval (
     https://haystack.deepset.ai/tutorials/evaluation).
+
+    EvalDocuments node is deprecated and will be removed in a future version.
+    Please use pipeline.eval() instead.
     """
     outgoing_edges = 1
 
@@ -33,6 +36,8 @@ def __init__(self, debug: bool=False, open_domain: bool=True, top_k: int=10):
         :param debug: When True, a record of each sample and its evaluation will be stored in EvalDocuments.log
         :param top_k: calculate eval metrics for top k results, e.g., recall@k
         """
+        logger.warning("EvalDocuments node is deprecated and will be removed in a future version. "
+                       "Please use pipeline.eval() instead.")
         self.init_counts()
         self.no_answer_warning = False
         self.debug = debug
@@ -152,6 +157,9 @@ class EvalAnswers(BaseComponent):
     Note that results from this Node may differ from that when calling Reader.eval()
     since that is a closed domain evaluation. Have a look at our evaluation tutorial for more info about
     open vs closed domain eval (https://haystack.deepset.ai/tutorials/evaluation).
+
+    EvalAnswers node is deprecated and will be removed in a future version.
+    Please use pipeline.eval() instead.
     """
 
     outgoing_edges = 1
@@ -178,6 +186,8 @@ def __init__(self,
                           - Large model for German only: "deepset/gbert-large-sts"
         :param debug: When True, a record of each sample and its evaluation will be stored in EvalAnswers.log
         """
+        logger.warning("EvalAnswers node is deprecated and will be removed in a future version. "
+                       "Please use pipeline.eval() instead.")
         self.log: List = []
         self.debug = debug
         self.skip_incorrect_retrieval = skip_incorrect_retrieval


Test patch:
Repository: deepset-ai/haystack

Issue:
The `duplicate_documents` argument for Milvus is overwritten by SQLDocumentStore
The `duplicate_documents` argument is overwritten by SQLDocumentStore. The argument is missing in the `super().__init__(..)` call. Applies to Milvus2 too


MilvusDocumentStore:
https://github.com/deepset-ai/haystack/blob/39573cf0a94b00464cb2b9f063b574fb0c50de2c/haystack/document_stores/milvus.py#L128-L133

SQLDocumentStore
https://github.com/deepset-ai/haystack/blob/39573cf0a94b00464cb2b9f063b574fb0c50de2c/haystack/document_stores/sql.py#L92-L100


Discussion:
Hi @yorickvanzweeden, thanks for raising this issue! The same seems to apply for the `FAISSDocumentStore`.

Do you want to raise a PR to solve this issue? Otherwise, we will work on this in the next weeks.

Resolving patch:
diff --git a/haystack/document_stores/faiss.py b/haystack/document_stores/faiss.py
--- a/haystack/document_stores/faiss.py
+++ b/haystack/document_stores/faiss.py
@@ -142,11 +142,11 @@ def __init__(
         self.embedding_field = embedding_field
 
         self.progress_bar = progress_bar
-        self.duplicate_documents = duplicate_documents
 
         super().__init__(
             url=sql_url,
-            index=index
+            index=index,
+            duplicate_documents=duplicate_documents
         )
 
         self._validate_index_sync()
diff --git a/haystack/document_stores/milvus.py b/haystack/document_stores/milvus.py
--- a/haystack/document_stores/milvus.py
+++ b/haystack/document_stores/milvus.py
@@ -125,11 +125,11 @@ def __init__(
         self.return_embedding = return_embedding
         self.embedding_field = embedding_field
         self.progress_bar = progress_bar
-        self.duplicate_documents = duplicate_documents
 
         super().__init__(
             url=sql_url,
-            index=index
+            index=index,
+            duplicate_documents=duplicate_documents
         )
 
     def __del__(self):
diff --git a/haystack/document_stores/milvus2x.py b/haystack/document_stores/milvus2x.py
--- a/haystack/document_stores/milvus2x.py
+++ b/haystack/document_stores/milvus2x.py
@@ -160,11 +160,11 @@ def __init__(
 
         self.return_embedding = return_embedding
         self.progress_bar = progress_bar
-        self.duplicate_documents = duplicate_documents
 
         super().__init__(
             url=sql_url,
-            index=index
+            index=index,
+            duplicate_documents=duplicate_documents
         )
 
     def _create_collection_and_index_if_not_exist(


Test patch:
Repository: deepset-ai/haystack

Issue:
Add model to demo docker image
When running our "quick start" via docker-compose, the API container directly triggers a download of the roberta reader model. Depending on your internet connection this might take a while. How about we add a docker image with the tag `demo_latest` that includes the roberta model and therefore doesn't trigger a download when starting the container 


Discussion:

Resolving patch:
diff --git a/haystack/utils/docker.py b/haystack/utils/docker.py
new file mode 100644
--- /dev/null
+++ b/haystack/utils/docker.py
@@ -0,0 +1,18 @@
+import logging
+
+def cache_models():
+    """
+    Small function that caches models and other data.
+    Used only in the Dockerfile to include these caches in the images.
+    """
+    # download punkt tokenizer
+    logging.info("Caching punkt data")
+    import nltk
+    nltk.download('punkt', download_dir='/root/nltk_data')
+    
+    # Cache roberta-base-squad2 model
+    logging.info("Caching deepset/roberta-base-squad2")
+    import transformers
+    model_to_cache='deepset/roberta-base-squad2'
+    transformers.AutoTokenizer.from_pretrained(model_to_cache)
+    transformers.AutoModel.from_pretrained(model_to_cache)
\ No newline at end of file


Test patch:
Repository: deepset-ai/haystack

Issue:
Missing embeddings not skipped if filters are used
**Describe the bug**
In the ElasticSearchDocumentStore, the filter that skips missing embeddings is overwritten if other filters are passed to the `query_by_embedding`function. Documents without embeddings are thus never skipped.

**Expected behavior**
We should combine the user-defined filters with the filter that skips missing embeddings.


Discussion:
Facing the same issue, when passing the filters along with the query in DPRretriver the missing embeddings are not being skipped and the client.search is generating exceptions. 

Resolving patch:
diff --git a/haystack/document_stores/elasticsearch.py b/haystack/document_stores/elasticsearch.py
--- a/haystack/document_stores/elasticsearch.py
+++ b/haystack/document_stores/elasticsearch.py
@@ -1212,9 +1212,11 @@ def query_by_embedding(
         # +1 in similarity to avoid negative numbers (for cosine sim)
         body = {"size": top_k, "query": self._get_vector_similarity_query(query_emb, top_k)}
         if filters:
-            body["query"]["script_score"]["query"] = {
-                "bool": {"filter": LogicalFilterClause.parse(filters).convert_to_elasticsearch()}
-            }
+            filter_ = {"bool": {"filter": LogicalFilterClause.parse(filters).convert_to_elasticsearch()}}
+            if body["query"]["script_score"]["query"] == {"match_all": {}}:
+                body["query"]["script_score"]["query"] = filter_
+            else:
+                body["query"]["script_score"]["query"]["bool"]["filter"]["bool"]["must"].append(filter_)
 
         excluded_meta_data: Optional[list] = None
 


Test patch:
diff --git a/test/test_document_store.py b/test/test_document_store.py
--- a/test/test_document_store.py
+++ b/test/test_document_store.py
@@ -1270,6 +1270,23 @@ def test_elasticsearch_delete_index():
     assert not index_exists
 
 
+@pytest.mark.parametrize("document_store", ["elasticsearch"], indirect=True)
+def test_elasticsearch_query_with_filters_and_missing_embeddings(document_store):
+    document_store.write_documents(DOCUMENTS)
+    document_without_embedding = Document(
+        content="Doc without embedding", meta={"name": "name_7", "year": "2021", "month": "04"}
+    )
+    document_store.write_documents([document_without_embedding])
+    filters = {"year": "2021"}
+    document_store.skip_missing_embeddings = False
+    with pytest.raises(RequestError):
+        document_store.query_by_embedding(np.random.rand(768), filters=filters)
+
+    document_store.skip_missing_embeddings = True
+    documents = document_store.query_by_embedding(np.random.rand(768), filters=filters)
+    assert len(documents) == 3
+
+
 @pytest.mark.elasticsearch
 def test_get_document_count_only_documents_without_embedding_arg():
     documents = [
Repository: deepset-ai/haystack

Issue:
Tutorial 11 fails on printing debug info in pipeline result
Executing the cell on "Debugging Pipelines" in [Tutorial 11 (pipelines)](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial11_Pipelines.ipynb) fails with an error message because there is no key called "_debug" in the pipeline result.
```
result = p_classifier.run(query="Who is the father of Arya Stark?", params={"debug": True})
result["_debug"]
```

```
KeyError                                  Traceback (most recent call last)
[<ipython-input-15-104fac63384e>](https://localhost:8080/#) in <module>()
      8 result = p_classifier.run(query="Who is the father of Arya Stark?", params={"debug": True})
      9 
---> 10 result["_debug"]

KeyError: '_debug'
```

Maybe this problem is related to this recent PR? https://github.com/deepset-ai/haystack/pull/2298/files @ZanSara Maybe you could briefly check that please?





Discussion:

Resolving patch:
diff --git a/haystack/pipelines/base.py b/haystack/pipelines/base.py
--- a/haystack/pipelines/base.py
+++ b/haystack/pipelines/base.py
@@ -640,7 +640,12 @@ def run(  # type: ignore
             # Apply debug attributes to the node input params
             # NOTE: global debug attributes will override the value specified
             # in each node's params dictionary.
+            if debug is None and node_input:
+                if node_input.get("params", {}):
+                    debug = params.get("debug", None)  # type: ignore
             if debug is not None:
+                if not node_input.get("params", None):
+                    node_input["params"] = {}
                 if node_id not in node_input["params"].keys():
                     node_input["params"][node_id] = {}
                 node_input["params"][node_id]["debug"] = debug


Test patch:
Repository: deepset-ai/haystack

Issue:
Exception: pdftotext is not installed. It is part of xpdf or poppler-utils software suite.
![image](https://user-images.githubusercontent.com/72295771/165184727-8112d044-a596-43ac-839b-0ec6f4408f47.png)

An issue of installing the `xpdfreader`!


Discussion:
To fix the issue, you need to install the new version

```python
!wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.04.tar.gz
!tar -xvf xpdf-tools-linux-4.04.tar.gz && sudo cp xpdf-tools-linux-4.04/bin64/pdftotext /usr/local/bin
```
Hi @AI-Ahmed thanks for pointing out this problem. We also came across the problem here: https://github.com/deepset-ai/haystack/pull/2443
We forgot to update the text in the Exception though. I will take care of that now and then close this issue. Thanks again! 👍 

Resolving patch:
diff --git a/haystack/nodes/file_converter/pdf.py b/haystack/nodes/file_converter/pdf.py
--- a/haystack/nodes/file_converter/pdf.py
+++ b/haystack/nodes/file_converter/pdf.py
@@ -54,8 +54,8 @@ def __init__(
                 """pdftotext is not installed. It is part of xpdf or poppler-utils software suite.
                 
                    Installation on Linux:
-                   wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.03.tar.gz &&
-                   tar -xvf xpdf-tools-linux-4.03.tar.gz && sudo cp xpdf-tools-linux-4.03/bin64/pdftotext /usr/local/bin
+                   wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.04.tar.gz &&
+                   tar -xvf xpdf-tools-linux-4.04.tar.gz && sudo cp xpdf-tools-linux-4.04/bin64/pdftotext /usr/local/bin
                    
                    Installation on MacOS:
                    brew install xpdf


Test patch:
Repository: deepset-ai/haystack

Issue:
Inconsistent default values for `FARMReader` and `TransformersReader`
`FARMReader`'s default value for `return_no_answer` is `False` and for `top_k_per_candidate` is `3`, while `TransformersReader`'s default value for `return_no_answer` is `True` and for `top_k_per_candidate` is `4`. The default values for both nodes should be the same, as changing between these nodes might result in unwanted behavior otherwise.


Discussion:

Resolving patch:
diff --git a/haystack/nodes/reader/transformers.py b/haystack/nodes/reader/transformers.py
--- a/haystack/nodes/reader/transformers.py
+++ b/haystack/nodes/reader/transformers.py
@@ -26,8 +26,8 @@ def __init__(
         context_window_size: int = 70,
         use_gpu: bool = True,
         top_k: int = 10,
-        top_k_per_candidate: int = 4,
-        return_no_answers: bool = True,
+        top_k_per_candidate: int = 3,
+        return_no_answers: bool = False,
         max_seq_len: int = 256,
         doc_stride: int = 128,
     ):


Test patch:
Repository: deepset-ai/haystack

Issue:
Error for dotproduct metric in PineconeDocumentStore
**Describe the bug**
The PineconeDocumentStore consumes `"dot_product"` as a similarity metric, Pinecone consumes `"dotproduct"` as a metric, creating a mismatch and making dot product impossible to use.

**Error message**
```
ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'text/plain; charset=utf-8', 'content-length': '38', 'date': 'Tue, 03 May 2022 21:25:59 GMT', 'server': 'envoy', 'x-envoy-upstream-service-time': '1'})
HTTP response body: Metric 'dot_product' is not supported.
```

**Expected behavior**
"dot_product" metric should not raise error

**Additional context**
N/A

**To Reproduce**
```python
from haystack.document_stores import PineconeDocumentStore

document_store = PineconeDocumentStore(
    api_key='<YOUR_API_KEY>',
    index='document',
    similarity="dot_product",
    embedding_dim=128
)
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Mac
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 1.3.0
 - DocumentStore: PineconeDocumentStore
 - Reader: N/A
 - Retriever: N/A



Discussion:
I will be submitting a PR to fix soon

Resolving patch:
diff --git a/haystack/document_stores/pinecone.py b/haystack/document_stores/pinecone.py
--- a/haystack/document_stores/pinecone.py
+++ b/haystack/document_stores/pinecone.py
@@ -63,9 +63,9 @@ def __init__(
         :param embedding_dim: The embedding vector size.
         :param return_embedding: Whether to return document embeddings.
         :param index: Name of index in document store to use.
-        :param similarity: The similarity function used to compare document vectors. `"dot_product"` is the default
-            since it is more performant with DPR embeddings. `"cosine"` is recommended if you are using a
-            Sentence-Transformer model.
+        :param similarity: The similarity function used to compare document vectors. `"cosine"` is the default
+            and is recommended if you are using a Sentence-Transformer model. `"dot_product"` is more performant
+            with DPR embeddings.
             In both cases, the returned values in Document.score are normalized to be in range [0,1]:
                 - For `"dot_product"`: `expit(np.asarray(raw_score / 100))`
                 - For `"cosine"`: `(raw_score + 1) / 2`
@@ -91,8 +91,10 @@ def __init__(
         self._api_key = api_key
 
         # Formal similarity string
-        if similarity in ("dot_product", "cosine"):
+        if similarity == "cosine":
             self.metric_type = similarity
+        elif similarity == "dot_product":
+            self.metric_type = "dotproduct"
         elif similarity in ("l2", "euclidean"):
             self.metric_type = "euclidean"
         else:


Test patch:
Repository: deepset-ai/haystack

Issue:
Random question in public demo fails
**Describe the bug**
When clicking on the "Random question" button in the [Haystack public demo](https://haystack-demo.deepset.ai/), the execution fails with the following error:
<img width="721" alt="image" src="https://user-images.githubusercontent.com/4181769/168665062-dec535b0-7b0e-4d86-9cf4-e1ab3c4c5854.png">



Discussion:
I found [this similar issue](https://copyfuture.com/blogs-details/202205050613280281).
It seems related to new Streamlit versions.

> I found myself installing 1.8.1, Now back to 1.7 Version of ：
> pip install streamlit==1.7

Resolving patch:
diff --git a/ui/webapp.py b/ui/webapp.py
--- a/ui/webapp.py
+++ b/ui/webapp.py
@@ -164,7 +164,7 @@ def reset_results(*args):
         st.session_state.random_question_requested = True
         # Re-runs the script setting the random question as the textbox value
         # Unfortunately necessary as the Random Question button is _below_ the textbox
-        raise st.script_runner.RerunException(st.script_request_queue.RerunData(None))
+        raise st.scriptrunner.script_runner.RerunException(st.scriptrunner.script_requests.RerunData(None))
     st.session_state.random_question_requested = False
 
     run_query = (


Test patch:
Repository: deepset-ai/haystack

Issue:
Code snippet wrongly formatted in docs
**Describe the bug**
The code snippet at:

https://github.com/deepset-ai/haystack/blob/master/haystack/nodes/ranker/sentence_transformers.py#L32

is rendered as plain text in the corresponding documentation page

https://haystack.deepset.ai/reference/ranker


**Expected behavior**
The snipped should be formatted as Python code



Discussion:

Resolving patch:
diff --git a/haystack/nodes/ranker/sentence_transformers.py b/haystack/nodes/ranker/sentence_transformers.py
--- a/haystack/nodes/ranker/sentence_transformers.py
+++ b/haystack/nodes/ranker/sentence_transformers.py
@@ -28,12 +28,14 @@ class SentenceTransformersRanker(BaseRanker):
      - directly get predictions via predict()
 
     Usage example:
-    ...
-    retriever = BM25Retriever(document_store=document_store)
-    ranker = SentenceTransformersRanker(model_name_or_path="cross-encoder/ms-marco-MiniLM-L-12-v2")
-    p = Pipeline()
-    p.add_node(component=retriever, name="ESRetriever", inputs=["Query"])
-    p.add_node(component=ranker, name="Ranker", inputs=["ESRetriever"])
+
+    ```python
+    |     retriever = BM25Retriever(document_store=document_store)
+    |     ranker = SentenceTransformersRanker(model_name_or_path="cross-encoder/ms-marco-MiniLM-L-12-v2")
+    |     p = Pipeline()
+    |     p.add_node(component=retriever, name="ESRetriever", inputs=["Query"])
+    |     p.add_node(component=ranker, name="Ranker", inputs=["ESRetriever"])
+    ```
     """
 
     def __init__(


Test patch:
Repository: deepset-ai/haystack

Issue:
Code snippet wrongly formatted in docs
**Describe the bug**
The code snippet at:

https://github.com/deepset-ai/haystack/blob/master/haystack/nodes/ranker/sentence_transformers.py#L32

is rendered as plain text in the corresponding documentation page

https://haystack.deepset.ai/reference/ranker


**Expected behavior**
The snipped should be formatted as Python code



Discussion:

Resolving patch:
diff --git a/haystack/nodes/ranker/sentence_transformers.py b/haystack/nodes/ranker/sentence_transformers.py
--- a/haystack/nodes/ranker/sentence_transformers.py
+++ b/haystack/nodes/ranker/sentence_transformers.py
@@ -30,11 +30,11 @@ class SentenceTransformersRanker(BaseRanker):
     Usage example:
 
     ```python
-    |     retriever = BM25Retriever(document_store=document_store)
-    |     ranker = SentenceTransformersRanker(model_name_or_path="cross-encoder/ms-marco-MiniLM-L-12-v2")
-    |     p = Pipeline()
-    |     p.add_node(component=retriever, name="ESRetriever", inputs=["Query"])
-    |     p.add_node(component=ranker, name="Ranker", inputs=["ESRetriever"])
+         retriever = BM25Retriever(document_store=document_store)
+         ranker = SentenceTransformersRanker(model_name_or_path="cross-encoder/ms-marco-MiniLM-L-12-v2")
+         p = Pipeline()
+         p.add_node(component=retriever, name="ESRetriever", inputs=["Query"])
+         p.add_node(component=ranker, name="Ranker", inputs=["ESRetriever"])
     ```
     """
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Enable isolated_node_eval for `AnswerGenerator` nodes (incl. OpenAI's)
**Is your feature request related to a problem? Please describe.**
When running pipeline.eval(), we have have two options to get metrics from our nodes: isolated and integrated. 
While integrated works already on our generative QA nodes, the isolated mode is currently only working for Readers not Generators. 

Example: 
When running `pipeline.eval()  ` on a pipeline consisting of a `BM25Retriever` -> `OpenAIAnswerGenerator`, we get the error message `No node(s) or global parameter(s) named add_isolated_node_eval found in pipeline.`

**Describe the solution you'd like**
Modify the run() method of the base class to use the perfect labels as input from previous nodes

**Additional context**
This will also be needed to allow experiment runs in deepset Cloud as the additional isolated mode there is the expected default.


Discussion:

Resolving patch:
diff --git a/haystack/nodes/answer_generator/base.py b/haystack/nodes/answer_generator/base.py
--- a/haystack/nodes/answer_generator/base.py
+++ b/haystack/nodes/answer_generator/base.py
@@ -4,7 +4,7 @@
 from tqdm.auto import tqdm
 
 from haystack.errors import HaystackError
-from haystack.schema import Answer, Document
+from haystack.schema import Answer, Document, MultiLabel
 from haystack.nodes.base import BaseComponent
 
 
@@ -31,13 +31,19 @@ def predict(self, query: str, documents: List[Document], top_k: Optional[int]) -
         """
         pass
 
-    def run(self, query: str, documents: List[Document], top_k: Optional[int] = None):  # type: ignore
+    def run(self, query: str, documents: List[Document], top_k: Optional[int] = None, labels: Optional[MultiLabel] = None, add_isolated_node_eval: bool = False):  # type: ignore
 
         if documents:
             results = self.predict(query=query, documents=documents, top_k=top_k)
         else:
             results = {"answers": []}
 
+        # run evaluation with "perfect" labels as node inputs to calculate "upper bound" metrics for just this node
+        if add_isolated_node_eval and labels is not None:
+            relevant_documents = list({label.document.id: label.document for label in labels.labels}.values())
+            results_label_input = self.predict(query=query, documents=relevant_documents, top_k=top_k)
+            results["answers_isolated"] = results_label_input["answers"]
+
         return results, "output_1"
 
     def run_batch(  # type: ignore


Test patch: