internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ietf-tools/datatracker

Issue:
Move "apply patches" check to its own management command
### Description

One of the checks in `checks.py`, run by `ietf/manage.py check`, applies patches from the `patch/` path to the Python libraries. In some cases, such as in the celery beat Docker container, we need to be sure these patches have been applied but do not need to run any of the other checks. Would be nice to break the patching out to a separate management command.

The check command could continue to apply the patches, but should at least fail if the patches haven't been applied.

### Code of Conduct

- [X] I agree to follow the [IETF's Code of Conduct](https://github.com/ietf-tools/.github/blob/main/CODE_OF_CONDUCT.md)


Discussion:

Resolving patch:
diff --git a/ietf/utils/management/commands/patch_libraries.py b/ietf/utils/management/commands/patch_libraries.py
new file mode 100644
--- /dev/null
+++ b/ietf/utils/management/commands/patch_libraries.py
@@ -0,0 +1,32 @@
+# Copyright The IETF Trust 2024, All Rights Reserved
+import django
+import os
+
+from django.conf import settings
+from django.core.management.base import BaseCommand, CommandError
+from pathlib import Path
+
+from ietf.utils import patch
+
+
+class Command(BaseCommand):
+    """Apply IETF patches to libraries"""
+    requires_system_checks = tuple()
+
+    def handle(self, *args, **options):
+        library_path = Path(django.__file__).parent.parent
+        top_dir = Path(settings.BASE_DIR).parent 
+
+        # All patches in settings.CHECKS_LIBRARY_PATCHES_TO_APPLY must have a
+        # relative file path starting from the site-packages dir, e.g.
+        # 'django/db/models/fields/__init__.py'
+        for patch_file in settings.CHECKS_LIBRARY_PATCHES_TO_APPLY:
+            patch_set = patch.fromfile(top_dir / Path(patch_file))
+            if not patch_set:
+                raise CommandError(f"Could not parse patch file '{patch_file}'")
+            if not patch_set.apply(root=bytes(library_path)):
+                raise CommandError(f"Could not apply the patch from '{patch_file}'")
+            if patch_set.already_patched:
+                self.stdout.write(f"Patch from '{patch_file}' was already applied")
+            else:
+                self.stdout.write(f"Applied the patch from '{patch_file}'")


Test patch:
Repository: DataDog/integrations-extras

Issue:
Filebeat instance cache is broken (which breaks increment metrics)
**Output of the [info page](https://docs.datadoghq.com/agent/faq/agent-commands/#agent-status-and-information)**

```text
====================
Collector (v 5.30.1)
====================

  Status date: 2019-01-08 06:09:58 (10s ago)
  Pid: 4468
  Platform: Linux-3.13.0-163-generic-x86_64-with-Ubuntu-14.04-trusty
  Python Version: 2.7.15, 64bit
  Logs: <stderr>, /var/log/datadog/collector.log
<snip>
    filebeat (custom)
    -----------------
      - instance #0 [OK]
      - Collected 1 metric, 0 events & 0 service checks
<snip>
```

**Additional environment details (Operating System, Cloud provider, etc):**
Filebeat version 6.5.1

**Steps to reproduce the issue:**
1. Install and configure filebeat check with:
```yaml
init_config:

instances:
  - registry_file_path: /var/lib/filebeat/registry
    stats_endpoint: http://localhost:5066/stats
```

**Describe the results you received:**
Metrics in `GAUGE_METRIC_NAMES` are reported to Datadog, but none of the metrics in `INCREMENT_METRIC_NAMES` are.

**Describe the results you expected:**
All listed metrics in the above variables and present in `http://localhost:5066/stats` are reported.

**Additional information you deem important (e.g. issue happens only occasionally):**
The issue is here: https://github.com/DataDog/integrations-extras/blob/2eff6e2dd2b123214cd562b93a9b45753e27a959/filebeat/check.py#L211-L218

Line 213 checks for `instance_key` in `self.instance_cache`, but that key is never actually put in that dictionary. The only keys in that dictionary are `'config'` and `'profiler'`, added on lines 217 and 218.

The result of this is that the `FilebeatCheckInstanceConfig` and `FilebeatCheckHttpProfiler` are re-created every check, which means that the `_previous_increment_values` property on FilebeatCheckHttpProfiler is always empty, which means no increment metrics are ever reported.

This appears to have been introduced in #250.


Discussion:

Resolving patch:
diff --git a/filebeat/check.py b/filebeat/check.py
--- a/filebeat/check.py
+++ b/filebeat/check.py
@@ -211,11 +211,12 @@ def __init__(self, *args, **kwargs):
     def check(self, instance):
         instance_key = hash_mutable(instance)
         if instance_key in self.instance_cache:
-            config = self.instance_cache['config']
-            profiler = self.instance_cache['profiler']
+            config = self.instance_cache[instance_key]['config']
+            profiler = self.instance_cache[instance_key]['profiler']
         else:
-            self.instance_cache['config'] = config = FilebeatCheckInstanceConfig(instance)
-            self.instance_cache['profiler'] = profiler = FilebeatCheckHttpProfiler(config)
+            config = FilebeatCheckInstanceConfig(instance)
+            profiler = FilebeatCheckHttpProfiler(config)
+            self.instance_cache[instance_key] = {'config': config, 'profiler': profiler}
 
         self._process_registry(config)
         self._gather_http_profiler_metrics(config, profiler)


Test patch:
Repository: DataDog/integrations-extras

Issue:
 Importing ABC directly from collections will be removed in Python 3.10
Use collections for Python 2 and collections.abc for Python 3

https://github.com/DataDog/integrations-extras/blob/7e676a2d256e142ddf4cb44ac11a6e590a502779/filebeat/datadog_checks/filebeat/filebeat.py#L148


Discussion:

Resolving patch:
diff --git a/filebeat/datadog_checks/filebeat/filebeat.py b/filebeat/datadog_checks/filebeat/filebeat.py
--- a/filebeat/datadog_checks/filebeat/filebeat.py
+++ b/filebeat/datadog_checks/filebeat/filebeat.py
@@ -3,7 +3,6 @@
 # Licensed under Simplified BSD License (see LICENSE)
 
 # stdlib
-import collections
 import errno
 import json
 import numbers
@@ -12,11 +11,17 @@
 import sre_constants
 
 import requests
+import six
 from six import iteritems
 
 from datadog_checks.base import AgentCheck, is_affirmative
 from datadog_checks.base.utils.containers import hash_mutable
 
+if six.PY3:
+    from collections.abc import MutableMapping
+else:
+    from collections import MutableMapping
+
 EVENT_TYPE = SOURCE_TYPE_NAME = "filebeat"
 
 
@@ -145,7 +150,7 @@ def flatten(self, d, parent_key="", sep="."):
         items = []
         for k, v in d.items():
             new_key = parent_key + sep + k if parent_key else k
-            if isinstance(v, collections.MutableMapping):
+            if isinstance(v, MutableMapping):
                 items.extend(self.flatten(v, new_key, sep=sep).items())
             else:
                 items.append((new_key, v))


Test patch:
Repository: DataDog/integrations-extras

Issue:
Collect clock_time_seconds metric from cert-manager
cert-manager v1.5+ exposes a `clock_time` metric which reports the current seconds since the Unix Epoch
See: https://github.com/jetstack/cert-manager/pull/4105

It would be useful to collect this metric in DataDog so we can alert on seconds until a given certificate expires



Discussion:

Resolving patch:
diff --git a/cert_manager/datadog_checks/cert_manager/metrics.py b/cert_manager/datadog_checks/cert_manager/metrics.py
--- a/cert_manager/datadog_checks/cert_manager/metrics.py
+++ b/cert_manager/datadog_checks/cert_manager/metrics.py
@@ -8,6 +8,7 @@
 }
 
 CONTROLLER_METRICS = {
+    'certmanager_clock_time_seconds': 'clock_time',
     'certmanager_controller_sync_call_count': 'controller.sync_call.count',
 }
 


Test patch:
diff --git a/cert_manager/tests/common.py b/cert_manager/tests/common.py
--- a/cert_manager/tests/common.py
+++ b/cert_manager/tests/common.py
@@ -16,6 +16,7 @@
 }
 
 CONTROLLER_METRICS = {
+    'cert_manager.clock_time': aggregator.MONOTONIC_COUNT,
     'cert_manager.controller.sync_call.count': aggregator.MONOTONIC_COUNT,
     'cert_manager.prometheus.health': aggregator.GAUGE,
 }
diff --git a/cert_manager/tests/conftest.py b/cert_manager/tests/conftest.py
--- a/cert_manager/tests/conftest.py
+++ b/cert_manager/tests/conftest.py
@@ -27,7 +27,7 @@ def setup_cert_manager():
             "kubectl",
             "apply",
             "-f",
-            "https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml",
+            "https://github.com/jetstack/cert-manager/releases/download/v1.5.0/cert-manager.yaml",
         ]
     )
     run_command(
diff --git a/cert_manager/tests/fixtures/cert_manager.txt b/cert_manager/tests/fixtures/cert_manager.txt
--- a/cert_manager/tests/fixtures/cert_manager.txt
+++ b/cert_manager/tests/fixtures/cert_manager.txt
@@ -18,6 +18,9 @@ certmanager_certificate_ready_status{condition="Unknown",name="acme-cert",namesp
 certmanager_certificate_ready_status{condition="Unknown",name="acme-cert2",namespace="default"} 0
 certmanager_certificate_ready_status{condition="Unknown",name="myingress-cert",namespace="cert-manager-test"} 0
 certmanager_certificate_ready_status{condition="Unknown",name="selfsigned-cert",namespace="cert-manager-test"} 0
+# HELP certmanager_clock_time_seconds The clock time given in seconds (from 1970/01/01 UTC).
+# TYPE certmanager_clock_time_seconds counter
+certmanager_clock_time_seconds 1.61915483e+09
 # HELP certmanager_controller_sync_call_count The number of sync() calls made by a controller.
 # TYPE certmanager_controller_sync_call_count counter
 certmanager_controller_sync_call_count{controller="CertificateIssuing"} 20
Repository: deepset-ai/haystack

Issue:
Wrong offsets returned by FARMReader
The offsets indicating the start and end of the answer are currently wrong. Seems that it was introduced by cab0932fabbfd3ad0cd497a7504f11f7beb0ee45

```
{
    "question": "Who is the father of Arya?",
    "answers": [
        {
            "answer": "Eddard",
            "score": 17.154027938842773,
            "probability": 0.8951305652110051,
            "context": "ry warrior queen. She travels with her father, Eddard, to King's Landing when he is made Hand of the",
            "offset_start": 47,
            "offset_end": 47,
            "document_id": null
        },
....
```


Discussion:

Resolving patch:
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -85,10 +85,10 @@ def predict(self, question, paragrahps, meta_data_paragraphs=None, top_k=None, m
                 if a["answer"]: #skip "no answer"
                     cur = {"answer": a["answer"],
                            "score": a["score"],
-                           "probability": expit(np.asarray([a["score"]]) / 8), #just a pseudo prob for now
+                           "probability": float(expit(np.asarray([a["score"]]) / 8)), #just a pseudo prob for now
                            "context": a["context"],
                            "offset_start": a["offset_answer_start"] - a["offset_context_start"],
-                           "offset_end": a["offset_answer_start"] - a["offset_context_start"],
+                           "offset_end": a["offset_answer_end"] - a["offset_context_start"],
                            "document_id": a["document_id"]}
                     answers.append(cur)
 


Test patch:
Repository: deepset-ai/haystack

Issue:
Add additional meta fields for FAQ-QA APIs
I couldn't figure out how to get additional meta fields also returned in the results response.
Currently, the results object contains 
```
question, 
answers.answer
answers.question
answer.probability
...

```

How do I get other fields from the same document? I think that should come under "meta" but I am not aware of how to specify that? Is there any environment variable?

_Originally posted by @sonnylaskar in https://github.com/deepset-ai/haystack/issues/101#issuecomment-625989945_


Discussion:

Resolving patch:
diff --git a/haystack/finder.py b/haystack/finder.py
--- a/haystack/finder.py
+++ b/haystack/finder.py
@@ -86,7 +86,7 @@ def get_answers_via_similar_questions(self, question: str, top_k_retriever: int
         for doc in documents:
             #TODO proper calibratation of pseudo probabilities
             cur_answer = {"question": doc.meta["question"], "answer": doc.text, "context": doc.text,
-                          "score": doc.query_score, "offset_start": 0, "offset_end": len(doc.text),
+                          "score": doc.query_score, "offset_start": 0, "offset_end": len(doc.text), "meta": doc.meta
                           }
             if self.retriever.embedding_model:
                 probability = (doc.query_score + 1) / 2


Test patch:
Repository: deepset-ai/haystack

Issue:
Making "question" field configurable via environment variables in FAQ retrieval
Hi,

The current implementation for FAQ style retrieval expects a "`question`" field to be present in the document. However, I think it makes sense to make that field configurable (via env variables).
There can be cases where data has already been indexed and using a separate process one can generate the embeddings and index them to the same elasticsearch document. Currently, the user will be forced to make a copy of the query field as "question" field as a workaround.

Please advice.


Discussion:

Resolving patch:
diff --git a/haystack/database/elasticsearch.py b/haystack/database/elasticsearch.py
--- a/haystack/database/elasticsearch.py
+++ b/haystack/database/elasticsearch.py
@@ -161,7 +161,7 @@ def query_by_embedding(self, query_emb, top_k=10, candidate_doc_ids=None) -> [Do
                     "script_score": {
                         "query": {"match_all": {}},
                         "script": {
-                            "source": "cosineSimilarity(params.query_vector,doc['question_emb']) + 1.0",
+                            "source": f"cosineSimilarity(params.query_vector,doc['{self.embedding_field}']) + 1.0",
                             "params": {
                                 "query_vector": query_emb
                             }


Test patch:
Repository: deepset-ai/haystack

Issue:
Embedding failed with large-nli-stsb model but work for base model
Trying to make use of the pre-trained model in sentence-transformer.  It works for base model (tried bert-base and roberta-base) but failed in large models (for roberta and I think bert as well) with the following errors at time of embedding corpus:

![image](https://user-images.githubusercontent.com/64902553/82432095-1f702e80-9ac2-11ea-86ad-60c9843717dd.png)

Seems like size of the embedding is too large to write as Elasticsearch index?


Discussion:

Resolving patch:
diff --git a/haystack/database/elasticsearch.py b/haystack/database/elasticsearch.py
--- a/haystack/database/elasticsearch.py
+++ b/haystack/database/elasticsearch.py
@@ -89,7 +89,7 @@ def write_documents(self, documents):
             doc["_op_type"] = "create"
             doc["_index"] = self.index
 
-        bulk(self.client, documents)
+        bulk(self.client, documents, request_timeout=30)
 
     def get_document_count(self):
         result = self.client.count()


Test patch:
Repository: deepset-ai/haystack

Issue:
Latest haystack requires python 3.8 upwards?
Typically I followed the installation recommendation and install haystack by:
```
git clone https://github.com/deepset-ai/haystack.git
cd haystack
pip install --editable .
````
This works until maybe a couple of days ago when there are changes to the dependency of 'Literal" from 'typing_extensions'.  From internet search it seems like that is only supported from python v3.8 upwards.  The first hit on that in my case is in /content/haystack/haystack/retriever/elasticsearch.py

I am developing my Haystack code in Google Colab and its python base is still v3.6.9.  So the code failed at the early import library stage.  What is the best way to get around such python dependencies?  


Discussion:

Resolving patch:
diff --git a/haystack/api/controller/search.py b/haystack/api/controller/search.py
--- a/haystack/api/controller/search.py
+++ b/haystack/api/controller/search.py
@@ -45,7 +45,7 @@
     retriever = EmbeddingRetriever(
         document_store=document_store,
         embedding_model=EMBEDDING_MODEL_PATH,
-        model_format=EMBEDDING_MODEL_FORMAT,  # type: ignore
+        model_format=EMBEDDING_MODEL_FORMAT,
         gpu=USE_GPU
     )  # type: BaseRetriever
 else:
diff --git a/haystack/retriever/elasticsearch.py b/haystack/retriever/elasticsearch.py
--- a/haystack/retriever/elasticsearch.py
+++ b/haystack/retriever/elasticsearch.py
@@ -2,7 +2,6 @@
 from typing import List, Union
 
 from farm.infer import Inferencer
-from typing_extensions import Literal
 
 from haystack.database.base import Document
 from haystack.database.elasticsearch import ElasticsearchDocumentStore
@@ -108,8 +107,8 @@ def __init__(
         document_store: ElasticsearchDocumentStore,
         embedding_model: str,
         gpu: bool = True,
-        model_format: Literal["farm", "transformers", "sentence_transformers"] = "farm",
-        pooling_strategy: Literal["cls_token", "reduce_mean", "reduce_max", "per_token", "s3e"] = "reduce_mean",
+        model_format: str = "farm",
+        pooling_strategy: str = "reduce_mean",
         emb_extraction_layer: int = -1,
     ):
         """


Test patch:
Repository: deepset-ai/haystack

Issue:
FARMReader return multiple is_impossible answers to one question
We need to adjust to the new format introduced in FARM 0.4.4


Discussion:
The issue seems to be coming from two place. One is within FARM the other is how haystack handles results coming back from FARM.

We will create separate PRs to fix both and release a newer version of FARM asap.

Resolving patch:
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -255,7 +255,9 @@ def predict(self, question: str, documents: List[Document], top_k: Optional[int]
             no_ans_gaps.append(pred["predictions"][0]["no_ans_gap"])
             for a in pred["predictions"][0]["answers"]:
                 # skip "no answers" here
-                if a["answer"]:
+                if self._check_no_answer(d=a):
+                    pass
+                else:
                     cur = {"answer": a["answer"],
                            "score": a["score"],
                            # just a pseudo prob for now
@@ -405,6 +407,19 @@ def eval(
         }
         return results
 
+
+    @staticmethod
+    def _check_no_answer(d: dict):
+        # check for correct value in "answer"
+        if d["offset_answer_start"] == -1 and d["offset_answer_end"] == -1:
+            assert d["answer"] == "is_impossible"
+
+        # check weather the model thinks there is no answer
+        if d["answer"] == "is_impossible":
+            return True
+        else:
+            return False
+
     @staticmethod
     def _calc_no_answer(no_ans_gaps: List[float], best_score_answer: float):
         # "no answer" scores and positive answers scores are difficult to compare, because


Test patch:
Repository: deepset-ai/haystack

Issue:
Whitespace removal during preprocessing
When splitting documents into passages based on wordcount and respecting sentence boundary we are missing a whitespace between sentences.



Discussion:

Resolving patch:
diff --git a/haystack/preprocessor/preprocessor.py b/haystack/preprocessor/preprocessor.py
--- a/haystack/preprocessor/preprocessor.py
+++ b/haystack/preprocessor/preprocessor.py
@@ -100,8 +100,10 @@ def split(self, document: dict) -> List[dict]:
                     text_splits.append(current_slice)
                     current_slice = ""
                     word_count = 0
+                if len(current_slice) != 0:
+                    sen = " " + sen
                 current_slice += sen
-                word_count += len(sen.split(" "))
+                word_count += current_word_count
             if current_slice:
                 text_splits.append(current_slice)
         else:


Test patch:
diff --git a/test/test_preprocessor.py b/test/test_preprocessor.py
--- a/test/test_preprocessor.py
+++ b/test/test_preprocessor.py
@@ -39,11 +39,13 @@ def test_preprocess_word_split():
     documents = preprocessor.process(document)
     assert len(documents) == 11
 
-    preprocessor = PreProcessor(split_length=10, split_stride=0, split_by="word", split_respect_sentence_boundary=True)
+    preprocessor = PreProcessor(split_length=15, split_stride=0, split_by="word", split_respect_sentence_boundary=True)
     documents = preprocessor.process(document)
-    for doc in documents:
-        assert len(doc["text"].split(" ")) <= 10 or doc["text"].startswith("This is to trick")
-    assert len(documents) == 15
+    for i,doc in enumerate(documents):
+        if i == 0:
+            assert len(doc["text"].split(" ")) == 14
+        assert len(doc["text"].split(" ")) <= 15 or doc["text"].startswith("This is to trick")
+    assert len(documents) == 8
 
 
 @pytest.mark.tika
@@ -70,3 +72,4 @@ def test_clean_header_footer():
 
     assert "This is a header." not in documents[0]["text"]
     assert "footer" not in documents[0]["text"]
+    
\ No newline at end of file
Repository: deepset-ai/haystack

Issue:
Remove TODOs from RAG Generator code
Recently Transformers lib was upgraded in FARM (refer https://github.com/deepset-ai/FARM/pull/649). Hence following ToDos in generator code can be fixed.
https://github.com/deepset-ai/haystack/blob/master/haystack/generator/transformers.py#L255
https://github.com/deepset-ai/haystack/blob/master/haystack/generator/transformers.py#L262

This can be picked along with https://github.com/deepset-ai/haystack/issues/710


Discussion:
Awesome, thanks for the pointers @lalitpagaria 
I will look into it in this sprint

Resolving patch:
diff --git a/haystack/generator/transformers.py b/haystack/generator/transformers.py
--- a/haystack/generator/transformers.py
+++ b/haystack/generator/transformers.py
@@ -251,18 +251,8 @@ def predict(self, query: str, documents: List[Document], top_k: Optional[int] =
         doc_scores = torch.bmm(query_embedding.unsqueeze(1),
                                passage_embeddings.unsqueeze(0).transpose(1, 2)).squeeze(1)
 
-        # TODO Need transformers 3.4.0
-        # Refer https://github.com/huggingface/transformers/issues/7874
-        # Pass it as parameter to generate function as follows -
-        # n_docs=len(flat_docs_dict["text"])
-        self.model.config.n_docs = len(flat_docs_dict["text"])
-
         # Get generated ids from generator
         generator_ids = self.model.generate(
-            # TODO: Need transformers 3.4.0
-            # Refer https://github.com/huggingface/transformers/issues/7871
-            # Remove input_ids parameter once upgraded to 3.4.0
-            input_ids=input_ids,
             context_input_ids=context_input_ids,
             context_attention_mask=context_attention_mask,
             doc_scores=doc_scores,
@@ -270,6 +260,7 @@ def predict(self, query: str, documents: List[Document], top_k: Optional[int] =
             num_beams=self.num_beams,
             max_length=self.max_length,
             min_length=self.min_length,
+            n_docs=len(flat_docs_dict["text"])
         )
 
         generated_answers = self.tokenizer.batch_decode(generator_ids, skip_special_tokens=True)


Test patch: