Repository: deepset-ai/haystack
Pull request: Add delete_labels() except for weaviate doc store
**Proposed changes**:
- Add delete_labels() method similar to delete_documents() for all document stores except for weaviate document store: labels can now be deleted by `id` (in elasticsearch this field is called `_id`) or by one of the other fields of label, such as `query`
- Small bug fix in InMemoryDocumentStore: delete_documents() did not use the passed index parameter in get_all_documents() but always used default self.index instead

closes #1502 

**Status (please check what you already did)**:
- [x] First draft (up for discussions & feedback)
- [x] Final code
- [x] Added tests
- [x] Updated documentation


Problem statement:
Add `delete_labels()` method to all document stores
**Is your feature request related to a problem? Please describe.**
We have a couple of methods to write labels to a document store. However, there's no easy way to delete them from within haystack.

**Describe the solution you'd like**
A method `delete_labels()` that works similarly to `delete_documents` and allows deletion of all labels or a subset based on a filter.

**Additional context**
Weaviate doesn't support yet the storage of labels, so I would exclude it from here as well.

Hints from issue discussion:

Commit message: re-add bugfix after merge

Patch to review:
diff --git a/docs/_src/api/api/document_store.md b/docs/_src/api/api/document_store.md
index 21061c2cc0..383b6da0d7 100644
--- a/docs/_src/api/api/document_store.md
+++ b/docs/_src/api/api/document_store.md
@@ -275,6 +275,7 @@ Write annotation labels into document store.
 **Arguments**:
 
 - `labels`: A list of Python dictionaries or a list of Haystack Label objects.
+- `index`: Elasticsearch index where the labels should be stored. If not supplied, self.label_index will be used.
 - `batch_size`: Number of labels that are passed to Elasticsearch's bulk function at a time.
 
 <a name="elasticsearch.ElasticsearchDocumentStore.update_document_meta"></a>
@@ -463,7 +464,8 @@ Delete documents in an index. All documents are deleted if no filters are passed
 
 **Arguments**:
 
-- `index`: Index name to delete the document from.
+- `index`: Index name to delete the documents from. If None, the
+              DocumentStore's default index (self.index) will be used
 - `ids`: Optional list of IDs to narrow down the documents to be deleted.
 - `filters`: Optional filters to narrow down the documents to be deleted.
     Example filters: {"name": ["some", "more"], "category": ["only_one"]}.
@@ -475,6 +477,26 @@ Delete documents in an index. All documents are deleted if no filters are passed
 
 None
 
+<a name="elasticsearch.ElasticsearchDocumentStore.delete_labels"></a>
+#### delete\_labels
+
+```python
+ | delete_labels(index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None)
+```
+
+Delete labels in an index. All labels are deleted if no filters are passed.
+
+**Arguments**:
+
+- `index`: Index name to delete the labels from. If None, the
+              DocumentStore's default label index (self.label_index) will be used
+- `filters`: Optional filters to narrow down the labels to be deleted.
+    Example filters: {"_id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+
+**Returns**:
+
+None
+
 <a name="elasticsearch.OpenSearchDocumentStore"></a>
 ## OpenSearchDocumentStore Objects
 
@@ -748,7 +770,7 @@ Delete documents in an index. All documents are deleted if no filters are passed
 
 **Arguments**:
 
-- `index`: Index name to delete the document from. If None, the
+- `index`: Index name to delete the documents from. If None, the
               DocumentStore's default index (self.index) will be used.
 - `ids`: Optional list of IDs to narrow down the documents to be deleted.
 - `filters`: Optional filters to narrow down the documents to be deleted.
@@ -761,6 +783,26 @@ Delete documents in an index. All documents are deleted if no filters are passed
 
 None
 
+<a name="memory.InMemoryDocumentStore.delete_labels"></a>
+#### delete\_labels
+
+```python
+ | delete_labels(index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None)
+```
+
+Delete labels in an index. All labels are deleted if no filters are passed.
+
+**Arguments**:
+
+- `index`: Index name to delete the labels from. If None, the
+              DocumentStore's default label index (self.label_index) will be used.
+- `filters`: Optional filters to narrow down the labels to be deleted.
+                Example filters: {"id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+
+**Returns**:
+
+None
+
 <a name="sql"></a>
 # Module sql
 
@@ -982,6 +1024,26 @@ Delete documents in an index. All documents are deleted if no filters are passed
 
 None
 
+<a name="sql.SQLDocumentStore.delete_labels"></a>
+#### delete\_labels
+
+```python
+ | delete_labels(index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None)
+```
+
+Delete labels from the document store. All labels are deleted if no filters are passed.
+
+**Arguments**:
+
+- `index`: Index name to delete the labels from. If None, the
+              DocumentStore's default label index (self.label_index) will be used.
+- `filters`: Optional filters to narrow down the labels to be deleted.
+                Example filters: {"id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+
+**Returns**:
+
+None
+
 <a name="faiss"></a>
 # Module faiss
 
@@ -1172,7 +1234,7 @@ Delete documents from the document store. All documents are deleted if no filter
 
 **Arguments**:
 
-- `index`: Index name to delete the document from. If None, the
+- `index`: Index name to delete the documents from. If None, the
               DocumentStore's default index (self.index) will be used.
 - `ids`: Optional list of IDs to narrow down the documents to be deleted.
 - `filters`: Optional filters to narrow down the documents to be deleted.
diff --git a/docs/_src/api/api/reader.md b/docs/_src/api/api/reader.md
index 3112742910..29ba05b465 100644
--- a/docs/_src/api/api/reader.md
+++ b/docs/_src/api/api/reader.md
@@ -103,7 +103,7 @@ and that FARM includes no_answer in the sorted list of predictions.
 #### train
 
 ```python
- | train(data_dir: str, train_filename: str, dev_filename: Optional[str] = None, test_filename: Optional[str] = None, use_gpu: Optional[bool] = None, batch_size: int = 10, n_epochs: int = 2, learning_rate: float = 1e-5, max_seq_len: Optional[int] = None, warmup_proportion: float = 0.2, dev_split: float = 0, evaluate_every: int = 300, save_dir: Optional[str] = None, num_processes: Optional[int] = None, use_amp: str = None)
+ | train(data_dir: str, train_filename: str, dev_filename: Optional[str] = None, test_filename: Optional[str] = None, use_gpu: Optional[bool] = None, batch_size: int = 10, n_epochs: int = 2, learning_rate: float = 1e-5, max_seq_len: Optional[int] = None, warmup_proportion: float = 0.2, dev_split: float = 0, evaluate_every: int = 300, save_dir: Optional[str] = None, num_processes: Optional[int] = None, use_amp: str = None, checkpoint_root_dir: Path = Path("model_checkpoints"), checkpoint_every: Optional[int] = None, checkpoints_to_keep: int = 3)
 ```
 
 Fine-tune a model on a QA dataset. Options:
@@ -111,6 +111,9 @@ Fine-tune a model on a QA dataset. Options:
 - Take a plain language model (e.g. `bert-base-cased`) and train it for QA (e.g. on SQuAD data)
 - Take a QA model (e.g. `deepset/bert-base-cased-squad2`) and fine-tune it for your domain (e.g. using your labels collected via the haystack annotation tool)
 
+Checkpoints can be stored via setting `checkpoint_every` to a custom number of steps.
+If any checkpoints are stored, a subsequent run of train() will resume training from the latest available checkpoint.
+
 **Arguments**:
 
 - `data_dir`: Path to directory containing your training data in SQuAD style
@@ -140,6 +143,10 @@ Fine-tune a model on a QA dataset. Options:
                 "O2" (Almost FP16)
                 "O3" (Pure FP16).
                 See details on: https://nvidia.github.io/apex/amp.html
+- `checkpoint_root_dir`: the Path of directory where all train checkpoints are saved. For each individual
+       checkpoint, a subdirectory with the name epoch_{epoch_num}_step_{step_num} is created.
+- `checkpoint_every`: save a train checkpoint after this many steps of training.
+- `checkpoints_to_keep`: maximum number of train checkpoints to save.
 
 **Returns**:
 
diff --git a/haystack/document_store/base.py b/haystack/document_store/base.py
index 45be0ae31a..8894cf0120 100644
--- a/haystack/document_store/base.py
+++ b/haystack/document_store/base.py
@@ -248,6 +248,10 @@ def delete_all_documents(self, index: Optional[str] = None, filters: Optional[Di
     def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]] = None, filters: Optional[Dict[str, List[str]]] = None):
         pass
 
+    @abstractmethod
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
+        pass
+
     def run(self, documents: List[dict], index: Optional[str] = None):  # type: ignore
         self.write_documents(documents=documents, index=index)
         return {}, "output_1"
diff --git a/haystack/document_store/elasticsearch.py b/haystack/document_store/elasticsearch.py
index af40587481..bdff0631b0 100644
--- a/haystack/document_store/elasticsearch.py
+++ b/haystack/document_store/elasticsearch.py
@@ -455,6 +455,7 @@ def write_labels(
         """Write annotation labels into document store.
 
         :param labels: A list of Python dictionaries or a list of Haystack Label objects.
+        :param index: Elasticsearch index where the labels should be stored. If not supplied, self.label_index will be used.
         :param batch_size: Number of labels that are passed to Elasticsearch's bulk function at a time.
         """
         index = index or self.label_index
@@ -997,7 +998,8 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         """
         Delete documents in an index. All documents are deleted if no filters are passed.
 
-        :param index: Index name to delete the document from.
+        :param index: Index name to delete the documents from. If None, the
+                      DocumentStore's default index (self.index) will be used
         :param ids: Optional list of IDs to narrow down the documents to be deleted.
         :param filters: Optional filters to narrow down the documents to be deleted.
             Example filters: {"name": ["some", "more"], "category": ["only_one"]}.
@@ -1031,6 +1033,19 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         if self.refresh_type == "wait_for":
             time.sleep(2)
 
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
+        """
+        Delete labels in an index. All labels are deleted if no filters are passed.
+
+        :param index: Index name to delete the labels from. If None, the
+                      DocumentStore's default label index (self.label_index) will be used
+        :param filters: Optional filters to narrow down the labels to be deleted.
+            Example filters: {"_id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+        :return: None
+        """
+        index = index or self.label_index
+        self.delete_documents(index=index, filters=filters)
+
 
 class OpenSearchDocumentStore(ElasticsearchDocumentStore):
     """
diff --git a/haystack/document_store/faiss.py b/haystack/document_store/faiss.py
index c498a080ef..0a01a8910e 100644
--- a/haystack/document_store/faiss.py
+++ b/haystack/document_store/faiss.py
@@ -403,7 +403,7 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         """
         Delete documents from the document store. All documents are deleted if no filters are passed.
 
-        :param index: Index name to delete the document from. If None, the
+        :param index: Index name to delete the documents from. If None, the
                       DocumentStore's default index (self.index) will be used.
         :param ids: Optional list of IDs to narrow down the documents to be deleted.
         :param filters: Optional filters to narrow down the documents to be deleted.
diff --git a/haystack/document_store/memory.py b/haystack/document_store/memory.py
index a08466b672..fb2e7d7869 100644
--- a/haystack/document_store/memory.py
+++ b/haystack/document_store/memory.py
@@ -395,7 +395,7 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         """
         Delete documents in an index. All documents are deleted if no filters are passed.
 
-        :param index: Index name to delete the document from. If None, the
+        :param index: Index name to delete the documents from. If None, the
                       DocumentStore's default index (self.index) will be used.
         :param ids: Optional list of IDs to narrow down the documents to be deleted.
         :param filters: Optional filters to narrow down the documents to be deleted.
@@ -410,8 +410,26 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         if not filters and not ids:
             self.indexes[index] = {}
             return
-        docs_to_delete = self.get_all_documents(filters=filters)
+        docs_to_delete = self.get_all_documents(index=index, filters=filters)
         if ids:
             docs_to_delete = [doc for doc in docs_to_delete if doc.id in ids]
         for doc in docs_to_delete:
             del self.indexes[index][doc.id]
+
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
+        """
+        Delete labels in an index. All labels are deleted if no filters are passed.
+
+        :param index: Index name to delete the labels from. If None, the
+                      DocumentStore's default label index (self.label_index) will be used.
+        :param filters: Optional filters to narrow down the labels to be deleted.
+                        Example filters: {"id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+        :return: None
+        """
+        index = index or self.label_index
+        if not filters:
+            self.indexes[index] = {}
+            return
+        for label in self.get_all_labels(index=index, filters=filters):
+            del self.indexes[index][label.id]
+
diff --git a/haystack/document_store/sql.py b/haystack/document_store/sql.py
index daa531064c..a5c9c53b8e 100644
--- a/haystack/document_store/sql.py
+++ b/haystack/document_store/sql.py
@@ -550,6 +550,32 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
 
         self.session.commit()
 
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
+        """
+        Delete labels from the document store. All labels are deleted if no filters are passed.
+
+        :param index: Index name to delete the labels from. If None, the
+                      DocumentStore's default label index (self.label_index) will be used.
+        :param filters: Optional filters to narrow down the labels to be deleted.
+                        Example filters: {"id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+        :return: None
+        """
+        index = index or self.label_index
+
+        if filters:
+            label_ids_to_delete = self.session.query(LabelORM.id).filter_by(index=index)
+            for key, values in filters.items():
+                label_attribute = getattr(LabelORM, key)
+                label_ids_to_delete = label_ids_to_delete.filter(
+                    label_attribute.in_(values)
+                )
+            self.session.query(LabelORM).filter(LabelORM.id.in_(label_ids_to_delete)).delete(
+                synchronize_session=False)
+        else:
+            self.session.query(LabelORM).filter_by(index=index).delete(synchronize_session=False)
+
+        self.session.commit()
+
     def _get_or_create(self, session, model, **kwargs):
         instance = session.query(model).filter_by(**kwargs).first()
         if instance:
diff --git a/test/test_document_store.py b/test/test_document_store.py
index e29b20518b..e5e9809a97 100644
--- a/test/test_document_store.py
+++ b/test/test_document_store.py
@@ -380,7 +380,7 @@ def test_delete_documents_with_filters(document_store_with_docs):
 
 def test_labels(document_store):
     label = Label(
-        query="question",
+        query="question1",
         answer=Answer(answer="answer",
                       type="extractive",
                       score=0.0,
@@ -405,7 +405,7 @@ def test_labels(document_store):
 
     # write second label + duplicate
     label2 = Label(
-        query="question",
+        query="question2",
         answer=Answer(answer="another answer",
                       type="extractive",
                       score=0.0,
@@ -422,11 +422,41 @@ def test_labels(document_store):
     document_store.write_labels([label, label2], index="haystack_test_label")
     labels = document_store.get_all_labels(index="haystack_test_label")
 
-    # duplicate should not be there
+    # check that second label has been added but not the duplicate
     assert len(labels) == 2
     assert label in labels
     assert label2 in labels
 
+    # delete filtered label2 by id
+    if isinstance(document_store, ElasticsearchDocumentStore):
+        document_store.delete_labels(index="haystack_test_label", filters={"_id": [labels[1].id]})
+    else:
+        document_store.delete_labels(index="haystack_test_label", filters={"id": [labels[1].id]})
+    labels = document_store.get_all_labels(index="haystack_test_label")
+    assert label == labels[0]
+    assert len(labels) == 1
+
+    # re-add label2
+    document_store.write_labels([label2], index="haystack_test_label")
+    labels = document_store.get_all_labels(index="haystack_test_label")
+    assert len(labels) == 2
+
+    # delete filtered label2 by query text
+    document_store.delete_labels(index="haystack_test_label", filters={"query": [labels[1].query]})
+    labels = document_store.get_all_labels(index="haystack_test_label")
+    assert label == labels[0]
+    assert len(labels) == 1
+
+    # re-add label2
+    document_store.write_labels([label2], index="haystack_test_label")
+    labels = document_store.get_all_labels(index="haystack_test_label")
+    assert len(labels) == 2
+
+    # delete all labels
+    document_store.delete_labels(index="haystack_test_label")
+    labels = document_store.get_all_labels(index="haystack_test_label")
+    assert len(labels) == 0
+
 
 def test_multilabel(document_store):
     labels =[


Code review comments:
diff_hunk: @@ -550,6 +550,32 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
 
         self.session.commit()
 
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
+        """
+        Delete labels from the document store. All labels are deleted if no filters are passed.
+
+        :param index: Index name to delete the labels from. If None, the
+                      DocumentStore's default label index (self.label_index) will be used.
+        :param filters: Optional filters to narrow down the labels to be deleted.
+                        Example filters: {"id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
+        :return: None
+        """
+        index = index or self.label_index
+
+        if filters:
+            label_ids_to_delete = self.session.query(LabelORM.id).filter_by(index=index)
path: haystack/document_store/sql.py
text: @user2:
Also this part can probably be aligned with the equivalent for `delete_documents()`. Please sync with @user1

diff_hunk: @@ -1031,6 +1033,19 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         if self.refresh_type == "wait_for":
             time.sleep(2)
 
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
path: haystack/document_store/elasticsearch.py
text: @user2:
@user1 Just worked on enabling the deletion of documents by ids. The signature there looks different as she decided to go for a separate `ids` param in addition to `filters`. I have no strong opinion about either of the two designs but we should make it consistent. Can the two of you please discuss and decide which signature we want to use?

@user2:
https://github.com/deepset-ai/haystack/pull/1606

diff_hunk: @@ -1031,6 +1033,19 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
         if self.refresh_type == "wait_for":
             time.sleep(2)
 
+    def delete_labels(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
+        """
+        Delete labels in an index. All labels are deleted if no filters are passed.
+
+        :param index: Index name to delete the labels from. If None, the
+                      DocumentStore's default label index (self.label_index) will be used
+        :param filters: Optional filters to narrow down the labels to be deleted.
+            Example filters: {"_id": ["9a196e41-f7b5-45b4-bd19-5feb7501c159", "9a196e41-f7b5-45b4-bd19-5feb7501c159"]} or {"query": ["question2"]}
path: haystack/document_store/elasticsearch.py
text: @user1:
I know that the id field in elasticsearch has the `_` prefix but we usually abstract this away from the user (for example when writing new docs by passing a dict the field is just `id`. Let's also make it here consistent across all doc stores