Repository: deepset-ai/haystack

Issue:
Preprocessing should allow keeping Document ids unchanged
Currently, preprocessing components such as the DocumentCleaner get documents as input and return preprocessed documents **but with different ids**.
We should enable users to specify custom ids and make sure these ids are used when writing the documents to a database.
We could achieve that in different ways:

1. Ensure that preprocessing components such as the DocumentCleaner keep the document ids unchanged. However that means the id is not based on the preprocessed content anymore but on the original content.
2. Add a new Component that calculates ids of documents right before the DocumentWriter. That means users with custom ids could provide a list of ids as inputs


A current workaround is to define a custom component that includes a method returning custom ids:

```python
from copy import deepcopy
from typing import List
from haystack import Document, component

@component
class DocumentIDCreator:
    # This is a pass-through component that returns the same Documents but with different ids
    @component.output_types(documents=List[Document])
    def run(self, documents: List[Document]):
        updated_documents = []
        for doc in documents:
            id = calculate_id(doc, id_hash_keys) # your code
            updated_documents.append(Document(id=id, content=doc.content, meta=deepcopy(doc.meta), embedding=doc.embedding))

        return {"documents": updated_documents}
```


Discussion:
Thanks for creating an issue on the topic. 

I can imagine, a keep_id flag when calling the run function could be enough. It could be set to False by default. 
The id could be then be reused in the end, when the new document is being created. 
I will take this issue. I believe @Phlasse approach seems to be the way to go, as it keeps things sompler.
Hey @CarlosFerLo @julian-risch @Phlasse 

Just saw your PR contribution @CarlosFerLo - thank you. 

I was wondering why not, if we are already adding another init parameter, make it more powerful and future proof yet as simple as the flag keep_id

We could add id_generator init parameter:

```
id_generator: Optional[Callable[[Document, str], str]] = None
```
where the first parameter of the callable is cleaned document, the second parameter is doc id of the old document and callable returns str.

assigned in init like this:
```
self.id_generator = id_generator or (lambda doc, id: doc.id) # or whatever default is
```

and the end of the run method is:

```
clean_doc = Document(content=text, meta=deepcopy(doc.meta))
clean_doc.id = self.id_generator(clean_doc, doc.id)
cleaned_docs.append(clean_doc)
```

Let me know your thoughts about this approach. 



@vblagoje then what are you suggesting? 

We keep the `keep_id` flag, and we add a `id_generator` where the id generator can overwrite the resulting id of the cleaned document receiving the original document and the new id.

Or we delete the `keep_flag` and add all the functionality in the `id_generator` parameter and we can add a keep_id id generator that has this behaviour. 

And the important part, what is the default id generator?

@CarlosFerLo I'm suggesting we don't use `keep_id` flag because `id_generator` callable can replace it and future use cases where people want to customize doc id generation. We can then use this familiar pattern not only in this component but in others as well - whenever some derived documents are created by a component and we need to generate derived docs ids. 
@vblagoje Okey. I will create a new pull request, to clean the commit tree. Should I create a Class for this kind of function, to make the code more readable and to add more functionality if needed in the future. Should I name the class `DocumentIdGenerator` or just `IdGenerator`. And to specify the input parameters: it receives the new document and the old id? or better if it gets the new document and the old one, it would give more flexibility, but also, haveing the new document alrady formatted seems a little redundant. 
@CarlosFerLo a class in our design principles would be an overkill. A simple [Callable](https://docs.python.org/3/library/typing.html#annotating-callable-objects) suffices here. See in these docs how you can specify the input parameters and return values.

Even better insight, let this Callable have old and new docs as inputs and return id. You can call this parameter id_generator. 
Thanks for working on this
@vblagoje Opened a new pull request that contains the new implementation.

Resolving patch:
diff --git a/haystack/components/preprocessors/document_cleaner.py b/haystack/components/preprocessors/document_cleaner.py
--- a/haystack/components/preprocessors/document_cleaner.py
+++ b/haystack/components/preprocessors/document_cleaner.py
@@ -40,6 +40,7 @@ def __init__(
         remove_empty_lines: bool = True,
         remove_extra_whitespaces: bool = True,
         remove_repeated_substrings: bool = False,
+        keep_id: bool = False,
         remove_substrings: Optional[List[str]] = None,
         remove_regex: Optional[str] = None,
     ):
@@ -53,6 +54,7 @@ def __init__(
             which is supported by `TextFileToDocument` and `AzureOCRDocumentConverter`.
         :param remove_substrings: List of substrings to remove from the text.
         :param remove_regex: Regex to match and replace substrings by "".
+        :param keep_id: keep the ids of the original documents
         """
 
         self.remove_empty_lines = remove_empty_lines
@@ -60,6 +62,7 @@ def __init__(
         self.remove_repeated_substrings = remove_repeated_substrings
         self.remove_substrings = remove_substrings
         self.remove_regex = remove_regex
+        self.keep_id = keep_id
 
     @component.output_types(documents=List[Document])
     def run(self, documents: List[Document]):
@@ -98,7 +101,7 @@ def run(self, documents: List[Document]):
             if self.remove_repeated_substrings:
                 text = self._remove_repeated_substrings(text)
 
-            cleaned_docs.append(Document(content=text, meta=deepcopy(doc.meta)))
+            cleaned_docs.append(Document(content=text, meta=deepcopy(doc.meta), id=doc.id if self.keep_id else ""))
 
         return {"documents": cleaned_docs}
 


Test patch:
diff --git a/test/components/preprocessors/test_document_cleaner.py b/test/components/preprocessors/test_document_cleaner.py
--- a/test/components/preprocessors/test_document_cleaner.py
+++ b/test/components/preprocessors/test_document_cleaner.py
@@ -17,6 +17,7 @@ def test_init(self):
         assert cleaner.remove_repeated_substrings is False
         assert cleaner.remove_substrings is None
         assert cleaner.remove_regex is None
+        assert cleaner.keep_id is False
 
     def test_non_text_document(self, caplog):
         with caplog.at_level(logging.WARNING):
@@ -130,3 +131,11 @@ def test_copy_metadata(self):
         for doc, cleaned_doc in zip(documents, result["documents"]):
             assert doc.meta == cleaned_doc.meta
             assert cleaned_doc.content == "Text."
+
+    def test_keep_id_does_not_alter_document_ids(self):
+        cleaner = DocumentCleaner(keep_id=True)
+        documents = [Document(content="Text. ", id="1"), Document(content="Text. ", id="2")]
+        result = cleaner.run(documents=documents)
+        assert len(result["documents"]) == 2
+        assert result["documents"][0].id == "1"
+        assert result["documents"][1].id == "2"