Repository: deepset-ai/haystack

Issue:
Bug: The DocumentCleaner is deleting the \f used to calculate the page_number of documents
**Describe the bug**
When using a DocumentCleaner in my file processing pipeline, the page_number of the documents are wrong.
After analysis, the page change is handled by the character \f which can be deleted by the DocumentCleaner when using the options remove_empty_lines or remove_extra_whitespaces

**Expected behavior**
If my document contains N \f characters when it enters the DocumentCleaner, it should have the same amount when it exits.

**Additional context**
The bug on remove_extra_whitespaces is because \f is part of the regex \s.
The bug on remove_empty_lines is on line.strip() which is probably removing the \f too.

**To Reproduce**
Here is some simple code to show the issue. I made a simple content with the \f in different position in a text and 4 options of DocumentCleaner. It only keeps the page change \f characters if I disable remove_empty_lines & remove_extra_whitespaces

``` python
from haystack import Document
from haystack.components.preprocessors import DocumentCleaner

doc_content="""This is a test

\ft
\f

Some more content

The end\fNo space
"""

doc = Document(content=doc_content)

document_cleaner_1 = DocumentCleaner(remove_empty_lines=True, remove_extra_whitespaces=True)
document_cleaner_2 = DocumentCleaner(remove_empty_lines=False, remove_extra_whitespaces=True)
document_cleaner_3 = DocumentCleaner(remove_empty_lines=True, remove_extra_whitespaces=False)
document_cleaner_4 = DocumentCleaner(remove_empty_lines=False, remove_extra_whitespaces=False)

cleaned_doc_1 = document_cleaner_1.run([doc])['documents'][0]
print(cleaned_doc_1.content.count('\f')) #Should be 3
print(cleaned_doc_1.content.replace('\f', '\\f'))
print()

cleaned_doc_2 = document_cleaner_1.run([doc])['documents'][0]
print(cleaned_doc_2.content.count('\f')) #Should be 3
print(cleaned_doc_2.content.replace('\f', '\\f'))
print()

cleaned_doc_3 = document_cleaner_3.run([doc])['documents'][0]
print(cleaned_doc_3.content.count('\f')) #Should be 3
print(cleaned_doc_3.content.replace('\f', '\\f'))
print()

cleaned_doc_4 = document_cleaner_4.run([doc])['documents'][0]
print(cleaned_doc_4.content.count('\f')) #Should be 3
print(cleaned_doc_4.content.replace('\f', '\\f'))
print()
```

Result:
``` bash
1
This is a test t Some more content The end\fNo space

1
This is a test t Some more content The end\fNo space

2
This is a test
\ft
Some more content
The end\fNo space

3
This is a test

\ft
\f

Some more content

The end\fNo space
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Windows
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 2.3.0
 - DocumentStore: None
 - Reader: None
 - Retriever: None



Discussion:
I looked at the code some more.

For remove_empty_lines, it's clearly a bug. If there's a \f in the line, it's not an empty line.
But for remove_extra_whitespaces, there's a debate. \f are considered as whitespace characters (https://en.wikipedia.org/wiki/Whitespace_character). But are they "extra whitespaces"? I'm not sure.
It also made me think I'm not sure that remove_extra_whitespaces should remove line returns (\r and or \n). They are whitespace characters. But is it what we expect from this DocumentCleaner. Wouldn't the RAG work better with those line returns?

I do believe that the DocumentCleaner with its default options should not remove the \f characters. For the rest, I'm not sure how it should work.
The issue is due to the usage of `strip()` in the different part of the code like `non_empty_lines = filter(lambda line: line.strip() != "", lines)`
`.strip()` methods removes all \n, \t, \f, \v ....
We can find a way to keep this behaviour while excluding \f removal, but I'm not sure how except with regex.
I have an idea, I will make a PR.
https://github.com/deepset-ai/haystack/pull/8078 Suggested fix



Patch:
diff --git a/haystack/components/preprocessors/document_cleaner.py b/haystack/components/preprocessors/document_cleaner.py
index 92e11eb342..8a1a0ce751 100644
--- a/haystack/components/preprocessors/document_cleaner.py
+++ b/haystack/components/preprocessors/document_cleaner.py
@@ -113,9 +113,9 @@ def _remove_empty_lines(self, text: str) -> str:
         :param text: Text to clean.
         :returns: The text without empty lines.
         """
-        lines = text.split("\n")
-        non_empty_lines = filter(lambda line: line.strip() != "", lines)
-        return "\n".join(non_empty_lines)
+        pages = text.split("\f")
+        cleaned_pages = ["\n".join(line for line in page.split("\n") if line.strip()) for page in pages]
+        return "\f".join(cleaned_pages)
 
     def _remove_extra_whitespaces(self, text: str) -> str:
         """
@@ -124,7 +124,9 @@ def _remove_extra_whitespaces(self, text: str) -> str:
         :param text: Text to clean.
         :returns: The text without extra whitespaces.
         """
-        return re.sub(r"\s\s+", " ", text).strip()
+        texts = text.split("\f")
+        cleaned_text = [re.sub(r"\s\s+", " ", text).strip() for text in texts]
+        return "\f".join(cleaned_text)
 
     def _remove_regex(self, text: str, regex: str) -> str:
         """
@@ -134,7 +136,9 @@ def _remove_regex(self, text: str, regex: str) -> str:
         :param regex: Regex to match and replace substrings by "".
         :returns: The text without the substrings that match the regex.
         """
-        return re.sub(regex, "", text).strip()
+        texts = text.split("\f")
+        cleaned_text = [re.sub(regex, "", text).strip() for text in texts]
+        return "\f".join(cleaned_text)
 
     def _remove_substrings(self, text: str, substrings: List[str]) -> str:
         """
diff --git a/releasenotes/notes/fix-document-cleaner-page-tag-15c66d6433b82b0a.yaml b/releasenotes/notes/fix-document-cleaner-page-tag-15c66d6433b82b0a.yaml
new file mode 100644
index 0000000000..b25bbb662c
--- /dev/null
+++ b/releasenotes/notes/fix-document-cleaner-page-tag-15c66d6433b82b0a.yaml
@@ -0,0 +1,4 @@
+---
+fixes:
+  - |
+    Fix the DocumentCleaner removing the `\f` tag from content preventing from counting page number (by Splitter for example).


Test patch:
diff --git a/test/components/preprocessors/test_document_cleaner.py b/test/components/preprocessors/test_document_cleaner.py
index 0acd9e8e8c..878fb27ebd 100644
--- a/test/components/preprocessors/test_document_cleaner.py
+++ b/test/components/preprocessors/test_document_cleaner.py
@@ -40,7 +40,7 @@ def test_remove_empty_lines(self):
         result = cleaner.run(
             documents=[
                 Document(
-                    content="This is a text with some words. "
+                    content="This is a text with some words. \f"
                     ""
                     "There is a second sentence. "
                     ""
@@ -51,7 +51,7 @@ def test_remove_empty_lines(self):
         assert len(result["documents"]) == 1
         assert (
             result["documents"][0].content
-            == "This is a text with some words. There is a second sentence. And there is a third sentence."
+            == "This is a text with some words. \fThere is a second sentence. And there is a third sentence."
         )
 
     def test_remove_whitespaces(self):
@@ -63,33 +63,33 @@ def test_remove_whitespaces(self):
                     ""
                     "There is a second sentence.  "
                     ""
-                    "And there  is a third sentence. "
+                    "And there  is a third sentence.\f "
                 )
             ]
         )
         assert len(result["documents"]) == 1
         assert result["documents"][0].content == (
-            "This is a text with some words. " "" "There is a second sentence. " "" "And there is a third sentence."
+            "This is a text with some words. " "" "There is a second sentence. " "" "And there is a third sentence.\f"
         )
 
     def test_remove_substrings(self):
         cleaner = DocumentCleaner(remove_substrings=["This", "A", "words", "🪲"])
-        result = cleaner.run(documents=[Document(content="This is a text with some words.🪲")])
+        result = cleaner.run(documents=[Document(content="This is a text with some words.\f🪲")])
         assert len(result["documents"]) == 1
-        assert result["documents"][0].content == " is a text with some ."
+        assert result["documents"][0].content == " is a text with some .\f"
 
     def test_remove_regex(self):
         cleaner = DocumentCleaner(remove_regex=r"\s\s+")
-        result = cleaner.run(documents=[Document(content="This is a  text with   some words.")])
+        result = cleaner.run(documents=[Document(content="This is a  text \f with   some words.")])
         assert len(result["documents"]) == 1
-        assert result["documents"][0].content == "This is a text with some words."
+        assert result["documents"][0].content == "This is a text\fwith some words."
 
     def test_remove_repeated_substrings(self):
         cleaner = DocumentCleaner(
             remove_empty_lines=False, remove_extra_whitespaces=False, remove_repeated_substrings=True
         )
 
-        text = """First PageThis is a header.
+        text = """First Page\fThis is a header.
         Page  of
         2
         4
@@ -109,7 +109,7 @@ def test_remove_repeated_substrings(self):
         This is a footer number 1
         This is footer number 2"""
 
-        expected_text = """First Page 2
+        expected_text = """First Page\f 2
         4
         Lorem ipsum dolor sit amet 3
         4