Repository: deepset-ai/haystack

Issue:
Page number in document meta data not correct
**Describe the bug**
The new "add page number" features, implemented in #2932, seem to have a bug in combination with the `PDFToTextConverter`

**Error message**
Wrong page number in metadata

**To Reproduce**
Use the following haystack components:

```python 

elastic_search_document_store = ElasticsearchDocumentStore(...)

retriever = BM25Retriever(document_store=elastic_search_document_store, all_terms_must_match=False)

converter = PDFToTextConverter(remove_numeric_tables=True, valid_languages=["de", "en"])

pre_processor = PreProcessor(split_length=100, add_page_number=True)  # here we use the new page number flag

reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=False)
```

Now we upload, convert and pre-process the document [tesla_annual_report.pdf](https://github.com/deepset-ai/haystack/files/9430730/tesla_annual_report.pdf)


```python

docs: list[Document] = converter.convert(Path(tmp_file.name), meta={'filename': filename})

docs_pre_processed: list[Document] = pre_processor.process(docs)

elastic_search_document_store.write_documents(docs_pre_processed)

```

Now we search for the search string *'financial statement'*:

```python

search_query = "financial statement"

result_documents: list[Document] = retriever.retrieve(query=search_query, top_k=30)

answers: list[Answer] = self.reader.predict(search_query, result_documents)['answers']

```

if we have a look at the top answer (and the corresponding `Document`) we see the answer
``` 
'reliability
of financial reporting and the preparation of financial statements for external purposes in accordance with generally accepted
accounting principles'
```
and the meta data **page** in the `Document` is **45**. If we though look into the pdf-document and search manually for the answer, we find the answer on page **48**.

The bug appears frequently, with a number of other pdf.


**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: *Ubuntu 20.04* 
 - GPU/CPU: *both*
 - Haystack version (commit or version number): *master, c0fbe45c029247428cee3823292bd574b1846c38*
 - DocumentStore: *see "To Reproduce"*
 - Reader: *see "To Reproduce"*
 - Retriever: *see "To Reproduce"*



Discussion:
Hi @bogdankostic, could you reproduce the issue described? Let me know if you need more information.

Resolving patch:
diff --git a/haystack/nodes/preprocessor/preprocessor.py b/haystack/nodes/preprocessor/preprocessor.py
--- a/haystack/nodes/preprocessor/preprocessor.py
+++ b/haystack/nodes/preprocessor/preprocessor.py
@@ -42,6 +42,8 @@
     "ml": "malayalam",
 }
 
+EMPTY_PAGE_PLACEHOLDER = "@@@HAYSTACK_KEEP_PAGE@@@."
+
 
 class PreProcessor(BasePreProcessor):
     def __init__(
@@ -259,13 +261,17 @@ def clean(
             cleaned_pages = []
             for page in pages:
                 if not page:
-                    continue
-                lines = page.splitlines()
-                cleaned_lines = []
-                for line in lines:
-                    line = line.strip()
-                    cleaned_lines.append(line)
-                cleaned_page = "\n".join(cleaned_lines)
+                    # there are many "empty text" pages in a marketing document, as for example the cover page. If we just forget about them, we have a mismatch
+                    # with page numbers which causes problems later on. Therefore, we replace them with a dummy text, which will not be found by any query.
+                    cleaned_page = EMPTY_PAGE_PLACEHOLDER
+                else:
+                    lines = page.splitlines()
+                    cleaned_lines = []
+                    for line in lines:
+                        line = line.strip()
+                        cleaned_lines.append(line)
+                    cleaned_page = "\n".join(cleaned_lines)
+
                 cleaned_pages.append(cleaned_page)
 
             text = "\f".join(cleaned_pages)
@@ -332,7 +338,7 @@ def split(
             list_splits = []
             current_slice: List[str] = []
             for sen in sentences:
-                if self.add_page_number and sen.startswith("[NEW_PAGE]"):
+                if self.add_page_number and "[NEW_PAGE]" in sen:
                     sen = sen.replace("[NEW_PAGE]", "\f")
 
                 word_count_sen = len(sen.split(" "))
@@ -429,7 +435,12 @@ def split(
         # create new document dicts for each text split
         documents = []
         for i, txt in enumerate(text_splits):
-            doc = Document(content=txt, meta=deepcopy(document.meta) or {}, id_hash_keys=id_hash_keys)
+            # now we want to get rid of the empty page placeholder and skip the split if there's nothing left
+            txt_clean = txt.replace(EMPTY_PAGE_PLACEHOLDER, "")
+            if not txt_clean.strip():
+                continue
+
+            doc = Document(content=txt_clean, meta=deepcopy(document.meta) or {}, id_hash_keys=id_hash_keys)
             doc.meta["_split_id"] = i
             if self.add_page_number:
                 doc.meta["page"] = splits_pages[i]


Test patch:
diff --git a/test/nodes/test_preprocessor.py b/test/nodes/test_preprocessor.py
--- a/test/nodes/test_preprocessor.py
+++ b/test/nodes/test_preprocessor.py
@@ -222,6 +222,28 @@ def test_page_number_extraction(test_input):
             assert doc.meta["page"] == 2
 
 
+def test_page_number_extraction_on_empty_pages():
+    """
+    Often "marketing" documents contain pages without text (visuals only). When extracting page numbers, these pages should be counted as well to avoid
+    issues when mapping results back to the original document.
+    """
+    preprocessor = PreProcessor(add_page_number=True, split_by="word", split_length=7, split_overlap=0)
+    text_page_one = "This is a text on page one."
+    text_page_three = "This is a text on page three."
+    # this is what we get from PDFToTextConverter in case of an "empty" page
+    document_with_empty_pages = f"{text_page_one}\f\f{text_page_three}"
+    document = Document(content=document_with_empty_pages)
+
+    documents = preprocessor.process(document)
+
+    assert documents[0].meta["page"] == 1
+    assert documents[1].meta["page"] == 3
+
+    # verify the placeholder for the empty page has been removed
+    assert documents[0].content.strip() == text_page_one
+    assert documents[1].content.strip() == text_page_three
+
+
 def test_substitute_page_break():
     # Page breaks at the end of sentences should be replaced by "[NEW_PAGE]", while page breaks in between of
     # sentences should not be replaced.