Repository: deepset-ai/haystack
Pull request: feat: Add headline extraction to `ParsrConverter`
### Related Issues
- fixes #3057

### Proposed Changes:
 <!--- In case of a bug: Describe what caused the issue and how you solved it -->
 <!--- In case of a feature: Describe what did you add and how it works -->

This PR adds the possibility to extract headlines out of PDF files using `ParsrConverter`. It follows the structure for headlines as defined in #3445.
During development, I noticed that Parsr's built-in headline extraction does not work well on all PDFs, so use this with caution.

### How did you test it?
<!-- unit tests, integration tests, manual verification, instructions for manual tests -->
I added a unit test.

### Checklist
- [x] I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- [ ] I have updated the related issue with new insights and changes
- [x] I added tests that demonstrate the correct behavior of the change
- [x] I've used the [conventional commit convention](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title
- [x] I documented my code
- [x] I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Make use of Parsr's heading detection

**Describe the solution you'd like**
Parsr has built-in heading detection. We should make use of it and add headline information of PDFs to the converted Documents. As far as I know, Parsr only detects the headings but not the hierarchy of the headings. We might determine the hierarchy of the headings by using a heuristic based on the font size of the headings.

Hints from issue discussion:

Commit message: Use extract_headlines if set in convert method

Patch to review:
diff --git a/haystack/nodes/file_converter/parsr.py b/haystack/nodes/file_converter/parsr.py
index a13d609f0a..4186ed1751 100644
--- a/haystack/nodes/file_converter/parsr.py
+++ b/haystack/nodes/file_converter/parsr.py
@@ -44,6 +44,7 @@ def __init__(
         valid_languages: Optional[List[str]] = None,
         id_hash_keys: Optional[List[str]] = None,
         add_page_number: bool = True,
+        extract_headlines: bool = True,
     ):
         """
         :param parsr_url: URL endpoint to Parsr"s REST API.
@@ -69,6 +70,7 @@ def __init__(
             In this case the id will be generated by using the content and the defined metadata.
         :param add_page_number: Adds the number of the page a table occurs in to the Document's meta field
                                 `"page"`.
+        :param extract_headlines: Whether to extract headings from the PDF file.
         """
         super().__init__(valid_languages=valid_languages, id_hash_keys=id_hash_keys)
 
@@ -96,6 +98,7 @@ def __init__(
         self.remove_page_footers = remove_page_footers
         self.remove_table_of_contents = remove_table_of_contents
         self.add_page_number = add_page_number
+        self.extract_headlines = extract_headlines
         super().__init__(valid_languages=valid_languages)
 
     def convert(
@@ -106,6 +109,7 @@ def convert(
         valid_languages: Optional[List[str]] = None,
         encoding: Optional[str] = "utf-8",
         id_hash_keys: Optional[List[str]] = None,
+        extract_headlines: Optional[bool] = None,
     ) -> List[Document]:
         """
         Extract text and tables from a PDF or DOCX using the open-source Parsr tool.
@@ -124,12 +128,11 @@ def convert(
             attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
             not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
             In this case the id will be generated by using the content and the defined metadata.
+        :param extract_headlines: Whether to extract headings from the PDF file.
         """
-        if valid_languages is None:
-            valid_languages = self.valid_languages
-
-        if id_hash_keys is None:
-            id_hash_keys = self.id_hash_keys
+        valid_languages = valid_languages if valid_languages is not None else self.valid_languages
+        id_hash_keys = id_hash_keys if id_hash_keys is not None else self.id_hash_keys
+        extract_headlines = extract_headlines if extract_headlines is not None else self.extract_headlines
 
         with open(file_path, "rb") as pdf_file:
             # Send file to Parsr
@@ -154,16 +157,28 @@ def convert(
             # Convert Parsr output to Haystack Documents
             text = ""
             tables = []
+            headlines = []
             for page_idx, page in enumerate(parsr_output["pages"]):
                 for elem_idx, element in enumerate(page["elements"]):
                     if element["type"] in ["paragraph", "heading", "table-of-contents"]:
                         current_paragraph = self._convert_text_element(element)
                         if current_paragraph:
+                            if element["type"] == "heading" and extract_headlines:
+                                headlines.append(
+                                    {"headline": current_paragraph, "start_idx": len(text), "level": element["level"]}
+                                )
                             text += f"{current_paragraph}\n\n"
 
                     elif element["type"] == "table":
                         table = self._convert_table_element(
-                            element, parsr_output["pages"], page_idx, elem_idx, meta, id_hash_keys
+                            element,
+                            parsr_output["pages"],
+                            page_idx,
+                            elem_idx,
+                            headlines,
+                            extract_headlines,
+                            meta,
+                            id_hash_keys,
                         )
                         tables.append(table)
                 if text[-1] != "\f":
@@ -184,6 +199,10 @@ def convert(
                     f"been decoded in the correct text format."
                 )
 
+        if self.extract_headlines:
+            meta = meta if meta else {}
+            meta["headlines"] = headlines
+
         docs = tables + [Document(content=text.strip(), meta=meta, id_hash_keys=id_hash_keys)]
         return docs
 
@@ -219,6 +238,8 @@ def _convert_table_element(
         all_pages: List[Dict],
         page_idx: int,
         elem_idx: int,
+        headlines: List[Dict],
+        extract_headlines: bool,
         meta: Optional[Dict[str, Any]] = None,
         id_hash_keys: Optional[List[str]] = None,
     ) -> Document:
@@ -290,5 +311,17 @@ def _convert_table_element(
         if self.add_page_number:
             table_meta["page"] = page_idx + 1
 
+        if extract_headlines:
+            relevant_headlines = []
+            cur_lowest_headline_level = 1000
+            for headline in reversed(headlines):
+                if headline["level"] < cur_lowest_headline_level:
+                    headline_copy = copy.deepcopy(headline)
+                    headline_copy["start_idx"] = None
+                    relevant_headlines.append(headline_copy)
+                    cur_lowest_headline_level = headline_copy["level"]
+            relevant_headlines = list(reversed(relevant_headlines))
+            table_meta["headlines"] = relevant_headlines
+
         table_df = pd.DataFrame(columns=table_list[0], data=table_list[1:])
         return Document(content=table_df, content_type="table", meta=table_meta, id_hash_keys=id_hash_keys)
diff --git a/test/nodes/test_file_converter.py b/test/nodes/test_file_converter.py
index 1cffe6b552..f579dd7c3d 100644
--- a/test/nodes/test_file_converter.py
+++ b/test/nodes/test_file_converter.py
@@ -220,6 +220,36 @@ def test_parsr_converter():
     assert docs[1].content.endswith("Page 4 of Sample PDF\n… the page 3 is empty.")
 
 
+@pytest.mark.skipif(sys.platform in ["win32", "cygwin"], reason="Parsr not running on Windows CI")
+def test_parsr_converter_headline_extraction():
+    expected_headlines = [
+        [("Lorem ipsum", 1), ("Cras fringilla ipsum magna, in fringilla dui commodo\na.", 2)],
+        [
+            ("Lorem ipsum", 1),
+            ("Lorem ipsum dolor sit amet, consectetur adipiscing\nelit. Nunc ac faucibus odio.", 2),
+            ("Cras fringilla ipsum magna, in fringilla dui commodo\na.", 2),
+            ("Lorem ipsum dolor sit amet, consectetur adipiscing\nelit.", 2),
+            ("Maecenas mauris lectus, lobortis et purus mattis, blandit\ndictum tellus.", 2),
+            ("In eleifend velit vitae libero sollicitudin euismod.", 2),
+        ],
+    ]
+
+    converter = ParsrConverter()
+
+    docs = converter.convert(file_path=str((SAMPLES_PATH / "pdf" / "sample_pdf_4.pdf").absolute()))
+    for doc, expectation in zip(docs, expected_headlines):
+        for extracted_headline, (expected_headline, expected_level) in zip(doc.meta["headlines"], expectation):
+            # Check if correct headline and level is extracted
+            assert extracted_headline["headline"] == expected_headline
+            assert extracted_headline["level"] == expected_level
+
+            # Check if correct start_idx is extracted
+            if doc.content_type == "text":
+                start_idx = extracted_headline["start_idx"]
+                hl_len = len(extracted_headline["headline"])
+                assert extracted_headline["headline"] == doc.content[start_idx : start_idx + hl_len]
+
+
 def test_id_hash_keys_from_pipeline_params():
     doc_path = SAMPLES_PATH / "docs" / "doc_1.txt"
     meta_1 = {"key": "a"}
diff --git a/test/samples/pdf/sample_pdf_4.pdf b/test/samples/pdf/sample_pdf_4.pdf
new file mode 100644
index 0000000000..94d9477974
Binary files /dev/null and b/test/samples/pdf/sample_pdf_4.pdf differ


Code review comments:
diff_hunk: @@ -290,5 +311,17 @@ def _convert_table_element(
         if self.add_page_number:
             table_meta["page"] = page_idx + 1
 
+        if extract_headlines:
+            relevant_headlines = []
+            cur_lowest_headline_level = 1000
+            for headline in reversed(headlines):
+                if headline["level"] < cur_lowest_headline_level:
+                    headline_copy = copy.deepcopy(headline)
+                    headline_copy["start_idx"] = None
+                    relevant_headlines.append(headline_copy)
+                    cur_lowest_headline_level = headline_copy["level"]
+            relevant_headlines = list(reversed(relevant_headlines))
path: haystack/nodes/file_converter/parsr.py
text: @user1:
nit: the "alien face" operator would be faster here: `relevant_headlines = relevant_headlines[::-1]`. I expect the list to be small enough for this not not matter, leaving here just in case.

diff_hunk: @@ -184,6 +199,10 @@ def convert(
                     f"been decoded in the correct text format."
                 )
 
+        if self.extract_headlines:
path: haystack/nodes/file_converter/parsr.py
text: @user1:
should this be `if extract_headlines`?

@author:
Good catch

diff_hunk: @@ -184,6 +199,10 @@ def convert(
                     f"been decoded in the correct text format."
                 )
 
+        if self.extract_headlines:
+            meta = meta if meta else {}
path: haystack/nodes/file_converter/parsr.py
text: @user1:
I would check if `meta` is `None` here, you could also move this "initialization" of the param at the beginning of the method, along with the others

diff_hunk: @@ -124,12 +128,11 @@ def convert(
             attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
             not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
             In this case the id will be generated by using the content and the defined metadata.
+        :param extract_headlines: Whether to extract headings from the PDF file.
         """
-        if valid_languages is None:
-            valid_languages = self.valid_languages
-
-        if id_hash_keys is None:
-            id_hash_keys = self.id_hash_keys
+        valid_languages = valid_languages if valid_languages is not None else self.valid_languages
+        id_hash_keys = id_hash_keys if id_hash_keys is not None else self.id_hash_keys
+        extract_headlines = extract_headlines if extract_headlines is not None else self.extract_headlines
path: haystack/nodes/file_converter/parsr.py
text: @user1:
nit: I find the other form more concise and readable, these one-liners go around the 100th column

diff_hunk: @@ -290,5 +311,17 @@ def _convert_table_element(
         if self.add_page_number:
             table_meta["page"] = page_idx + 1
 
+        if extract_headlines:
+            relevant_headlines = []
+            cur_lowest_headline_level = 1000
path: haystack/nodes/file_converter/parsr.py
text: @user1:
nit: if you use `sys.maxsize` instead of 1000, your intent is more obvious (for example, at first I was wondering if 1000 was somehow special in this context)

diff_hunk: @@ -220,6 +220,36 @@ def test_parsr_converter():
     assert docs[1].content.endswith("Page 4 of Sample PDF\n… the page 3 is empty.")
 
 
+@pytest.mark.skipif(sys.platform in ["win32", "cygwin"], reason="Parsr not running on Windows CI")
+def test_parsr_converter_headline_extraction():
+    expected_headlines = [
+        [("Lorem ipsum", 1), ("Cras fringilla ipsum magna, in fringilla dui commodo\na.", 2)],
+        [
+            ("Lorem ipsum", 1),
+            ("Lorem ipsum dolor sit amet, consectetur adipiscing\nelit. Nunc ac faucibus odio.", 2),
+            ("Cras fringilla ipsum magna, in fringilla dui commodo\na.", 2),
+            ("Lorem ipsum dolor sit amet, consectetur adipiscing\nelit.", 2),
+            ("Maecenas mauris lectus, lobortis et purus mattis, blandit\ndictum tellus.", 2),
+            ("In eleifend velit vitae libero sollicitudin euismod.", 2),
+        ],
+    ]
+
+    converter = ParsrConverter()
+
+    docs = converter.convert(file_path=str((SAMPLES_PATH / "pdf" / "sample_pdf_4.pdf").absolute()))
+    for doc, expectation in zip(docs, expected_headlines):
path: test/nodes/test_file_converter.py
text: @user1:
should we assert the number of docs is exactly 2 before getting here? The second item in `expected_headlines` wouldn't be tested if some day `docs` contained only one item.