Repository: deepset-ai/haystack
Pull request: feat: add frontmatter to meta in `MarkdownConverter`
### Related Issues
- fixes #3945 

### Proposed Changes:
I propose using the `python-frontmatter` package to split the content of a markdown from it's frontmatter and providing an option for people to add the frontmatter to the `meta` 

### How did you test it?
`MarkdownConverter(add_frontmatter_to_meta=True)`
Use tutorial 1 and convert it to a Document

### Notes for the reviewer
This is something we would like to implement to enable search for tutorials

Is this an ok fix? If yes I can update the docstrings. And let me know if there is anywhere I should add tests to

### Checklist
- [x] I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- [x] I have updated the related issue with new insights and changes
- [x] I added tests that demonstrate the correct behavior of the change
- [x] I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- [x] I documented my code
- [x] I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Remove frontmatter and add to `meta` in markdown converter
**Is your feature request related to a problem? Please describe.**
We would like to use the markdown converter for the `haystack-tutorials` search

**Describe the solution you'd like**
Frontmatter is often used to store some meta information of markdown files. So it would be a useful feature to be able to auto add these to the `meta` of the created documents
It's also a good idea to remove the fronmatter from the created documents too.

Hints from issue discussion:

Commit message: running black and pre-commit

Patch to review:
diff --git a/haystack/nodes/file_converter/markdown.py b/haystack/nodes/file_converter/markdown.py
index 7c5af831c9..e17aaa7ade 100644
--- a/haystack/nodes/file_converter/markdown.py
+++ b/haystack/nodes/file_converter/markdown.py
@@ -1,5 +1,6 @@
 import logging
 import re
+import frontmatter
 from pathlib import Path
 from typing import Dict, List, Optional, Tuple, Any
 
@@ -27,6 +28,7 @@ def __init__(
         progress_bar: bool = True,
         remove_code_snippets: bool = True,
         extract_headlines: bool = False,
+        add_frontmatter_to_meta: bool = False,
     ):
         """
         :param remove_numeric_tables: Not applicable.
@@ -48,6 +50,7 @@ def __init__(
 
         self.remove_code_snippets = remove_code_snippets
         self.extract_headlines = extract_headlines
+        self.add_frontmatter_to_meta = add_frontmatter_to_meta
 
     def convert(
         self,
@@ -59,6 +62,7 @@ def convert(
         id_hash_keys: Optional[List[str]] = None,
         remove_code_snippets: Optional[bool] = None,
         extract_headlines: Optional[bool] = None,
+        add_frontmatter_to_meta: Optional[bool] = None,
     ) -> List[Document]:
         """
         Reads text from a markdown file and executes optional preprocessing steps.
@@ -79,9 +83,12 @@ def convert(
         id_hash_keys = id_hash_keys if id_hash_keys is not None else self.id_hash_keys
         remove_code_snippets = remove_code_snippets if remove_code_snippets is not None else self.remove_code_snippets
         extract_headlines = extract_headlines if extract_headlines is not None else self.extract_headlines
+        add_frontmatter_to_meta = (
+            add_frontmatter_to_meta if add_frontmatter_to_meta is not None else self.add_frontmatter_to_meta
+        )
 
         with open(file_path, encoding=encoding, errors="ignore") as f:
-            markdown_text = f.read()
+            metadata, markdown_text = frontmatter.parse(f.read())
 
         # md -> html -> text since BeautifulSoup can extract text cleanly
         html = markdown(markdown_text)
@@ -92,6 +99,12 @@ def convert(
             html = re.sub(r"<code>(.*?)</code>", " ", html, flags=re.DOTALL)
         soup = BeautifulSoup(html, "html.parser")
 
+        if add_frontmatter_to_meta:
+            if meta is None:
+                meta = metadata
+            else:
+                meta.update(metadata)
+
         if extract_headlines:
             text, headlines = self._extract_text_and_headlines(soup)
             if meta is None:
diff --git a/pyproject.toml b/pyproject.toml
index ec33231bf1..26b141bc1e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -162,6 +162,7 @@ crawler = [
 preprocessing = [
   "beautifulsoup4",
   "markdown",
+  "python-frontmatter",
   "python-magic; platform_system != 'Windows'",  # Depends on libmagic: https://pypi.org/project/python-magic/
   "python-magic-bin; platform_system == 'Windows'",  # Needs to be installed without python-magic, otherwise Windows CI gets stuck.
 ]


Code review comments:
diff_hunk: @@ -1,5 +1,6 @@
 import logging
 import re
+import frontmatter
path: haystack/nodes/file_converter/markdown.py
text: @user1:
Could you put the `import frontmatter` in the `try:` block on line 7? Since this will be an optional package that can be installed using the `preprocessing` dependencies we want to catch the `ImportError` and print the optional component not installed.