Repository: deepset-ai/haystack
Pull request: feat: Enable PDFToTextConverter multiprocessing, increase general performance and simplify installation
### Related Issues
- fixes #4225 

### Proposed Changes:
- ~~Remove xpdf binary installation by the user, it will be automatic as a fallback mechanism~~
- Remove xpdf binary dependency completely
- Enable **pure pythonic** implementation for PDF converter
- **Increase** PDF converter realiability for non-standard PDF files and uncommon encodings
- Support **multiprocessing** (up to 6x in-house tests, for highly complex PDFs)
- Automatic **encoding detection**

#### ⚠️ Breaking Change ⚠️
This PR changes the backend for the `PDFToTextConverter` from `xpdf` to `PyMuPDFF`. Therefore, in rare cases the extracted text for some PDFs might change. Also, as `PyMuPDF` doesn't support the `keep_physical_layout` parameter, `keep_physical_layout` is deprecated and setting it to `True` will have no effect.  Instead `PDFToTextConverter` now provides the parameter `sort_by_position` which will sort the extracted text by vertical and horizontal position.

### How did you test it?
- CI
- Local tests at Python 3.8 and Python 3.10

### Notes for the reviewer

**_Official developer tests_**

[Test documents](https://pymupdf.readthedocs.io/en/latest/app4.html#appendix4-files-used)
[Test methodology](https://pymupdf.readthedocs.io/en/latest/app4.html#appendix4)

![image](https://user-images.githubusercontent.com/90110233/220449245-6c33c2d6-1bec-4eef-b020-b0fd26774cad.png)

![image](https://user-images.githubusercontent.com/90110233/220449834-c7c21552-827b-42d9-b4eb-fc2e3669c43b.png)

This new PDF converter will make it elementary to implement e-book converters (mobi, cbz, ...) and XPS in the future, as the base code can be shared.

PyMuPDF will provide a more reliable way of extracting texts from PDF files. During in-house processing, we noticed xPdf throwing plenty of errors caused by non-standard encoding. The usage of PyMuPDF will solve this by its encoding detection and handling. 

Multiprocessing was implemented to speed up PDF processing (PyMuPDF only, as xPdf multiprocessing/multithreading proved after tests to be of no benefit, and even slow processing times in small PDFs).

PyMuPDF is only faster than xPdf in random cases (after further personal investigation). We were unable to fully reproduce the official tests. But when we are thinking about easy-to-use implementation and for large enterprise pipelines, where we don't have previous encoding all setup for each file, it proved to be the faster solution. 

Example of common encoding issues: `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe7 in position 104: invalid continuation byte`, which would need to manually set it to Latin1. We face numerous issues internally caused by mixed UTF-8 and Latin1 encodings.

### Checklist
- [x] I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- [x] I have updated the related issue with new insights and changes
- [x] I added tests that demonstrate the correct behavior of the change
- [x] I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- [x] I documented my code
- [x] I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Improve PDF converter performance and simplify installation
**Is your feature request related to a problem? Please describe.**
As a user, I want to use the PDF converter without extra installation steps. And I would like the performance to be reasonable, specially when processing large PDF files.

**Describe the solution you'd like**
Implement a new PDF converter base class, which uses more performant code and simplify installation process.

**Describe alternatives you've considered**
- xpdf
- PyPDF2
- PDFMiner
- Tika

**Additional context**

Hints from issue discussion:

Commit message: fix: regression when moved code

Patch to review:
diff --git a/haystack/nodes/file_converter/__init__.py b/haystack/nodes/file_converter/__init__.py
index 53a83e8bda..1a60448a92 100644
--- a/haystack/nodes/file_converter/__init__.py
+++ b/haystack/nodes/file_converter/__init__.py
@@ -9,6 +9,7 @@
 from haystack.nodes.file_converter.txt import TextConverter
 from haystack.nodes.file_converter.azure import AzureConverter
 from haystack.nodes.file_converter.parsr import ParsrConverter
+from haystack.nodes.file_converter.pdf import PDFToTextConverter
 
 MarkdownConverter = safe_import(
     "haystack.nodes.file_converter.markdown", "MarkdownConverter", "preprocessing"
@@ -16,9 +17,6 @@
 ImageToTextConverter = safe_import(
     "haystack.nodes.file_converter.image", "ImageToTextConverter", "ocr"
 )  # Has optional dependencies
-PDFToTextConverter = safe_import(
-    "haystack.nodes.file_converter.pdf", "PDFToTextConverter", "ocr"
-)  # Has optional dependencies
 PDFToTextOCRConverter = safe_import(
-    "haystack.nodes.file_converter.pdf", "PDFToTextOCRConverter", "ocr"
+    "haystack.nodes.file_converter.pdf_ocr", "PDFToTextOCRConverter", "ocr"
 )  # Has optional dependencies
diff --git a/haystack/nodes/file_converter/pdf.py b/haystack/nodes/file_converter/pdf.py
index 6594e4cfb9..5d885998a0 100644
--- a/haystack/nodes/file_converter/pdf.py
+++ b/haystack/nodes/file_converter/pdf.py
@@ -1,18 +1,11 @@
 import logging
-import subprocess
-import tempfile
+from multiprocessing import Pool, cpu_count
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Union
 
-try:
-    from pdf2image import convert_from_path
-except (ImportError, ModuleNotFoundError) as ie:
-    from haystack.utils.import_utils import _optional_component_not_installed
-
-    _optional_component_not_installed(__name__, "ocr", ie)
+import fitz
 
 from haystack.nodes.file_converter.base import BaseConverter
-from haystack.nodes.file_converter.image import ImageToTextConverter
 from haystack.schema import Document
 
 logger = logging.getLogger(__name__)
@@ -24,8 +17,9 @@ def __init__(
         remove_numeric_tables: bool = False,
         valid_languages: Optional[List[str]] = None,
         id_hash_keys: Optional[List[str]] = None,
-        encoding: Optional[str] = "UTF-8",
+        encoding: Optional[str] = None,
         keep_physical_layout: bool = False,
+        multiprocessing: Optional[Union[bool, int]] = None,
     ):
         """
         :param remove_numeric_tables: This option uses heuristics to remove numeric rows from the tables.
@@ -42,35 +36,24 @@ def __init__(
             attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
             not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
             In this case the id will be generated by using the content and the defined metadata.
-        :param encoding: Encoding that will be passed as `-enc` parameter to `pdftotext`.
-                         Defaults to "UTF-8" in order to support special characters (e.g. German Umlauts, Cyrillic ...).
-                         (See list of available encodings, such as "Latin1", by running `pdftotext -listenc` in the terminal)
+        :param encoding: This parameter is deprecated and will be removed in future versions.
+                         Encoding is automatically detected now.
         :param keep_physical_layout: This option will maintain original physical layout on the extracted text.
             It works by passing the `-layout` parameter to `pdftotext`. When disabled, PDF is read in the stream order.
+        :param multiprocessing: Whether to use multiprocessing to speed up the conversion. If set to True, the total number of cores will be used.
+                                If set to an integer, that number of cores will be used.
         """
         super().__init__(
             remove_numeric_tables=remove_numeric_tables, valid_languages=valid_languages, id_hash_keys=id_hash_keys
         )
-        try:
-            subprocess.run(["pdftotext", "-v"], shell=False, check=False)
-        except FileNotFoundError:
-            raise FileNotFoundError(
-                """pdftotext is not installed. It is part of xpdf or poppler-utils software suite.
-
-                   Installation on Linux:
-                   wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.04.tar.gz &&
-                   tar -xvf xpdf-tools-linux-4.04.tar.gz && sudo cp xpdf-tools-linux-4.04/bin64/pdftotext /usr/local/bin
 
-                   Installation on MacOS:
-                   brew install xpdf
+        if encoding is not None:
+            logger.warning("The `encoding` parameter is deprecated and will be removed in future versions.")
 
-                   You can find more details here: https://www.xpdfreader.com
-                """
-            )
-
-        self.encoding = encoding
         self.keep_physical_layout = keep_physical_layout
 
+        self.multi_processing = multiprocessing
+
     def convert(
         self,
         file_path: Path,
@@ -81,6 +64,7 @@ def convert(
         id_hash_keys: Optional[List[str]] = None,
         start_page: Optional[int] = None,
         end_page: Optional[int] = None,
+        multiprocessing: Optional[Union[bool, int]] = None,
     ) -> List[Document]:
         """
         Extract text from a .pdf file using the pdftotext library (https://www.xpdfreader.com/pdftotext-man.html)
@@ -98,8 +82,8 @@ def convert(
                                 This option can be used to add test for encoding errors. If the extracted text is
                                 not one of the valid languages, then it might likely be encoding error resulting
                                 in garbled text.
-        :param encoding: Encoding that overwrites self.encoding and will be passed as `-enc` parameter to `pdftotext`.
-                         (See list of available encodings by running `pdftotext -listenc` in the terminal)
+        :param encoding: This parameter is deprecated and will be removed in future versions.
+                         Encoding is automatically detected now.
         :param keep_physical_layout: This option will maintain original physical layout on the extracted text.
             It works by passing the `-layout` parameter to `pdftotext`. When disabled, PDF is read in the stream order.
         :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's
@@ -108,6 +92,8 @@ def convert(
             In this case the id will be generated by using the content and the defined metadata.
         :param start_page: The page number where to start the conversion
         :param end_page: The page number where to end the conversion.
+        :param multiprocessing: Whether to use multiprocessing to speed up the conversion. If set to True, the total number of cores will be used.
+                                If set to an integer, that number of cores will be used.
         """
         if remove_numeric_tables is None:
             remove_numeric_tables = self.remove_numeric_tables
@@ -115,25 +101,24 @@ def convert(
             valid_languages = self.valid_languages
         if id_hash_keys is None:
             id_hash_keys = self.id_hash_keys
+        if multiprocessing is None:
+            multiprocessing = self.multi_processing
+
+        if encoding is not None:
+            logger.warning("The `encoding` parameter is deprecated and will be removed in future versions.")
 
         keep_physical_layout = self.keep_physical_layout
 
         pages = self._read_pdf(
-            file_path, layout=keep_physical_layout, encoding=encoding, start_page=start_page, end_page=end_page
+            file_path,
+            layout=keep_physical_layout,
+            start_page=start_page,
+            end_page=end_page,
+            multiprocessing=multiprocessing,
         )
 
         cleaned_pages = []
         for page in pages:
-            # pdftotext tool provides an option to retain the original physical layout of a PDF page. This behaviour
-            # can be toggled by using the layout param.
-            #  layout=True
-            #      + table structures get retained better
-            #      - multi-column pages(eg, research papers) gets extracted with text from multiple columns on same line
-            #  layout=False
-            #      + keeps strings in content stream order, hence multi column layout works well
-            #      - cells of tables gets split across line
-            #
-            #  Here, as a "safe" default, layout is turned off.
             lines = page.splitlines()
             cleaned_lines = []
             for line in lines:
@@ -164,13 +149,29 @@ def convert(
         document = Document(content=text, meta=meta, id_hash_keys=id_hash_keys)
         return [document]
 
+    def _get_page_text(self, page_mp):
+        idx, cpu, filename, layout, start, end = page_mp
+
+        doc = fitz.open(filename)
+        num_pages = end - start
+        seg_size = int(num_pages / cpu + 1)
+        seg_from = (idx * seg_size) + start  # our first page number
+        seg_to = min(seg_from + seg_size, num_pages + 1)  # last page number
+
+        text = ""
+        for i in range(seg_from, seg_to):
+            page = doc[i]
+            text += page.get_text("text", sort=layout) + "\f"
+
+        return text
+
     def _read_pdf(
         self,
         file_path: Path,
         layout: bool,
-        encoding: Optional[str] = None,
         start_page: Optional[int] = None,
         end_page: Optional[int] = None,
+        multiprocessing: Optional[Union[bool, int]] = None,
     ) -> List[str]:
         """
         Extract pages from the pdf file at file_path.
@@ -178,118 +179,41 @@ def _read_pdf(
         :param file_path: path of the pdf file
         :param layout: whether to retain the original physical layout for a page. If disabled, PDF pages are read in
                        the content stream order.
-        :param encoding: Encoding that overwrites self.encoding and will be passed as `-enc` parameter to `pdftotext`.
-                         (See list of available encodings by running `pdftotext -listenc` in the terminal)
-        :param start_page: The page number where to start the conversion
+        :param start_page: The page number where to start the conversion, starting from 1.
         :param end_page: The page number where to end the conversion.
+        :param multiprocessing: Whether to use multiprocessing to speed up the conversion. If set to True, the total number of cores will be used.
+                                If set to an integer, that number of cores will be used.
         """
-        if not encoding:
-            encoding = self.encoding
-
-        start_page = start_page or 1
-
-        command = ["pdftotext", "-enc", str(encoding), "-layout" if layout else "-raw", "-f", str(start_page)]
-
-        if end_page is not None:
-            command.extend(["-l", str(end_page)])
-
-        command.extend([str(file_path), "-"])
 
-        output = subprocess.run(command, stdout=subprocess.PIPE, shell=False, check=False)
-        document = output.stdout.decode(errors="ignore")
-        document = "\f" * (start_page - 1) + document  # tracking skipped pages for correct page numbering
+        doc = fitz.open(str(file_path))
+
+        if start_page is None:
+            start_page = 0
+        else:
+            start_page = start_page - 1
+
+        if end_page is None or (end_page is not None and end_page > doc.page_count):
+            end_page = doc.page_count
+
+        document = ""
+        if multiprocessing is not None and (
+            (isinstance(multiprocessing, bool) and multiprocessing)
+            or (isinstance(multiprocessing, int) and multiprocessing > 1)
+        ):
+            cpu = multiprocessing if isinstance(multiprocessing, int) else cpu_count()
+            pages_mp = [(i, cpu, file_path, layout, start_page, end_page) for i in range(cpu)]
+            pool = Pool(processes=cpu)
+            results = pool.map(self._get_page_text, pages_mp)
+
+            for page in results:
+                document += page
+        else:
+            for i in range(start_page, end_page):
+                page = doc[i]
+                document += page.get_text("text", sort=layout) + "\f"
+
+        document = "\f" * start_page + document  # tracking skipped pages for correct page numbering
         pages = document.split("\f")
         pages = pages[:-1]  # the last page in the split is always empty.
 
         return pages
-
-
-class PDFToTextOCRConverter(BaseConverter):
-    def __init__(
-        self,
-        remove_numeric_tables: bool = False,
-        valid_languages: Optional[List[str]] = None,
-        id_hash_keys: Optional[List[str]] = None,
-    ):
-        """
-        Extract text from image file using the pytesseract library (https://github.com/madmaze/pytesseract)
-
-        :param remove_numeric_tables: This option uses heuristics to remove numeric rows from the tables.
-                                      The tabular structures in documents might be noise for the reader model if it
-                                      does not have table parsing capability for finding answers. However, tables
-                                      may also have long strings that could possible candidate for searching answers.
-                                      The rows containing strings are thus retained in this option.
-        :param valid_languages: validate languages from a list of languages supported by tessarect
-                                (https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html).
-                                This option can be used to add test for encoding errors. If the extracted text is
-                                not one of the valid languages, then it might likely be encoding error resulting
-                                in garbled text. If no value is provided, English will be set as default.
-        :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's
-            attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
-            not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
-            In this case the id will be generated by using the content and the defined metadata.
-        """
-        if valid_languages is None:
-            valid_languages = ["eng"]
-        # init image to text instance
-        self.image_2_text = ImageToTextConverter(remove_numeric_tables, valid_languages)
-
-        super().__init__(
-            remove_numeric_tables=remove_numeric_tables, valid_languages=valid_languages, id_hash_keys=id_hash_keys
-        )
-
-    def convert(
-        self,
-        file_path: Path,
-        meta: Optional[Dict[str, Any]] = None,
-        remove_numeric_tables: Optional[bool] = None,
-        valid_languages: Optional[List[str]] = None,
-        encoding: Optional[str] = None,
-        id_hash_keys: Optional[List[str]] = None,
-        start_page: Optional[int] = None,
-        end_page: Optional[int] = None,
-    ) -> List[Document]:
-        """
-        Convert a file to a dictionary containing the text and any associated meta data.
-
-        File converters may extract file meta like name or size. In addition to it, user
-        supplied meta data like author, url, external IDs can be supplied as a dictionary.
-
-        :param file_path: path of the file to convert
-        :param meta: dictionary of meta data key-value pairs to append in the returned document.
-        :param remove_numeric_tables: This option uses heuristics to remove numeric rows from the tables.
-                                      The tabular structures in documents might be noise for the reader model if it
-                                      does not have table parsing capability for finding answers. However, tables
-                                      may also have long strings that could possible candidate for searching answers.
-                                      The rows containing strings are thus retained in this option.
-        :param valid_languages: validate languages from a list of languages specified in the ISO 639-1
-                                (https://en.wikipedia.org/wiki/ISO_639-1) format.
-                                This option can be used to add test for encoding errors. If the extracted text is
-                                not one of the valid languages, then it might likely be encoding error resulting
-                                in garbled text.
-        :param encoding: Not applicable
-        :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's
-            attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
-            not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
-            In this case the id will be generated by using the content and the defined metadata.
-        :param start_page: The page number where to start the conversion
-        :param end_page: The page number where to end the conversion.
-        """
-        if id_hash_keys is None:
-            id_hash_keys = self.id_hash_keys
-
-        start_page = start_page or 1
-
-        pages = []
-        try:
-            images = convert_from_path(file_path, first_page=start_page, last_page=end_page)
-            for image in images:
-                temp_img = tempfile.NamedTemporaryFile(suffix=".jpeg")
-                image.save(temp_img.name)
-                pages.append(self.image_2_text.convert(file_path=temp_img.name)[0].content)
-        except Exception as exception:
-            logger.error("File %s has an error:\n%s", file_path, exception)
-
-        raw_text = "\f" * (start_page - 1) + "\f".join(pages)  # tracking skipped pages for correct page numbering
-        document = Document(content=raw_text, meta=meta, id_hash_keys=id_hash_keys)
-        return [document]
diff --git a/haystack/nodes/file_converter/pdf_ocr.py b/haystack/nodes/file_converter/pdf_ocr.py
new file mode 100644
index 0000000000..a5ca8918e6
--- /dev/null
+++ b/haystack/nodes/file_converter/pdf_ocr.py
@@ -0,0 +1,108 @@
+import logging
+import tempfile
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+try:
+    from pdf2image import convert_from_path
+except (ImportError, ModuleNotFoundError) as ie:
+    from haystack.utils.import_utils import _optional_component_not_installed
+
+    _optional_component_not_installed(__name__, "ocr", ie)
+
+from haystack.nodes.file_converter.base import BaseConverter
+from haystack.nodes.file_converter.image import ImageToTextConverter
+from haystack.schema import Document
+
+logger = logging.getLogger(__name__)
+
+
+class PDFToTextOCRConverter(BaseConverter):
+    def __init__(
+        self,
+        remove_numeric_tables: bool = False,
+        valid_languages: Optional[List[str]] = None,
+        id_hash_keys: Optional[List[str]] = None,
+    ):
+        """
+        Extract text from image file using the pytesseract library (https://github.com/madmaze/pytesseract)
+
+        :param remove_numeric_tables: This option uses heuristics to remove numeric rows from the tables.
+                                      The tabular structures in documents might be noise for the reader model if it
+                                      does not have table parsing capability for finding answers. However, tables
+                                      may also have long strings that could possible candidate for searching answers.
+                                      The rows containing strings are thus retained in this option.
+        :param valid_languages: validate languages from a list of languages supported by tessarect
+                                (https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html).
+                                This option can be used to add test for encoding errors. If the extracted text is
+                                not one of the valid languages, then it might likely be encoding error resulting
+                                in garbled text. If no value is provided, English will be set as default.
+        :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's
+            attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
+            not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
+            In this case the id will be generated by using the content and the defined metadata.
+        """
+        if valid_languages is None:
+            valid_languages = ["eng"]
+        # init image to text instance
+        self.image_2_text = ImageToTextConverter(remove_numeric_tables, valid_languages)
+
+        super().__init__(
+            remove_numeric_tables=remove_numeric_tables, valid_languages=valid_languages, id_hash_keys=id_hash_keys
+        )
+
+    def convert(
+        self,
+        file_path: Path,
+        meta: Optional[Dict[str, Any]] = None,
+        remove_numeric_tables: Optional[bool] = None,
+        valid_languages: Optional[List[str]] = None,
+        encoding: Optional[str] = None,
+        id_hash_keys: Optional[List[str]] = None,
+        start_page: Optional[int] = None,
+        end_page: Optional[int] = None,
+    ) -> List[Document]:
+        """
+        Convert a file to a dictionary containing the text and any associated meta data.
+
+        File converters may extract file meta like name or size. In addition to it, user
+        supplied meta data like author, url, external IDs can be supplied as a dictionary.
+
+        :param file_path: path of the file to convert
+        :param meta: dictionary of meta data key-value pairs to append in the returned document.
+        :param remove_numeric_tables: This option uses heuristics to remove numeric rows from the tables.
+                                      The tabular structures in documents might be noise for the reader model if it
+                                      does not have table parsing capability for finding answers. However, tables
+                                      may also have long strings that could possible candidate for searching answers.
+                                      The rows containing strings are thus retained in this option.
+        :param valid_languages: validate languages from a list of languages specified in the ISO 639-1
+                                (https://en.wikipedia.org/wiki/ISO_639-1) format.
+                                This option can be used to add test for encoding errors. If the extracted text is
+                                not one of the valid languages, then it might likely be encoding error resulting
+                                in garbled text.
+        :param encoding: Not applicable
+        :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's
+            attributes. If you want to ensure you don't have duplicate documents in your DocumentStore but texts are
+            not unique, you can modify the metadata and pass e.g. `"meta"` to this field (e.g. [`"content"`, `"meta"`]).
+            In this case the id will be generated by using the content and the defined metadata.
+        :param start_page: The page number where to start the conversion
+        :param end_page: The page number where to end the conversion.
+        """
+        if id_hash_keys is None:
+            id_hash_keys = self.id_hash_keys
+
+        start_page = start_page or 1
+
+        pages = []
+        try:
+            images = convert_from_path(file_path, first_page=start_page, last_page=end_page)
+            for image in images:
+                temp_img = tempfile.NamedTemporaryFile(suffix=".jpeg")
+                image.save(temp_img.name)
+                pages.append(self.image_2_text.convert(file_path=temp_img.name)[0].content)
+        except Exception as exception:
+            logger.error("File %s has an error:\n%s", file_path, exception)
+
+        raw_text = "\f" * (start_page - 1) + "\f".join(pages)  # tracking skipped pages for correct page numbering
+        document = Document(content=raw_text, meta=meta, id_hash_keys=id_hash_keys)
+        return [document]
diff --git a/pyproject.toml b/pyproject.toml
index bff1377e0a..dd540fb8f4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -73,6 +73,7 @@ dependencies = [
   "python-docx",
   "langdetect",  # for PDF conversions
   "tika",  # Apache Tika (text & metadata extractor)
+  "PyMuPDF>=1.18.16" ,  # PDF text extraction
 
   # See haystack/nodes/retriever/_embedding_encoder.py, _SentenceTransformersEmbeddingEncoder
   "sentence-transformers>=2.2.0",
diff --git a/test/nodes/test_file_converter.py b/test/nodes/test_file_converter.py
index b904abd89b..395df81620 100644
--- a/test/nodes/test_file_converter.py
+++ b/test/nodes/test_file_converter.py
@@ -79,12 +79,12 @@ def test_pdf_command_whitespaces(Converter):
 def test_pdf_encoding(Converter):
     converter = Converter()
 
+    document = converter.run(file_paths=SAMPLES_PATH / "pdf" / "sample_pdf_5.pdf")[0]["documents"][0]
+    assert "Ж" in document.content
+
     document = converter.run(file_paths=SAMPLES_PATH / "pdf" / "sample_pdf_2.pdf")[0]["documents"][0]
     assert "ɪ" in document.content
 
-    document = converter.run(file_paths=SAMPLES_PATH / "pdf" / "sample_pdf_2.pdf", encoding="Latin1")[0]["documents"][0]
-    assert "ɪ" not in document.content
-
 
 @pytest.mark.parametrize("Converter", [PDFToTextConverter])
 def test_pdf_layout(Converter):
@@ -142,6 +142,30 @@ def test_page_range_numbers(Converter):
     assert documents[1].meta["page"] == 4
 
 
+@pytest.mark.parametrize("Converter", [PDFToTextConverter])
+def test_pdf_multiprocessing(Converter):
+    converter = Converter(multiprocessing=True)
+    document = converter.convert(file_path=SAMPLES_PATH / "pdf" / "sample_pdf_6.pdf")[0]
+
+    pages = document.content.split("\f")
+
+    assert pages[0] == "This is the page 1 of the document."
+    assert pages[-1] == "This is the page 3000 of the document."
+
+
+@pytest.mark.parametrize("Converter", [PDFToTextConverter])
+def test_pdf_multiprocessing_page_range(Converter):
+    converter = Converter(multiprocessing=True)
+    document = converter.convert(file_path=SAMPLES_PATH / "pdf" / "sample_pdf_6.pdf", start_page=2)[0]
+
+    pages = document.content.split("\f")
+
+    assert pages[0] == ""
+    assert pages[-1] == "This is the page 3000 of the document."
+
+    assert len(pages) == 3000
+
+
 @pytest.mark.tika
 @pytest.mark.parametrize("Converter", [PDFToTextConverter, TikaConverter])
 def test_table_removal(Converter):
diff --git a/test/samples/pdf/sample_pdf_5.pdf b/test/samples/pdf/sample_pdf_5.pdf
new file mode 100644
index 0000000000..66b4ff4151
Binary files /dev/null and b/test/samples/pdf/sample_pdf_5.pdf differ
diff --git a/test/samples/pdf/sample_pdf_6.pdf b/test/samples/pdf/sample_pdf_6.pdf
new file mode 100644
index 0000000000..de535a2a4f
Binary files /dev/null and b/test/samples/pdf/sample_pdf_6.pdf differ


Code review comments:
diff_hunk: @@ -73,6 +73,7 @@ dependencies = [
   "python-docx",
   "langdetect",  # for PDF conversions
   "tika",  # Apache Tika (text & metadata extractor)
+  "PyMuPDF>=1.18.16" ,  # PDF text extraction
path: pyproject.toml
text: @user1:
This creates a conflict with [Apache-2.0 license](https://github.com/deepset-ai/haystack/blob/main/LICENSE):
https://mupdf.com/licensing/index.html

@user2:
@user1 how so? We don't modify the original source code nor we distribute it directly. It would be different if this PR contained source modules from PyMuPDF, then yes Haystack as a whole would need to be distributed under AGPLv3. 

I might still see how somebody might want to avoid bringing in AGPL software, but if that's the concern maybe we can make it an optional dep under a setup extras?

@user1:
To my understanding, including PyMuPDF as a dependency would convert haystack's license to AGPLv3 (as haystack would include some of PyMuPDF's functionality).

@author:
My perspective is similar to [Heather Meeker](https://www.synopsys.com/blogs/software-security/agpl-affero-gpl-3/), lawyer: 

> Enduring questions
> Unfortunately, the language of the additional network use condition in AGPL leaves some questions unanswered. The most common question is how far “down the stack” the condition reaches. Suppose you are using AGPL software as an operating system or language component to serve up an application. The user interacts with the application, but not with this infrastructure element. The copyleft condition says you must offer source code to “users interacting with it [i.e. the Program] remotely through a computer network.” That seems to mean that infrastructure elements will never trip this condition, because any user interaction would be too indirect.

> Plato’s cave
> Not surprisingly, AGPL3 has been slower to propagate than GPL3; if GPL3 has gained acceptance slowly, AGPL3 has been tagging along behind like a younger sibling. But the license is not as scary as it seems. The network interaction provision only triggers source code offer requirements when you modify the Program. In fact, most open source software is used without modification from community versions. After all, most businesses use open source because it is already tried and tested, free of charge, and supported by the community. Modifying it only undercuts those advantages. So, particularly with killer apps like MongoDB, many companies have become cautiously comfortable approving AGPL software for SaaS use. They still worry—and probably rightly so—that modifications could be made to the code after approval, and they lack internal controls to revisit license compliance when the modifications occur.

AGPLv3 is quite misunderstood. And even some people which uses AGPLv3 as their open-source licensing model, sometime uses it trying to create a freeware. Which is not the case, AGPLv3 is not a freeware license.

About user concerns regarding usage of Haystack with this feature, this would be clearly a case of _linking exception._

@author:
We can add to the list of open-source projects which are under AGPLv3 and don't impose to anybody who makes usage of them to convert it's licensing to AGPLv3:

- MongoDB
- Grafana
- Minio

Imagine that each deployment which used that stack would need to change it's licensing. Many corporations would need to turn their product's license into AGPLv3.
You may say that PyMuPDF is a module, different from the others, which are not. But the license applies to the same direction: source code. That source code produces a result, in the case of PyMuPDF, PDF handling. When we are using the outputs of the module in a public distribution repository (PyPi), and our code clearly shows how it was used 
(good faith demonstration principle), without changing the source code, we are not breaking AGPLv3 in any point. Nor obligating Haystack to convert it's licensing, as the _source-code_ was not incorporated and/or changed.

So, there is no:
- **distribution** -> The package is not distributed by Haystack, but by PyPi, a public repository
- **remote network interaction** -> We are not exposing PyMuPDF directly over the internet to users, and this will probably never be the case to Haystack users. Haystack being a framework, it will be used to build _backend infrastructure_.
- **corresponding source** -> We are not providing any custom installation scripts, methods even without changing the source code. The package is installed from a public repository, PyPi.
- **source code change** -> We have _not changed_ any of the _PyMuPDF original source code_, and this _repository_ is the _source of truth_ of such statement. Haystack is open source, and the usage of PyMuPDF is open to any analysis by any-third party.

Of course, there are cases where some companies try to create additional restrictions, but international law doesn't accept such “estoppel” technique.

On the other side, a violation of AGPLv3 would occur if someone:
- creates a SaaS which has as only product output the PDF handling, even without changing the code.
- change the source code and keep it private
- create custom installation mechanisms for the package
- stores the package _into_ its software distribution (e.g, if the package was already in the Haystack distribution, and not being installed from a public repository, specially when the own licensor company made it available)

@author:
In any case, I'm refactoring this PR to support both xpdf and PyMuPDF. The default installation will use xpdf, and each user will be able to choose to install PyMuPDF as an extra: `[pdf]`. I'll reopen it when finished.

@user2:
@author I got further confirmation we are good to go with `PyMuPDF` so I would actually ask you to keep it like this if the only concern is about the license.

@user1:
Thanks a lot for the discussion, I'm not an expert w.r.t. to licensing, and this is certainly helpful.
I also saw that this repo uses https://fossa.com/ for license scanning :)

@author:
@user2 Thank you for the approval.

I was pretty sure about my perspective because after what @user1 mentioned, I took some hours reading and investigating court cases in the US and BR regarding open-source licensing, dual licensing and multi licensing. Furthermore, I asked for advice from one of our co-founders, which is a lawyer authorized to work both in BR (OAB) and the US (BAR). Going further, I did a free consultancy with @user3, as he is a commercial mediator.

All the replies are condensed in the previous posts in an easy-to-understand explanation.

I was making changes to support both scenarios, as this conversation created a small concern about common user understanding of licensing. I'll revert them and reopen the PR.