Repository: deepset-ai/haystack

Issue:
Explicit Encoding Handling for PDF Parsing
**Is your feature request related to a problem? Please describe.**
PDFs with non-UTF-8 encoding (e.g., ANSI, cp1252) are not indexed correctly in Haystack’s document pipeline. This results in missing text, corrupted characters (e.g., (cid:xx) artifacts), or unreadable embeddings. I request an enhancement to support automatic encoding detection and conversion in the Haystack PDF parsing component and explicit encoding selection options.


**Describe the solution you'd like**
Enhance the PDF parsing components by:
Auto-detecting encoding before indexing using libraries like chardet or cchardet.
Providing an explicit encoding parameter (e.g., encoding="utf-8" or encoding="auto") in PDFToTextConverter, PDFPlumberConverter, and PyMuPDFConverter.
Converting extracted text to UTF-8 before it is passed to the embedding pipeline.





Discussion:
From my research into how both PyPDF and PDFMiner handle text extraction for #8491, I’ve found that the presence of `cid:x` values often signals that the PDF itself is missing necessary character-to-Unicode mappings. This tends to happen when the fonts or character encodings in the PDF are incomplete or poorly defined. In this case the issue might not be with the extraction itself but with the underlying PDF structure.

For PDFMiner, the `cid:x` error usually happens because it defaults to showing raw character IDs when it cannot map characters to Unicode. This happens when the PDF uses fonts with no corresponding Unicode mapping (a common occurrence with custom or embedded fonts). If you open the PDF in a viewer, try copying the text and pasting it into a text editor. If it results in gibberish, that usually confirms that the issue lies within the PDF's encoding itself.

Similarly, with PyPDF, when encountering issues with certain fonts not being extracted properly, it could be due to the absence of a translation table (such as the `/ToUnicode` field for embedded fonts), which makes it difficult to decode the characters properly.

So, while we may be unable to correct the poorly extracted text, we can do a post-conversion cleanup to ensure the fallback behavior of the converters producing the raw character IDs is removed, preventing these artifacts from affecting downstream tasks (although this could result in some data loss depending on how much it fallbacks). One approach could be to use the DocumentCleaner component's `remove_substrings` or `remove_regex` to clean the unwanted patterns.
More context about the `cid:x` in this stack overflow post https://stackoverflow.com/questions/66656067/replace-cidnumber-with-chars-using-python-when-extracting-text-from-pdf-fil

We load our PDF files using byte loading (e.g `open(stream, "rb") as fh:`) which means I don't believe the encoding parameter applies in this case. 

Looking more into the stack overflow link it appears that the issue is with the PDF itself as Ibux mentions. E.g. The PDF could be using a custom font that the extractor doesn't know how to convert since that information is missing in the PDF. Quote from one of the comments in the stack overflow post
```
the PDF doesn't contain the information required for regular text extraction, so normal text extractors will fail
```
Also relevant is PDFMiner's documentation for [cid:x](https://pdfminersix.readthedocs.io/en/latest/faq.html#why-are-there-cid-x-values-in-the-textual-output). For PyPDF issues, it seems like the project recommends people to create an issue if you can copy from the PDF into a destination and have no issues with that (as this would likely mean that the issue is not with the PDF and instead PyPDF). https://github.com/py-pdf/pypdf/issues/2295
I researched to understand this problem in more detail. Here are my notes/learnings.

#### Why do we see CID values? 

The encoding represented by (cid:xx) is not a standard encoding like UTF-8 or ASCII. It's part of a specific PDF font encoding system called "CID" (Character Identifier).

PDFs use a complex system for text representation. Characters are stored as numeric codes (CIDs - Character Identifiers) that map to specific glyphs in fonts. A glyph is a visual representation of a character. When text extraction tools can't properly resolve these mappings, they may output the raw CID values instead of the actual characters.

When we see text like (cid:21) in extracted content, it means the tool recognized this was character #21 in some CID font, but couldn't determine what actual character that corresponds to in standard Unicode.

There isn't a single universal "CID encoding", each PDF font can have its own unique CID-to-character mapping. This mapping depends on the specific font used in the document. Mapping them to a character requires accessing the font's character mapping information embedded in the PDF. 

PDF files may not include this character mapping information between CIDs and Unicode characters. The "ToUnicode" mapping table is essential for text extraction, and when it's missing or incomplete, extraction tools fall back to showing CID values.

There isn't a single universal "CID encoding" - each PDF font can have its own unique CID-to-character mapping.



Patch:
diff --git a/haystack/components/converters/pdfminer.py b/haystack/components/converters/pdfminer.py
index 6c8fc6cdc7..1ba6137b50 100644
--- a/haystack/components/converters/pdfminer.py
+++ b/haystack/components/converters/pdfminer.py
@@ -4,6 +4,7 @@
 
 import io
 import os
+import re
 from pathlib import Path
 from typing import Any, Dict, Iterator, List, Optional, Union
 
@@ -18,6 +19,8 @@
 
 logger = logging.getLogger(__name__)
 
+CID_PATTERN = r"\(cid:\d+\)"  # regex pattern to detect CID characters
+
 
 @component
 class PDFMinerToDocument:
@@ -97,6 +100,7 @@ def __init__(  # pylint: disable=too-many-positional-arguments
             all_texts=all_texts,
         )
         self.store_full_path = store_full_path
+        self.cid_pattern = re.compile(CID_PATTERN)
 
     @staticmethod
     def _converter(lt_page_objs: Iterator) -> str:
@@ -126,6 +130,32 @@ def _converter(lt_page_objs: Iterator) -> str:
 
         return delimited_pages
 
+    def detect_undecoded_cid_characters(self, text: str) -> Dict[str, Any]:
+        """
+        Look for character sequences of CID, i.e.: characters that haven't been properly decoded from their CID format.
+
+        This is useful to detect if the text extractor is not able to extract the text correctly, e.g. if the PDF uses
+        non-standard fonts.
+
+        A PDF font may include a ToUnicode map (mapping from character code to Unicode) to support operations like
+        searching strings or copy & paste in a PDF viewer. This map immediately provides the mapping the text extractor
+        needs. If that map is not available the text extractor cannot decode the CID characters and will return them
+        as is.
+
+        see: https://pdfminersix.readthedocs.io/en/latest/faq.html#why-are-there-cid-x-values-in-the-textual-output
+
+        :param: text: The text to check for undecoded CID characters
+        :returns:
+            A dictionary containing detection results
+        """
+
+        matches = re.findall(self.cid_pattern, text)
+        total_chars = len(text)
+        cid_chars = sum(len(match) for match in matches)
+        percentage = (cid_chars / total_chars * 100) if total_chars > 0 else 0
+
+        return {"total_chars": total_chars, "cid_chars": cid_chars, "percentage": round(percentage, 2)}
+
     @component.output_types(documents=List[Document])
     def run(
         self,
@@ -178,6 +208,19 @@ def run(
 
             if not self.store_full_path and (file_path := bytestream.meta.get("file_path")):
                 merged_metadata["file_path"] = os.path.basename(file_path)
+
+            analysis = self.detect_undecoded_cid_characters(text)
+
+            if analysis["percentage"] > 0:
+                logger.warning(
+                    "Detected {cid_chars} undecoded CID characters in {total_chars} characters"
+                    " ({percentage}%) in {source}.",
+                    cid_chars=analysis["cid_chars"],
+                    total_chars=analysis["total_chars"],
+                    percentage=analysis["percentage"],
+                    source=source,
+                )
+
             document = Document(content=text, meta=merged_metadata)
             documents.append(document)
 
diff --git a/releasenotes/notes/adding-CID-detection-PDFMinerToDocument-0195a929d64cd502.yaml b/releasenotes/notes/adding-CID-detection-PDFMinerToDocument-0195a929d64cd502.yaml
new file mode 100644
index 0000000000..2855d16bc9
--- /dev/null
+++ b/releasenotes/notes/adding-CID-detection-PDFMinerToDocument-0195a929d64cd502.yaml
@@ -0,0 +1,5 @@
+---
+enhancements:
+  - |
+    Added `PDFMinerToDocument` functionality to detect and report undecoded CID characters in PDF text extraction, helping users identify potential
+    text extraction quality issues when processing PDFs with non-standard fonts.


Test patch:
diff --git a/test/components/converters/test_pdfminer_to_document.py b/test/components/converters/test_pdfminer_to_document.py
index 4691a2a1a2..00d751efa4 100644
--- a/test/components/converters/test_pdfminer_to_document.py
+++ b/test/components/converters/test_pdfminer_to_document.py
@@ -2,6 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 import logging
+from unittest.mock import patch
 
 import pytest
 
@@ -185,3 +186,56 @@ def test_run_detect_paragraphs_to_be_used_in_split_passage(self, test_files_path
             "structure, allowing structure to emerge according to the \nneeds of the users.[1] \n\n"
         )
         assert docs["documents"][6].content == expected
+
+    def test_detect_undecoded_cid_characters(self):
+        """
+        Test if the component correctly detects and reports undecoded CID characters in text.
+        """
+        converter = PDFMinerToDocument()
+
+        # Test text with no CID characters
+        text = "This is a normal text without any CID characters."
+        result = converter.detect_undecoded_cid_characters(text)
+        assert result["total_chars"] == len(text)
+        assert result["cid_chars"] == 0
+        assert result["percentage"] == 0
+
+        # Test text with CID characters
+        text = "Some text with (cid:123) and (cid:456) characters"
+        result = converter.detect_undecoded_cid_characters(text)
+        assert result["total_chars"] == len(text)
+        assert result["cid_chars"] == len("(cid:123)") + len("(cid:456)")  # 18 characters total
+        assert result["percentage"] == round((18 / len(text)) * 100, 2)
+
+        # Test text with multiple consecutive CID characters
+        text = "(cid:123)(cid:456)(cid:789)"
+        result = converter.detect_undecoded_cid_characters(text)
+        assert result["total_chars"] == len(text)
+        assert result["cid_chars"] == len("(cid:123)(cid:456)(cid:789)")
+        assert result["percentage"] == 100.0
+
+        # Test empty text
+        text = ""
+        result = converter.detect_undecoded_cid_characters(text)
+        assert result["total_chars"] == 0
+        assert result["cid_chars"] == 0
+        assert result["percentage"] == 0
+
+    def test_pdfminer_logs_warning_for_cid_characters(self, caplog, monkeypatch):
+        """
+        Test if the component correctly logs a warning when undecoded CID characters are detected.
+        """
+        test_data = ByteStream(data=b"fake", meta={"file_path": "test.pdf"})
+
+        def mock_converter(*args, **kwargs):
+            return "This is text with (cid:123) and (cid:456) characters"
+
+        def mock_extract_pages(*args, **kwargs):
+            return ["mocked page"]
+
+        with patch("haystack.components.converters.pdfminer.extract_pages", side_effect=mock_extract_pages):
+            with patch.object(PDFMinerToDocument, "_converter", side_effect=mock_converter):
+                with caplog.at_level(logging.WARNING):
+                    converter = PDFMinerToDocument()
+                    converter.run(sources=[test_data])
+                    assert "Detected 18 undecoded CID characters in 52 characters (34.62%)" in caplog.text