Repository: opendataloader-project/opendataloader-pdf (2026-04-03T06:09:10Z)

Issue:
Words concatenation - space between words is removed
### When i try to parse my PDF file with "markdown-with-html" format, all the words are being concatenated, there is no space between words
The PDF i have used here is not scanned pdf, the texts from the PDF is selectable


#### Input - My PDF file contains the following text
 
Evolution Of Search
Lexical Search:
Traditional search relies on keyword matching, retrieving exact keywords or phrases entered by the user, but often yields
irrelevant results if the keywords are not precise, lacking context and intent understanding.
 
#### Output - This is the parsed markdown content which i got
EvolutionOfSearch LexicalSearch:
 
Traditionalsearchreliesonkeywordmatching,retrievingexactkeywordsorphrasesenteredbytheuser,butoftenyields irelevantresultsifthekeywordsarenotprecise,lackingcontextandintentunderstanding.


#### Code used
import opendataloader_pdf
 
opendataloader_pdf.convert(
        input_path=input_pdf_path,
        output_dir=output_dir,
        hybrid_url="http://localhost:5002",
        hybrid_fallback=True,
        format="markdown-with-html",
        image_output="off"
    )

 
opendataloader_pdf Version: 2.0.2
Java version : 17



Discussion:
Thanks for reporting this!

To investigate further, could you share the PDF file that produces this issue? 
Please send it to **open.dataloader@hancom.com** — we'll use it to reproduce and fix the problem.

In the meantime, could you try running with `--force-ocr` to see if it produces correct output?

```python
opendataloader_pdf.convert(
    input_path=input_pdf_path,
    output_dir=output_dir,
    hybrid_url="http://localhost:5002",
    hybrid_fallback=True,
    format="markdown-with-html",
    image_output="off",
    force_ocr=True
)
```

This would help us narrow down whether the issue is in the text extraction layer or elsewhere.

---

> 🤖 *AI Issue Fix* — status: **needs-info**
>
> Words are concatenated (spaces removed) in markdown-with-html output for a selectable-text PDF. Root cause likely in text extraction or hybrid mode text reconstruction. Awaiting sample PDF to reproduce and verify.
### Installation details and Backend started with ocr
opendataloader-pdf[hybrid]
opendataloader-pdf-hybrid --port 5002 --force-ocr

### Code used
import opendataloader_pdf

opendataloader_pdf.convert(
input_path=input_pdf_path,
output_dir=output_dir,
hybrid_url="http://localhost:5002/",
hybrid_fallback=True,
format="markdown-with-html",
image_output="off",
 hybrid="docling-fast" # tried with and without docling-fast, but still facing same issue
)

### Version
opendataloader_pdf Version: 2.0.2
Java version : 17

Shared the pdf to [open.dataloader@hancom.com](mailto:open.dataloader@hancom.com)

Thanks for reporting this. I've received the PDF and will look into it.
Thanks for sharing the PDF — we've identified and fixed the root cause.

**What was happening:** The text extraction layer was dropping explicit space characters from the PDF, then trying to re-insert them based on physical gaps between characters. Your PDF uses a font (Roboto) with tight kerning where characters have almost no physical gap, so the spaces were never re-inserted.

**The fix:** We now record where space characters existed in the original PDF and use that as an additional signal for space insertion, regardless of the physical gap size. PR: #383

**Before → After:**
```
Before: EvolutionOfSearch LexicalSearch:
After:  Evolution Of Search Lexical Search:

Before: Traditionalsearchreliesonkeywordmatching,retrievingexact...
After:  Traditional search relies on keyword matching, retrieving exact...
```

Note: some characters appear as substitutions (e.g., "aplications" instead of "applications") — this is a separate font glyph issue unrelated to spacing.

The fix will be available in the next release.

---

> 🤖 *AI Issue Fix* — status: **resolved**
>
> Root cause: `TextLineProcessor` dropped whitespace chunks and failed to restore spaces for tight-kerning fonts. Fixed by preserving whitespace positions as a spacing signal. PR #383.



Patch:
diff --git a/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextLineProcessor.java b/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextLineProcessor.java
index 86cfc9622..23ebbce3a 100644
--- a/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextLineProcessor.java
+++ b/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextLineProcessor.java
@@ -27,8 +27,11 @@
 import org.verapdf.wcag.algorithms.semanticalgorithms.utils.TextChunkUtils;
 
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.Comparator;
+import java.util.IdentityHashMap;
 import java.util.List;
+import java.util.Set;
 
 public class TextLineProcessor {
 
@@ -38,14 +41,28 @@ public class TextLineProcessor {
 
     public static List<IObject> processTextLines(List<IObject> contents) {
         List<IObject> newContents = new ArrayList<>();
+        // Track which TextChunk immediately follows a whitespace chunk in stream order,
+        // using reference identity so lookups are immune to TextChunk.equals() semantics.
+        // Stream order may differ from visual (leftX) order in rare PDFs, but whitespace
+        // chunks originate from the same PDF text operator as their adjacent text chunks,
+        // so stream-order adjacency is reliable for this signal.
+        Set<TextChunk> chunksAfterWhitespace = Collections.newSetFromMap(new IdentityHashMap<>());
         TextLine previousLine = new TextLine(new TextChunk(""));
         boolean isSeparateLine = false;
+        boolean pendingWhitespace = false;
         for (IObject content : contents) {
             if (content instanceof TextChunk) {
                 TextChunk textChunk = (TextChunk) content;
                 if (textChunk.isWhiteSpaceChunk() || textChunk.isEmpty()) {
+                    if (textChunk.isWhiteSpaceChunk()) {
+                        pendingWhitespace = true;
+                    }
                     continue;
                 }
+                if (pendingWhitespace) {
+                    chunksAfterWhitespace.add(textChunk);
+                    pendingWhitespace = false;
+                }
                 TextLine currentLine = new TextLine(textChunk);
                 double oneLineProbability = ChunksMergeUtils.countOneLineProbability(new SemanticTextNode(), previousLine, currentLine);
                 isSeparateLine |= (oneLineProbability < ONE_LINE_PROBABILITY) || previousLine.isHiddenText() != currentLine.isHiddenText();
@@ -62,6 +79,7 @@ public static List<IObject> processTextLines(List<IObject> contents) {
                     isSeparateLine = true;
                 }
                 newContents.add(content);
+                pendingWhitespace = false;
             }
         }
         for (int i = 0; i < newContents.size(); i++) {
@@ -70,14 +88,15 @@ public static List<IObject> processTextLines(List<IObject> contents) {
                 TextLine textLine = (TextLine) content;
                 textLine.getTextChunks().sort(TEXT_CHUNK_COMPARATOR);
                 double threshold = textLine.getFontSize() * TextChunkUtils.TEXT_LINE_SPACE_RATIO;
-                newContents.set(i, getTextLineWithSpaces(textLine, threshold));
+                newContents.set(i, getTextLineWithSpaces(textLine, threshold, chunksAfterWhitespace));
             }
         }
         linkTextLinesWithConnectedLineArtBullet(newContents);
         return newContents;
     }
 
-    private static TextLine getTextLineWithSpaces(TextLine textLine, double threshold) {
+    private static TextLine getTextLineWithSpaces(TextLine textLine, double threshold,
+                                                   Set<TextChunk> chunksAfterWhitespace) {
         List<TextChunk> textChunks = textLine.getTextChunks();
         TextChunk currentTextChunk = textChunks.get(0);
         double previousEnd = currentTextChunk.getBoundingBox().getRightX();
@@ -86,10 +105,14 @@ private static TextLine getTextLineWithSpaces(TextLine textLine, double threshol
         for (int i = 1; i < textChunks.size(); i++) {
             currentTextChunk = textChunks.get(i);
             double currentStart = currentTextChunk.getBoundingBox().getLeftX();
-            if (currentStart - previousEnd > threshold) {
+            boolean hasGap = currentStart - previousEnd > threshold;
+            boolean hadWhitespace = chunksAfterWhitespace.contains(currentTextChunk);
+            if (hasGap || hadWhitespace) {
+                double spaceLeft = Math.min(previousEnd, currentStart);
+                double spaceRight = Math.max(previousEnd, currentStart);
                 BoundingBox spaceBBox = new BoundingBox(currentTextChunk.getBoundingBox());
-                spaceBBox.setLeftX(previousEnd);
-                spaceBBox.setRightX(currentStart);
+                spaceBBox.setLeftX(spaceLeft);
+                spaceBBox.setRightX(spaceRight);
                 TextChunk spaceChunk = new TextChunk(spaceBBox, " ", textLine.getFontSize(), textLine.getBaseLine());
                 newLine.add(spaceChunk);
             }


Test patch:
diff --git a/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextLineProcessorTest.java b/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextLineProcessorTest.java
index 855aa827d..0cf9b4bbd 100644
--- a/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextLineProcessorTest.java
+++ b/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextLineProcessorTest.java
@@ -83,6 +83,42 @@ public void testProcessTextLinesSortsChunksByLeftX() {
             "Text line should start with 'Q:' (leftmost chunk), but got: " + textLine.getValue());
     }
 
+    /**
+     * Regression test for issue #358: when a whitespace chunk exists between two text chunks
+     * but the physical gap is smaller than the threshold, a space should still be inserted
+     * because the PDF explicitly contains a space character at that position.
+     */
+    @Test
+    public void testProcessTextLinesPreservesSpaceFromWhitespaceChunk() {
+        StaticContainers.setIsIgnoreCharactersWithoutUnicode(false);
+        StaticContainers.setIsDataLoader(true);
+        List<IObject> contents = new ArrayList<>();
+
+        // "Evolution" at x=46..85.5, font size 9.5 (threshold = 9.5*0.17 = 1.615)
+        TextChunk chunk1 = new TextChunk(new BoundingBox(0, 46.0, 300.0, 85.5, 310.0),
+            "Evolution", 9.5, 300.0);
+        // Whitespace chunk at x=85.5..87.9 — will be dropped by isWhiteSpaceChunk()
+        TextChunk spaceChunk = new TextChunk(new BoundingBox(0, 85.5, 300.0, 87.9, 310.0),
+            " ", 9.5, 300.0);
+        // "Of" at x=86.0..94.4 — gap from chunk1 = 0.5 < threshold 1.615, so no gap-based space
+        TextChunk chunk2 = new TextChunk(new BoundingBox(0, 86.0, 300.0, 94.4, 310.0),
+            "Of", 9.5, 300.0);
+
+        contents.add(chunk1);
+        contents.add(spaceChunk);
+        contents.add(chunk2);
+
+        contents = TextLineProcessor.processTextLines(contents);
+
+        Assertions.assertEquals(1, contents.size());
+        Assertions.assertTrue(contents.get(0) instanceof TextLine);
+
+        TextLine textLine = (TextLine) contents.get(0);
+        // Space must be preserved even though the physical gap is below threshold
+        Assertions.assertEquals("Evolution Of", textLine.getValue(),
+            "Space from whitespace chunk should be preserved, but got: " + textLine.getValue());
+    }
+
     /**
      * Regression test for issue #150: spaces should be inserted between sorted chunks
      * when there is a physical gap between them.