Repository: py-pdf/pypdf
Pull request: BUG: tolerate truncated files and no warning when jumping startxref
closes #2853

Problem statement:
Handling of incomplete EOF markers
I just stumbled upon a PDF sent by e-mail where the final `OF` of the EOF marker has been missing. Apparently, pdf.js and Evince are able to display it correctly, while *pypdf* fails.

## Environment

Which environment were you using when you encountered the problem?

```bash
$ python -m platform
Linux-5.14.21-150400.24.100-default-x86_64-with-glibc2.31

$ python -c "import pypdf;print(pypdf._debug_versions)"
pypdf==4.3.1, crypt_provider=('pycryptodome', '3.18.0'), PIL=10.0.0
```

Note: The version is the latest *main* code.

## Code + PDF

This is a minimal, complete example that shows the issue:

```python
from pypdf import PdfReader

reader = PdfReader('file.pdf')
```

For the file, just take any valid PDF file and remove the final `OF` characters of `%%EOF`.

## Traceback

This is the complete traceback I see:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stefan/temp/pdf/pypdf/_reader.py", line 142, in __init__
    self.read(stream)
  File "/home/stefan/temp/pdf/pypdf/_reader.py", line 566, in read
    self._find_eof_marker(stream)
  File "/home/stefan/temp/pdf/pypdf/_reader.py", line 658, in _find_eof_marker
    line = read_previous_line(stream)
  File "/home/stefan/temp/pdf/pypdf/_utils.py", line 279, in read_previous_line
    raise PdfStreamError(STREAM_TRUNCATED_PREMATURELY)
pypdf.errors.PdfStreamError: Stream has ended unexpectedly

```

Hints from issue discussion:
there is one extra point behind that I've detected : If the %%EOF is truncated/missing, pypdf searches further in the file. If the file is linearized or has been incrementally built, a full "section" may be ignored without warning.

Commit message: BUG: tolerate truncated files and no warning when jumping startxref

closes #2853

Patch to review:
diff --git a/pypdf/_reader.py b/pypdf/_reader.py
index 9948cbea3..8fc4573cb 100644
--- a/pypdf/_reader.py
+++ b/pypdf/_reader.py
@@ -649,7 +649,23 @@ def _find_eof_marker(self, stream: StreamType) -> None:
         """
         HEADER_SIZE = 8  # to parse whole file, Header is e.g. '%PDF-1.6'
         line = b""
+        first = True
         while line[:5] != b"%%EOF":
+            if line != b"" and first:
+                if any(
+                    line.strip().endswith(tr) for tr in (b"%%EO", b"%%E", b"%%", b"%")
+                ):
+                    # I consider the file has truncated and
+                    # I have enough confidence to carry on
+                    logger_warning("EOF marker seems truncated", __name__)
+                    break
+                first = False
+            if b"startxref" in line:
+                logger_warning(
+                    "CAUTION : startxref found while searching for %%EOF\n"
+                    "This could the file is truncated and mean some data will not be read",
+                    __name__,
+                )
             if stream.tell() < HEADER_SIZE:
                 if self.strict:
                     raise PdfReadError("EOF marker not found")
diff --git a/tests/test_reader.py b/tests/test_reader.py
index 99555cd22..e43719a35 100644
--- a/tests/test_reader.py
+++ b/tests/test_reader.py
@@ -1617,3 +1617,27 @@ def test_iss2817():
         reader.pages[0]["/Annots"][0].get_object()["/Contents"]
         == "A\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0 B"
     )
+
+
+@pytest.mark.enable_socket()
+def test_truncated_files(caplog):
+    """Cf #2853"""
+    url = "https://github.com/user-attachments/files/16796095/f5471sm-2.pdf"
+    name = "iss2780.pdf"  # reused
+    b = get_data_from_url(url, name=name)
+    reader = PdfReader(BytesIO(b))
+    assert caplog.text == ""
+    # remove \n at end of file : invisible
+    reader = PdfReader(BytesIO(b[:-1]))
+    assert caplog.text == ""
+    # truncate but still detectable
+    for i in range(-2, -6, -1):
+        caplog.clear()
+        reader = PdfReader(BytesIO(b[:i]))
+        assert "EOF marker seems truncated" in caplog.text
+        assert reader._startxref == 100993
+    # remove completely EOF : we will not read last section
+    caplog.clear()
+    reader = PdfReader(BytesIO(b[:-6]))
+    assert "CAUTION : startxref found while searching for %%EOF" in caplog.text
+    assert reader._startxref < 100993


Code review comments:
diff_hunk: @@ -649,7 +649,23 @@ def _find_eof_marker(self, stream: StreamType) -> None:
         """
         HEADER_SIZE = 8  # to parse whole file, Header is e.g. '%PDF-1.6'
         line = b""
+        first = True
         while line[:5] != b"%%EOF":
+            if line != b"" and first:
+                if any(
+                    line.strip().endswith(tr) for tr in (b"%%EO", b"%%E", b"%%", b"%")
+                ):
+                    # I consider the file has truncated and
+                    # I have enough confidence to carry on
path: pypdf/_reader.py
text: @user1:
```suggestion
                    # having enough confidence to carry on.
```

diff_hunk: @@ -649,7 +649,23 @@ def _find_eof_marker(self, stream: StreamType) -> None:
         """
         HEADER_SIZE = 8  # to parse whole file, Header is e.g. '%PDF-1.6'
         line = b""
+        first = True
         while line[:5] != b"%%EOF":
+            if line != b"" and first:
+                if any(
+                    line.strip().endswith(tr) for tr in (b"%%EO", b"%%E", b"%%", b"%")
+                ):
+                    # I consider the file has truncated and
path: pypdf/_reader.py
text: @user1:
```suggestion
                    # Consider the file as truncated while
```

diff_hunk: @@ -649,7 +649,23 @@ def _find_eof_marker(self, stream: StreamType) -> None:
         """
         HEADER_SIZE = 8  # to parse whole file, Header is e.g. '%PDF-1.6'
         line = b""
+        first = True
         while line[:5] != b"%%EOF":
+            if line != b"" and first:
+                if any(
+                    line.strip().endswith(tr) for tr in (b"%%EO", b"%%E", b"%%", b"%")
+                ):
+                    # I consider the file has truncated and
+                    # I have enough confidence to carry on
+                    logger_warning("EOF marker seems truncated", __name__)
+                    break
+                first = False
+            if b"startxref" in line:
+                logger_warning(
+                    "CAUTION : startxref found while searching for %%EOF\n"
+                    "This could the file is truncated and mean some data will not be read",
path: pypdf/_reader.py
text: @user1:
```suggestion
                    "The file might be truncated and some data might not be read.",
```

diff_hunk: @@ -649,7 +649,23 @@ def _find_eof_marker(self, stream: StreamType) -> None:
         """
         HEADER_SIZE = 8  # to parse whole file, Header is e.g. '%PDF-1.6'
         line = b""
+        first = True
         while line[:5] != b"%%EOF":
+            if line != b"" and first:
+                if any(
+                    line.strip().endswith(tr) for tr in (b"%%EO", b"%%E", b"%%", b"%")
+                ):
+                    # I consider the file has truncated and
+                    # I have enough confidence to carry on
+                    logger_warning("EOF marker seems truncated", __name__)
+                    break
+                first = False
+            if b"startxref" in line:
+                logger_warning(
+                    "CAUTION : startxref found while searching for %%EOF\n"
path: pypdf/_reader.py
text: @user1:
```suggestion
                    "CAUTION: startxref found while searching for %%EOF. "
```