Repository: py-pdf/pypdf
Pull request: BUG: Prevent updating page contents after merging page (stamping/watermarking)
add also over param in merge_page

closes #1951
closes #1953 

Problem statement:
Huge file produced when merging (watermarking/stamping)
## Environment

Which environment were you using when you encountered the problem?

```bash
$ python -m platform
macOS-13.4.1-arm64-arm-64bit

$ python -c "import pypdf;print(pypdf.__version__)"
3.12.0
```

## Explanation

Good day!

1) Sometimes I need to _merge (append)_ background to the **text pdf** file.
Its not simple, because `pypdf.PdfReader(file).pages[0].merge_page(page)` append `page` under the `PdfReader(file).pages[0]`.
Its good for the draft information, but is not good idea for the background image.
Its need to create one and one background page `bkg = pypdf.PdfReader(f_bkg).pages[0]`, and then merge it with next text pdf page.

I think its a good idea, to append method **.bg** : `pypdf.PdfReader(file).pages[0].bg(background_page)` to append `background` **down** the text page `PdfReader(file).pages[0]`.

2) The problem with multipage pdf file.
I try to merge background file `bg.pdf` with text file `source.pdf` to the `out_with_bg.pdf`, Im getting very huge **132 Mb** file:
```
bg.pdf, 761 kb (contain image, simple page)
source.pdf, 680 kb (contain text, 171 pages)
out_with_bg.pdf, 132 Mb !!!
```

but when I use `pdftk`,  Im getting only **1,5 Mb** file:
```bash
 pdftk out.pdf background bg.pdf output out-BG_pdftk.pdf
out-BG_pdftk.pdf, 1.5 Mb !!!
```

Its **too slow** process with **huge output file**. 

I think its a good idea, to add methods **`.merge`** and **`.bg`** to the whole pdf obj: 

```python
PdfReader(file).merge(page)
PdfReader(file).bg(page)
```

## Code Example

How would your feature be used? (Remove this if it is not applicable.)

```python
import pypdf

with open('source.pdf', 'rb') as f_pdf, \
        open('bg.pdf', 'rb') as f_bkg:
    pdf = pypdf.PdfReader(f_pdf) # source text pdf file
    out = pypdf.PdfWriter()
    for p, page in enumerate(pdf.pages):
        bkg = pypdf.PdfReader(f_bkg).pages[0] # background pdf file with image
        bkg.merge_page(page)
        out.add_page(bkg)
    with open('out_with_bg.pdf', 'wb') as f:
        out.write(f)
```


Change in behavior from 3.10.0 to 3.11.0
The minimal example below works fine with pypdf 3.10.0, but fails beginning with 3.11.0.  The expectation is that the code would work in 3.11.0, or else a deprecation message would be displayed.

## Environment

Python 3.11.4

## Code + PDF
No PDF is generated due to the error.

```python
from io import BytesIO
from pathlib import Path

from pypdf import PdfReader, PdfWriter, Transformation

LETTER_W = 612
LETTER_H = 792


def create_song_sheet(input_file: Path, output_file: Path) -> None:
    pdf_pages = []
    reader = PdfReader(input_file)
    pdf_page_count = len(reader.pages)

    for page in reader.pages:
        new_letter_page_writer = PdfWriter()

        ## scale the PDF to fit a 8.5 x 11 inch US Letter page
        ## but preserve the aspect ratio by setting the height
        ## to 11, scaling the width accordingly, and centering
        ## the result on the page
        new_letter_page = new_letter_page_writer.add_blank_page(
            width=LETTER_W, height=LETTER_H
        )

        ## set the mediabox width and height so that things don't
        ## get chopped off when we center the scaled PDF
        source_width = page.mediabox.width
        source_height = page.mediabox.height
        source_aspect = source_width / source_height
        new_width = LETTER_H * source_aspect
        width_offset = (LETTER_W - new_width) / 2

        new_letter_page.merge_transformed_page(
            page, Transformation((1, 0, 0, 1, width_offset, 0))
        )

        tmp_pdf = BytesIO()
        new_letter_page_writer.write(tmp_pdf)

        new_letter_page_writer.close()
        pdf_pages.append(
            {"file": tmp_pdf, "name": input_file, "pagecount": pdf_page_count}
        )   

    merger = PdfWriter()
    for pdf_page in pdf_pages:
        ## ERROR OCCURS HERE ON PROCESSING THE FIRST PAGE ##
        merger.append(pdf_page["file"])

    merger.write(output_file)
    merger.close()


create_song_sheet(
    Path("/home/moose/Github/py-pdf/pypdf/resources/crazyones.pdf"),
    Path("out-1953.pdf"),
)
```

## Traceback

Here is the traceback from the file above
```
Traceback (most recent call last):
  File "/home/moose/Github/py-pdf/pypdf/issue-1953.py", line 55, in <module>
    create_song_sheet(
  File "/home/moose/Github/py-pdf/pypdf/issue-1953.py", line 49, in create_song_sheet
    merger.append(pdf_page["file"])
  File "/home/moose/Github/py-pdf/pypdf/pypdf/_writer.py", line 2768, in append
    self.merge(
  File "/home/moose/Github/py-pdf/pypdf/pypdf/_utils.py", line 432, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/moose/Github/py-pdf/pypdf/pypdf/_writer.py", line 2849, in merge
    srcpages[pg.indirect_reference.idnum] = self.add_page(
                                            ^^^^^^^^^^^^^^
  File "/home/moose/Github/py-pdf/pypdf/pypdf/_writer.py", line 389, in add_page
    return self._add_page(page, list.append, excluded_keys)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/moose/Github/py-pdf/pypdf/pypdf/_writer.py", line 322, in _add_page
    page = cast("PageObject", page_org.clone(self, False, excluded_keys))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/moose/Github/py-pdf/pypdf/pypdf/generic/_data_structures.py", line 186, in clone
    d__._clone(self, pdf_dest, force_duplicate, ignore_fields)
  File "/home/moose/Github/py-pdf/pypdf/pypdf/generic/_data_structures.py", line 286, in _clone
    v.clone(pdf_dest, force_duplicate, ignore_fields)
  File "/home/moose/Github/py-pdf/pypdf/pypdf/generic/_base.py", line 294, in clone
    obj.clone(pdf_dest, force_duplicate, ignore_fields)
  File "/home/moose/Github/py-pdf/pypdf/pypdf/generic/_data_structures.py", line 102, in clone
    arr.append(dup.indirect_reference)
               ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DecodedStreamObject' object has no attribute 'indirect_reference'
```

Hints from issue discussion:
@ucomru Do you have example files which we could use? I guess your real files are private, but maybe you can re-create one that is not?

As a side-note: I've added this kind of use-case to https://github.com/py-pdf/benchmarks#watermarking-speed already and noticed that pdfrw is a lot faster + produces smaller results. See also https://github.com/py-pdf/benchmarks/issues/7

It's an issue I am aware of, but of which I'm uncertain how to tackle it.
> @ucomru Do you have example files which we could use? I guess your real files are private, but maybe you can re-create one that is not?
> 
> As a side-note: I've added this kind of use-case to https://github.com/py-pdf/benchmarks#watermarking-speed already and noticed that pdfrw is a lot faster + produces smaller results. See also [py-pdf/benchmarks#7](https://github.com/py-pdf/benchmarks/issues/7)
> 
> It's an issue I am aware of, but of which I'm uncertain how to tackle it.

Thanks for answer!!!

I ran 2 tests `pypdf_test.py` and `pdftk`:

```bash
❯ pypdf_test.py
Total time: 1.640826940536499 sec.

❯ time pdftk source.pdf background bg.pdf output out-BG_pdftk.pdf
pdftk source.pdf background bg.pdf output out-BG_pdftk.pdf  0.37s user 0.07s system 139% cpu 0.317 total

> exa -l
.rw-r--r--@ 721k tm  7 Jul 22:57 bg.pdf
.rw-r--r--  1.0M tm  7 Jul 23:10 out-BG_pdftk.pdf
.rw-r--r--   87M tm  7 Jul 23:00 out_with_bg.pdf
.rw-r--r--@  527 tm  7 Jul 22:59 pypdf_test.py
.rw-r--r--@ 262k tm  7 Jul 22:52 source.pdf
```

I think it's not bad time, but (as I think) if append new method - `.merge_page_bg`, its will be faster.
I don't now deep pdf structure, may be Im wrong... Word docx contains header and footer.
May be pdf contain so?..

**pypdf_test.py**:
```python
import time

import pypdf # pip install pypdf

t = time.time()

with open('source.pdf', 'rb') as f_pdf, \
        open('bg.pdf', 'rb') as f_bkg:
    pdf = pypdf.PdfReader(f_pdf) # source text pdf file
    out = pypdf.PdfWriter()
    for p, page in enumerate(pdf.pages):
        bkg = pypdf.PdfReader(f_bkg).pages[0] # background pdf file with image
        bkg.merge_page(page)
        out.add_page(bkg)
    with open('out_with_bg.pdf', 'wb') as f:
        out.write(f)

print(f'Total time: {time.time() - t} sec.')
```

[source.pdf](https://github.com/py-pdf/pypdf/files/11985888/source.pdf) -- 262 k
[bg.pdf](https://github.com/py-pdf/pypdf/files/11985889/bg.pdf) -- 721 k
[out-BG_pdftk.pdf](https://github.com/py-pdf/pypdf/files/11985890/out-BG_pdftk.pdf) -- 1.0 M
[out_with_bg.pdf](https://drive.google.com/file/d/1UDPRzDi6pwuDnSB81dxwfPr96dr9UekI/view) -- 87 M
In your process you are adding everytime a new page based on a new copy of the bg.pdf, therefore the image is multiplied many times explaining the hudge size. This is my proposal:
```python
import pypdf

r = pypdf.PdfReader("bg.pdf")
w = pypdf.PdfWriter(clone_from="source.pdf")

for p in w.pages:
    p.merge_transformed_page(r.pages[0], pypdf.Transformation(), over=False)
    p[pypdf.generic.NameObject("/Contents")] = w._add_object(p.get_contents())

w.write("out.pdf")
```
The current `merge_page` does not offer the parameter over, this is why I've used `merge_transformed_page`.(@MartinThoma, I think It would be great the add the parameter to merge_page, but to be consistent it should precede the existing expand. are you OK with this)

with this approach, the image is added only once.

the replacement of the contents is because currently the Contents is an array of ContentStreams which are not IndirectObjects.
it is also to be noted that this will leave some object unused in the file increasnig the size abnormally :  : I will propose a PR to not have to do that in the future. 

Finally I think we should update the documentation
> The current merge_page does not offer the parameter over, this is why I've used merge_transformed_page.(@MartinThoma, I think It would be great the add the parameter to merge_page, but to be consistent it should precede the existing expand. are you OK with this)

Do you mean something like this:

```
def merge_page(self, page2: "PageObject", expand: bool = False, backgorund: PageObject) -> None:
    ...
```

?
> > The current merge_page does not offer the parameter over, this is why I've used merge_transformed_page.(@MartinThoma, I think It would be great the add the parameter to merge_page, but to be consistent it should precede the existing expand. are you OK with this)
> 
> Do you mean something like this:
> 
> ```
> def merge_page(self, page2: "PageObject", expand: bool = False, backgorund: PageObject) -> None:
>     ...
> ```
> 
> ?

no : page2 is already the background it would be like:

`def merge_page(self, page2: "PageObject", over: bool = True, expand: bool = False) -> None:`

Thanks for reporting the bug!

I've adjusted the example to make it smaller. 
I've retetested with #1952 (still pending mergin)  and the issue seems solved with it

Commit message: Update docs/user/add-watermark.md

Co-authored-by: Martin Thoma <info@martin-thoma.de>

Patch to review:
diff --git a/docs/user/add-watermark.md b/docs/user/add-watermark.md
index 4e1190156..7fd8e0d44 100644
--- a/docs/user/add-watermark.md
+++ b/docs/user/add-watermark.md
@@ -4,80 +4,54 @@ Adding stamps or watermarks are two common ways to manipulate PDF files.
 A stamp is adding something on top of the document, a watermark is in the
 background of the document.
 
-## Stamp (Overlay)
+## Stamp (Overlay) / Watermark(Undelay)
 
-Using the ``Transformation()`` class, one can translate, rotate, scale, etc. the stamp before merging it to the content page.
+The process of stamping and watermarking is the same, you just need to set `over` parameter to `True` for stamping and `False` for watermarking.
 
+You can use `merge_page()` if you don't need to transform the stamp:
 ```python
-from pathlib import Path
-from typing import Union, Literal, List
-
 from pypdf import PdfWriter, PdfReader
 
+stamp = PdfReader("bg.pdf").pages[0]
+writer = PdfWriter(clone_from="source.pdf")
+for page in writer.pages:
+    page.merge_page(stamp, over=False)  # here set to False for watermarking
 
-def stamp(
-    content_pdf: Path,
-    stamp_pdf: Path,
-    pdf_result: Path,
-    page_indices: Union[Literal["ALL"], List[int]] = "ALL",
-):
-    stamp_page = PdfReader(stamp_pdf).pages[0]
-
-    writer = PdfWriter()
-
-    reader = PdfReader(content_pdf)
-    if page_indices == "ALL":
-        page_indices = list(range(0, len(reader.pages)))
-    for index in page_indices:
-        content_page = reader.pages[index]
-        content_page.merge_transformed_page(
-            stamp_page,
-            Transformation(),
-        )
-        writer.add_page(content_page)
-
-    with open(pdf_result, "wb") as fp:
-        writer.write(fp)
+writer.write("out.pdf")
 ```
 
-![stamp.png](stamp.png)
-
-## Watermark (Underlay)
-
-To merge the watermark *under* the content, use the argument ``over=False`` of the method ``merge_transformed_page()``.
-
-Once again, watermark size and position (and more) can be customized using the ``Transformation()`` class.
+Else use `merge_transformed_page()` with Transformation() if you need to translate, rotate, scale, etc. the stamp before merging it to the content page.
 
 ```python
 from pathlib import Path
 from typing import Union, Literal, List
 
-from pypdf import PdfWriter, PdfReader, Transformation
+from pypdf import PdfWriter, PdfReader
 
 
-def watermark(
+def stamp(
     content_pdf: Path,
     stamp_pdf: Path,
     pdf_result: Path,
     page_indices: Union[Literal["ALL"], List[int]] = "ALL",
 ):
-    reader = PdfReader(content_pdf)
-    if page_indices == "ALL":
-        page_indices = range(len(reader.pages))
+    stamp_page = PdfReader(stamp_pdf).pages[0]
 
     writer = PdfWriter()
-    watermark_page = PdfReader(stamp_pdf).pages[0]
-    for index in page_indices:
-        content_page = reader.pages[index]
+    # page_indices can be a List(array) of page, tuples are for range definition
+    writer.append(content, pages=None if page_indices == "ALL" else page_indices)
+
+    for content_page in writer.pages:
         content_page.merge_transformed_page(
-            watermark_page,
-            Transformation(),
-            over=False,
+            stamp_page,
+            Transformation().scale(0.5),
         )
-        writer.add_page(content_page)
 
-    with open(pdf_result, "wb") as fp:
-        writer.write(fp)
+    writer.write(pdf_result)
 ```
 
+example of stamp:
+![stamp.png](stamp.png)
+
+example of watermark:
 ![watermark.png](watermark.png)
diff --git a/pypdf/_page.py b/pypdf/_page.py
index a9885f84b..868270749 100644
--- a/pypdf/_page.py
+++ b/pypdf/_page.py
@@ -938,6 +938,11 @@ def replace_contents(
                     self._objects[o.indirect_reference.idnum - 1] = NullObject()  # type: ignore
                 except AttributeError:
                     pass
+
+        if isinstance(content, ArrayObject):
+            for i in range(len(content)):
+                content[i] = self.indirect_reference.pdf._add_object(content[i])
+
         if content is None:
             if PG.CONTENTS not in self:
                 return
@@ -972,7 +977,9 @@ def replace_contents(
                 # this will be fixed with the _add_object
                 self[NameObject(PG.CONTENTS)] = content
 
-    def merge_page(self, page2: "PageObject", expand: bool = False) -> None:
+    def merge_page(
+        self, page2: "PageObject", over: bool = True, expand: bool = False
+    ) -> None:
         """
         Merge the content streams of two pages into one.
 
@@ -985,10 +992,11 @@ def merge_page(self, page2: "PageObject", expand: bool = False) -> None:
         Args:
             page2: The page to be merged into this one. Should be
                 an instance of :class:`PageObject<PageObject>`.
+            over: set the page2 content over page1 if True(default) else under
             expand: If true, the current page dimensions will be
                 expanded to accommodate the dimensions of the page to be merged.
         """
-        self._merge_page(page2, expand=expand)
+        self._merge_page(page2, over=over, expand=expand)
 
     def mergePage(self, page2: "PageObject") -> None:  # deprecated
         """
diff --git a/tests/test_page.py b/tests/test_page.py
index 2f7d2d7b7..c9a55d39b 100644
--- a/tests/test_page.py
+++ b/tests/test_page.py
@@ -1237,12 +1237,15 @@ def create_stamp_pdf() -> BytesIO:
     writer.append(SAMPLE_ROOT / "009-pdflatex-geotopo/GeoTopo.pdf", [1])
     nb1 = len(writer._objects)
 
+    # 1 page only is modified
     for page in writer.pages:
         page.merge_page(template_page)
-    assert len(writer._objects) == nb1 + 1  # font is added that's all
+    # font is added; +1 streamobjects + 1 ArrayObject
+    assert len(writer._objects) == nb1 + 1 + 2
     for page in writer.pages:
         page.compress_content_streams()
-    assert len(writer._objects) == nb1 + 1
+    # objects are recycled
+    assert len(writer._objects) == nb1 + 1 + 2
 
     contents = writer.pages[0]["/Contents"]
     writer.pages[0].replace_contents(None)
diff --git a/tests/test_writer.py b/tests/test_writer.py
index 7a11ab5f1..75842896d 100644
--- a/tests/test_writer.py
+++ b/tests/test_writer.py
@@ -624,7 +624,9 @@ def test_add_named_destination_sort_order(pdf_file_path):
     root = writer.get_named_dest_root()
 
     assert len(root) == 4
-    assert root[0] == "a", '"a" was not inserted before "b" in the named destination root'
+    assert (
+        root[0] == "a"
+    ), '"a" was not inserted before "b" in the named destination root'
     assert root[2] == "b"
 
     # write "output" to pypdf-output.pdf
@@ -1478,3 +1480,22 @@ def test_empty_objects_before_cloning():
         {x: 1 for x, y in reader.xref_objStm.values()}
     )  # to remove object streams
     assert len(writer._objects) == nb_obj_reader
+
+
+@pytest.mark.enable_socket()
+def test_watermark():
+    url = "https://github.com/py-pdf/pypdf/files/11985889/bg.pdf"
+    name = "bgwatermark.pdf"
+    reader = PdfReader(BytesIO(get_pdf_from_url(url, name=name)))
+    url = "https://github.com/py-pdf/pypdf/files/11985888/source.pdf"
+    name = "srcwatermark.pdf"
+    writer = PdfWriter(clone_from=BytesIO(get_pdf_from_url(url, name=name)))
+    for p in writer.pages:
+        p.merge_page(reader.pages[0], over=False)
+
+    assert isinstance(p["/Contents"], ArrayObject)
+    assert isinstance(p["/Contents"][0], IndirectObject)
+
+    b = BytesIO()
+    writer.write(b)
+    assert len(b.getvalue()) < 2.1 * 1024 * 1024


Code review comments:
diff_hunk: @@ -972,7 +977,9 @@ def replace_contents(
                 # this will be fixed with the _add_object
                 self[NameObject(PG.CONTENTS)] = content
 
-    def merge_page(self, page2: "PageObject", expand: bool = False) -> None:
+    def merge_page(
+        self, page2: "PageObject", over: bool = True, expand: bool = False
path: pypdf/_page.py
text: @user1:
```suggestion
        self, page2: "PageObject", expand: bool = False,  over: bool = True
```