Repository: py-pdf/pypdf
Pull request: ENH: add get_pages_from_field 
closes #2452 
requires #2493 

Problem statement:
Getting the page number of a field
## Explanation

I am trying to update each field of a PDF file with some data but I don't know which PageObject to use after getting the list of fields from the function PdfReader.get_field().

## Code Example

> How would your feature be used? (Remove this if it is not applicable.)

Maybe with a new function: 

```python
PdfReader.get_form_page(self, field: Field) -> tuple[int, PageObject]
```
Return:
- the page number
- the page object

Example:

```python
from pypdf import PdfReader

reader = PdfReader("form.pdf")
fields = reader.get_fields()

page_number, page = reader.get_form_page(fields[0])
```

Hints from issue discussion:
Thanks for the remark. [The forms documentation](https://pypdf.readthedocs.io/en/latest/user/forms.html) should probably be expanded to explain the relationship between AcroForm and Annotations better. I'm not an exper there myself, but here is what I understood:

## AcroForm and Annotation
`/AcroForm` is defined in the trailer and looks like this:

```
  /Fields [ 15 0 R 16 0 R 17 0 R ]
  /DR <<
    /Font <<
      /ZaDb 5 0 R
      /Helv 6 0 R
    >>
  >>
  /DA (/Helv 10 Tf 0 g)
  /NeedAppearances true
```

This form has 3 fields. The fields have corresponding annotations which are defined on the pages on which they appear. In my case:

```
15 0 obj
<<
  /Type /Annot
  /Rect [ 182.198 650.66 269.23 668.194 ]
  /Subtype /Widget
  /F 4
  /T <FEFF004E0061006D0065>
  /FT /Tx
  /Q 0
  /BS <<
    /W 1
    /S /S
  >>
  /MK <<
    /BC [ 1 0 0 ]
    /BG [ 1 1 1 ]
  >>
  /DA (/Helv 10 Tf 0 0 0 rg)
  /DV ()
  /V ()
>>
endobj

16 0 obj
<<
  /Type /Annot
  /Rect [ 183.582 623.163 195.537 640.697 ]
  /Subtype /Widget
  /F 4
  /T <FEFF0043006800650063006B>
  /FT /Btn
  /Q 0
  /BS <<
    /W 1
    /S /S
  >>
  /AP <<
    /N <<
      /Yes <<
      >>
    >>
  >>
  /MK <<
    /BC [ 1 0 0 ]
    /BG [ 1 1 1 ]
    /CA (4)
  >>
  /DA (/ZaDb 10 Tf 0 0 0 rg)
  /H /P
  /V /Off
  /AS /Off
>>
endobj

17 0 obj
<<
  /Type /Annot
  /Rect [ 153.694 598.703 189.235 613.2 ]
  /Subtype /Widget
  /F 4
  /T (Submit)
  /FT /Btn
  /Ff 65540
  /H /P
  /BS <<
    /W 1
    /S /S
  >>
  /MK <<
    /BC [ 1 0 0 ]
  >>
  /A <<
    /S /SubmitForm
    /F <<
      /FS /URL
      /F (http://exampe.com)
    >>
  >>
  /AP <<
    /N 10 0 R
    /D 13 0 R
  >>
>>
endobj
```

## What pypdf does: The current interface

For the PDF above, pypdf would give:

```
>>> from pypdf import PdfReader
>>> reader = PdfReader("pdflatex-forms.pdf")
>>> fields = reader.get_fields()
>>> fields
{'Name': {'/T': 'Name', '/FT': '/Tx', '/V': '', '/DV': ''},
'Check': {'/T': 'Check', '/FT': '/Btn', '/V': '/Off', '/_States_': ['/Yes', '/Off']},
'Submit': {'/T': 'Submit', '/FT': '/Btn', '/Ff': 65540, '/_States_': ['/Type', '/Subtype', '/BBox', '/FormType', '/Matrix', '/Resources', '/Filter', '/Off']}}
```

and

```
>>> from pypdf import PdfReader
>>> reader = PdfReader("pdflatex-forms.pdf")
>>> for page in reader.pages:
...     for annot in page.annotations:
...         print(annot.get_object())
... 
{'/Type': '/Annot', '/Rect': [182.198, 650.66, 269.23, 668.194], '/Subtype': '/Widget', '/F': 4, '/T': 'Name', '/FT': '/Tx', '/Q': 0, '/BS': {'/W': 1, '/S': '/S'}, '/MK': {'/BC': [1, 0, 0], '/BG': [1, 1, 1]}, '/DA': '/Helv 10 Tf 0 0 0 rg', '/DV': '', '/V': ''}
{'/Type': '/Annot', '/Rect': [183.582, 623.163, 195.537, 640.697], '/Subtype': '/Widget', '/F': 4, '/T': 'Check', '/FT': '/Btn', '/Q': 0, '/BS': {'/W': 1, '/S': '/S'}, '/AP': {'/N': {'/Yes': {}}}, '/MK': {'/BC': [1, 0, 0], '/BG': [1, 1, 1], '/CA': '4'}, '/DA': '/ZaDb 10 Tf 0 0 0 rg', '/H': '/P', '/V': '/Off', '/AS': '/Off'}
{'/Type': '/Annot', '/Rect': [153.694, 598.703, 189.235, 613.2], '/Subtype': '/Widget', '/F': 4, '/T': 'Submit', '/FT': '/Btn', '/Ff': 65540, '/H': '/P', '/BS': {'/W': 1, '/S': '/S'}, '/MK': {'/BC': [1, 0, 0]}, '/A': {'/S': '/SubmitForm', '/F': {'/FS': '/URL', '/F': 'http://exampe.com'}}, '/AP': {'/N': IndirectObject(10, 0, 140368148709328), '/D': IndirectObject(13, 0, 140368148709328)}}
```

## What pypdf does: Under the hood

Within `_reader.py` there is the part:

```
        if "/Fields" in tree:
            fields = cast(ArrayObject, tree["/Fields"])
            for f in fields:
                field = f.get_object()  # <--- if you print "f" here, you get the indirect object ID
                self._build_field(field, retval, fileobj, field_attributes)
```

Given that, you can print:

```python
from pypdf import PdfReader
reader = PdfReader("pdflatex-forms.pdf")
for page in reader.pages:
    for annot in page.annotations:
        print(annot)
```

which gives

```
IndirectObject(15, 0, 140141046791376)
IndirectObject(16, 0, 140141046791376)
IndirectObject(17, 0, 140141046791376)
```

The 3 numbers are:

* First (15/16/17): idnum - the Identifier for that indirect object
* Second (0 / 0 / 0): generation - the "version" of that object. Should be 0 in most PDFs.
* Third ( 140141046791376 / 140141046791376 / 140141046791376): A reference back to the PdfReader / PdfWriter

## Some tinkering

For the widgets you can do something like this:

```python
from pypdf import PdfReader

reader = PdfReader("pdflatex-forms.pdf")


widget_id2page_object = {}

for page_i, page in enumerate(reader.pages):
    for annot in page.annotations:
        widget_id2page_object[annot.idnum] = (page_i, annot)

print(widget_id2page_object)
```

giving

> {15: (0, IndirectObject(15, 0, 140141046951824)), 16: (0, IndirectObject(16, 0, 140141046951824)), 17: (0, IndirectObject(17, 0, 140141046951824))}

then you can adjust `_build_field` to pass the `f.idnum` and add:

```python
        # the new line:
        retval[key][NameObject("idnum")] = NumberObject(idnum)
        # those are old:
        if obj.get(FA.FT, "") == "/Ch":
            retval[key][NameObject("/_States_")] = obj[NameObject(FA.Opt)]
```

Now, if you call `get_fields` you get:

```
{'Name': {'/T': 'Name', '/FT': '/Tx', '/V': '', '/DV': '', 'idnum': 15},
 'Check': {'/T': 'Check',
  '/FT': '/Btn',
  '/V': '/Off',
  'idnum': 16,
  '/_States_': ['/Yes', '/Off']},
 'Submit': {'/T': 'Submit',
  '/FT': '/Btn',
  '/Ff': 65540,
  'idnum': 17,
  '/_States_': ['/Type',
   '/Subtype',
   '/BBox',
   '/FormType',
   '/Matrix',
   '/Resources',
   '/Filter',
   '/Off']}}
```

Allowing you to take idnum to get to the annotation and thus to the page.

## How do we continue?

Now the questions are:

1. Is there already a simpler way?
2. How/for what do people use `get_fields` at the moment? Could we simply add this type of information?
3. What is a clean way to add this feature?
Thank you Martin for this nice clarification.

According to the PDF specification (Document management — Portable document format — Part 1: PDF 1.7), if I understand correctly, an interactive form does not always has a widget annotation. 

source : 12.7. Interactive Forms, in 12.7.1 General
> A field’s children in the hierarchy **may** also include widget annotations (see 12.5.6.19, “Widget Annotations”) that define its appearance on the page

Maybe I should not rely on annotation to find the page number of a field?

> 3. What is a clean way to add this feature?

[The PDF Format](https://pypdf.readthedocs.io/en/stable/dev/pdf-format.html#the-pdf-format) page explains that each object has a numerical ID, which certainly also includes field objects, so maybe we could add a function that return the PageObject of a PdfObject?

```python
def get_page_object(PdfObject) -> Optional[PageObject]
```


Things are a little more complex in PDFs:
There is the Fields which are objects hierarchically organized under the acroform/fields which are page independants and the XObject (Widget) annotation which are the actual visual objects within the pages. 
these are within the page annotations are therefore page belonging. linking exists between the field and the widget in 2 manners:
a) the two objects are merged, having all widgets and fields properties mixed together
b) the two objects are linked, with the "/Parent" property in the widget pointing(indirectobject) to the field object and the widget pointed within the Kids Array inside the field object.

This last case is standard when you have radio buttons where one field only stores the choice value and multiple visual objects (radio toggles) are attached in the Kids property.
I see no objection to have the radio toggles on different pages (example in a form where the different pages will correspond to mutually exclusive subforms)
Fields repeated on multiple pages uses also a common field parent.

Also I see no reason (but maybe I'm wrong) to have a field without any widget : these woul be purely page independent.

therefore, if we implement a get_page function from my point of view it should return a tuple of pages.
a) `()` => no page linked
b) `(page_obj0)` => 1 page
c) `(page_obj0,page_obj1,page_obj2)` => for multiple widgets

**Do we agree ?**

complement: the standard provides an optional property `/page` in widgets that should point to the page. using some lookup to find the page the annotation belongs too should only be engaged if missing

edit: this PDF https://github.com/py-pdf/pypdf/files/14031491/Form_Structure_v50.pdf from #2425 shows some good examples

@MartinThoma, @stefan6419846 @sbourlon 
Do you agree with this proposal ?
If I understand it correctly, the proposed implementation is to enrich the form fields class with a method `get_page_indices`? Or do we want to implement this on the `PdfObject` base class?

I am against using `get_pages` if we indeed just return (zero-based) page indices and thus propose `def get_page_indices(self: PdfObject) -> List[int]` instead.
> I am against using `get_pages` if we indeed just return (zero-based) page indices and thus propose `def get_page_indices(self: PdfObject) -> List[int]` instead.

can you clarify what you dislike in get_pages return list of pages objects ? the function should  be in PdfReader :
def get_page_using_field(self, fld: PdfObject) -> List[PageObject]:

> can you clarify what you dislike in get_pages return list of pages objects ?

I have misread your message and thought we would just return the page indices - this would have been a misleading name, while it is completely fine for `PageObject`s.

`def get_page_using_field(self, field: PdfObject) -> List[PageObject]` would indeed work, although I am not sure whether the using `field` here for naming purposes does make sense if we can pass all(?) types of `PdfObject`s.
> `def get_page_using_field(self, field: PdfObject) -> List[PageObject]` would indeed work, although I am not sure whether the using `field` here for naming purposes does make sense if we can pass all(?) types of `PdfObject`s.

we can include some code to confirm it is a field (checking for expected properties)

Commit message: code + test

Patch to review:
diff --git a/pypdf/_reader.py b/pypdf/_reader.py
index a2ec36288..3dbd37265 100644
--- a/pypdf/_reader.py
+++ b/pypdf/_reader.py
@@ -667,6 +667,75 @@ def indexed_key(k: str, fields: Dict[Any, Any]) -> str:
                     ff[indexed_key(cast(str, value["/T"]), ff)] = value.get("/V")
         return ff
 
+    def get_pages_showing_field(
+        self, field: Union[Field, PdfObject, IndirectObject]
+    ) -> List[PageObject]:
+        """
+        provides list of pages where the field is called
+
+        Args:
+            field: Field Object, PdfObject or IndirectObject referencing a Field
+
+        Returns: List:
+            Empty list : the field has no widgets attached
+                         (either hidden field or ancestor field)
+            single page list : page where the widget is present (most common)
+            multiple page list: field multiple times with multiple kids widgets
+                         (ex:radio buttons, field repeated on multiple pages)
+
+        Note:
+            to get page indexes, use:
+            [ p.page_number for p in pdfdoc.get_pages_showing_field(field) ]
+        """
+
+        def _get_inherit(obj: DictionaryObject, key: str) -> Any:
+            if key in obj:
+                return obj[key]
+            elif "/Parent" in obj:
+                return _get_inherit(
+                    cast(DictionaryObject, obj["/Parent"].get_object()), key
+                )
+            else:
+                return None
+
+        try:
+            # to cope with all types
+            field = cast(DictionaryObject, field.indirect_reference.get_object())  # type: ignore
+        except Exception as exc:
+            raise ValueError("field type is invalid") from exc
+        if _get_inherit(field, "/FT") is None:
+            raise ValueError("field is not valid")
+        ret = []
+        if field.get("/Subtype", "") == "/Widget":
+            if "/P" in field:
+                ret = [field["/P"].get_object()]
+            else:
+                ret = [
+                    p
+                    for p in self.pages
+                    if field.indirect_reference in p.get("/Annots", "")
+                ]
+        else:
+            kids = field.get("/Kids", ())
+            for k in kids:
+                k = k.get_object()
+                if (k.get("/Subtype", "") == "/Widget") and ("/V" not in k):
+                    # kids that is just a widget not an field:
+                    if "/P" in field:
+                        ret += [field["/P"].get_object()]
+                    else:
+                        ret += [
+                            p
+                            for p in self.pages
+                            if k.indirect_reference in p.get("/Annots", "")
+                        ]
+        return [
+            x
+            if isinstance(x, PageObject)
+            else (self.pages[self._get_page_number_by_indirect(x.indirect_reference)])  # type: ignore
+            for x in ret
+        ]
+
     def _get_named_destinations(
         self,
         tree: Union[TreeObject, None] = None,
diff --git a/pypdf/_writer.py b/pypdf/_writer.py
index 9c268b908..61b9eb242 100644
--- a/pypdf/_writer.py
+++ b/pypdf/_writer.py
@@ -96,6 +96,7 @@
     DecodedStreamObject,
     Destination,
     DictionaryObject,
+    Field,
     Fit,
     FloatObject,
     IndirectObject,
@@ -424,6 +425,29 @@ def _get_num_pages(self) -> int:
         pages = cast(Dict[str, Any], self.get_object(self._pages))
         return int(pages[NameObject("/Count")])
 
+    def _get_page_number_by_indirect(
+        self, indirect_reference: Union[None, int, NullObject, IndirectObject]
+    ) -> Optional[int]:
+        """
+        Generate _page_id2num.
+
+        Args:
+            indirect_reference:
+
+        Returns:
+            The page number or None
+        """
+        # to provide same function as in PdfReader
+        if indirect_reference is None or isinstance(indirect_reference, NullObject):
+            return None
+        if isinstance(indirect_reference, int):
+            indirect_reference = IndirectObject(indirect_reference, 0, self)
+        obj = indirect_reference.get_object()
+        if isinstance(obj, PageObject):
+            return obj.page_number
+        else:
+            return None
+
     @property
     def pages(self) -> List[PageObject]:
         """
@@ -981,6 +1005,75 @@ def reattach_fields(
                 lst.append(ano)
         return lst
 
+    def get_pages_showing_field(
+        self, field: Union[Field, PdfObject, IndirectObject]
+    ) -> List[PageObject]:
+        """
+        provides list of pages where the field is called
+
+        Args:
+            field: Field Object, PdfObject or IndirectObject referencing a Field
+
+        Returns: List:
+            Empty list : the field has no widgets attached
+                         (either hidden field or ancestor field)
+            single page list : page where the widget is present (most common)
+            multiple page list: field multiple times with multiple kids widgets
+                         (ex:radio buttons, field repeated on multiple pages)
+
+        Note:
+            to get page indexes, use:
+            [ p.page_number for p in pdfdoc.get_pages_showing_field(field) ]
+        """
+
+        def _get_inherit(obj: DictionaryObject, key: str) -> Any:
+            if key in obj:
+                return obj[key]
+            elif "/Parent" in obj:
+                return _get_inherit(
+                    cast(DictionaryObject, obj["/Parent"].get_object()), key
+                )
+            else:
+                return None
+
+        try:
+            # to cope with all types
+            field = cast(DictionaryObject, field.indirect_reference.get_object())  # type: ignore
+        except Exception as exc:
+            raise ValueError("field type is invalid") from exc
+        if _get_inherit(field, "/FT") is None:
+            raise ValueError("field is not valid")
+        ret = []
+        if field.get("/Subtype", "") == "/Widget":
+            if "/P" in field:
+                ret = [field["/P"].get_object()]
+            else:
+                ret = [
+                    p
+                    for p in self.pages
+                    if field.indirect_reference in p.get("/Annots", "")
+                ]
+        else:
+            kids = field.get("/Kids", ())
+            for k in kids:
+                k = k.get_object()
+                if (k.get("/Subtype", "") == "/Widget") and ("/V" not in k):
+                    # kids that is just a widget not an field:
+                    if "/P" in field:
+                        ret += [field["/P"].get_object()]
+                    else:
+                        ret += [
+                            p
+                            for p in self.pages
+                            if k.indirect_reference in p.get("/Annots", "")
+                        ]
+        return [
+            x
+            if isinstance(x, PageObject)
+            else (self.pages[self._get_page_number_by_indirect(x.indirect_reference)])  # type: ignore
+            for x in ret
+        ]
+
     def clone_reader_document_root(self, reader: PdfReader) -> None:
         """
         Copy the reader document root to the writer and all sub elements,
diff --git a/tests/test_workflows.py b/tests/test_workflows.py
index 07394adc7..eacf8cca5 100644
--- a/tests/test_workflows.py
+++ b/tests/test_workflows.py
@@ -1108,3 +1108,52 @@ def test_text_extraction_invalid_mode():
     reader = PdfReader(pdf_path)
     with pytest.raises(ValueError, match="Invalid text extraction mode"):
         reader.pages[0].extract_text(extraction_mode="foo")  # type: ignore
+
+
+@pytest.mark.enable_socket()
+def test_get_page_showing_field():
+    """
+    Test new function to get pages
+    Uses testfile from #2452 in order to get fiels on multiple pages,
+        choices boxes,...
+    """
+    url = "https://github.com/py-pdf/pypdf/files/14031491/Form_Structure_v50.pdf"
+    name = "iss2452.pdf"
+    reader = PdfReader(BytesIO(get_data_from_url(url, name)))
+    writer = PdfWriter(clone_from=reader)
+
+    # validate with Field:  only works on Reader (no get_field on writer yet)
+    fld = reader.get_fields()
+    assert [
+        p.page_number for p in reader.get_pages_showing_field(fld["FormVersion"])
+    ] == [0]
+
+    # validate with dictionary object
+    # NRCategory field is a radio box
+    assert [
+        p.page_number
+        for p in reader.get_pages_showing_field(
+            reader.trailer["/Root"]["/AcroForm"]["/Fields"][8].get_object()
+        )
+    ] == [0, 0, 0, 0, 0]
+    assert [
+        p.page_number
+        for p in writer.get_pages_showing_field(
+            writer._root_object["/AcroForm"]["/Fields"][8].get_object()
+        )
+    ] == [0, 0, 0, 0, 0]
+
+    # validate with indirectObject
+    # SiteID field is a textbox on multiple pages
+    assert [
+        p.page_number
+        for p in reader.get_pages_showing_field(
+            reader.trailer["/Root"]["/AcroForm"]["/Fields"][99].get_object()
+        )
+    ] == [0, 1]
+    assert [
+        p.page_number
+        for p in writer.get_pages_showing_field(
+            writer._root_object["/AcroForm"]["/Fields"][99].get_object()
+        )
+    ] == [0, 1]
diff --git a/tests/test_writer.py b/tests/test_writer.py
index 89c79f681..7ed2fd961 100644
--- a/tests/test_writer.py
+++ b/tests/test_writer.py
@@ -2011,3 +2011,19 @@ def test_reattach_fields():
     assert isinstance(writer.pages[0]["/Annots"][0], IndirectObject)
     del writer.pages[1]["/Annots"]
     assert len(writer.reattach_fields(writer.pages[1])) == 0
+
+
+def test_get_pagenumber_from_indirectobject():
+    """
+    Test Reattach function
+    addressed in #2453
+    """
+    pdf_path = RESOURCE_ROOT / "crazyones.pdf"
+    writer = PdfWriter(clone_from=pdf_path)
+    assert writer._get_page_number_by_indirect(None) is None
+    assert writer._get_page_number_by_indirect(NullObject()) is None
+
+    ind = writer.pages[0].indirect_reference
+    assert writer._get_page_number_by_indirect(ind) == 0
+    assert writer._get_page_number_by_indirect(ind.idnum) == 0
+    assert writer._get_page_number_by_indirect(ind.idnum + 1) is None


Code review comments:
diff_hunk: @@ -667,6 +667,75 @@ def indexed_key(k: str, fields: Dict[Any, Any]) -> str:
                     ff[indexed_key(cast(str, value["/T"]), ff)] = value.get("/V")
         return ff
 
+    def get_pages_showing_field(
+        self, field: Union[Field, PdfObject, IndirectObject]
+    ) -> List[PageObject]:
+        """
+        provides list of pages where the field is called
+
+        Args:
+            field: Field Object, PdfObject or IndirectObject referencing a Field
+
+        Returns: List:
+            Empty list : the field has no widgets attached
+                         (either hidden field or ancestor field)
+            single page list : page where the widget is present (most common)
+            multiple page list: field multiple times with multiple kids widgets
+                         (ex:radio buttons, field repeated on multiple pages)
+
+        Note:
+            to get page indexes, use:
+            [ p.page_number for p in pdfdoc.get_pages_showing_field(field) ]
+        """
+
+        def _get_inherit(obj: DictionaryObject, key: str) -> Any:
path: pypdf/_reader.py
text: @user1:
Should probably be `_get_inherited` instead.

@author:
fixed