Repository: Kozea/WeasyPrint

Issue:
Support for editable forms lacks support for radio buttons
See 

https://print-css.rocks/lesson/lesson-forms
https://print-css.rocks/lesson/lesson-forms/download/weasyprint.pdf

No offense but your implementation is only a partial implementation for editable forms (compared to other tools)  since
the support for radio buttons is missing (and the rendering is broken).



Discussion:
@liZe - Is the constraint that only one option in a radio button group is selected that complicates including it in the editable (fillable) forms? Is there anywhere that talks more about what's missing? 

I was looking at #1796, and [you'd outline an approach to adding PDF forms when responding](https://github.com/Kozea/WeasyPrint/issues/61#issuecomment-301051989) for adding support. Looking at [`add_inputs`](https://github.com/Kozea/WeasyPrint/pull/1796/files#diff-accc4e9798687fde6ed3d860869ec6f460910047b7e37736ccf651a20d928857R94-R194) to see if I might get a handle on how the new support for textfield, textarea, & checkbox were added. 

Is there any workaround? There's older response [over here](https://github.com/Kozea/WeasyPrint/issues/328#issuecomment-528347963) that points to a "tips & tricks" docs page that is to the previous host of the docs. I originally thought you might be able to render an underlying checkbox, but style it to appear as a radio button (and you'd lose the constraint for having only one _selected_). 

Would looking at the [pydyf](https://github.com/CourtBouillon/pydyf) approach to creating the PDF form tag help if I was brave enough to take a shot at a pull request? 

Thanks for all the time you've put into this project & other Kozea projects 
> Is the constraint that only one option in a radio button group is selected that complicates including it in the editable (fillable) forms?

The problem is to create these groups: we must find radio buttons that have the same name (shouldn’t be too hard) in the same form (that’s the hard part, because we don’t store this information). Rendering these inputs is [the easy part](https://github.com/Kozea/WeasyPrint/blob/361afcb4995b43f69f113295b5faa3c1ebf7b75f/weasyprint/pdf/anchors.py#L93) (after spending some time reading the PDF specification, of course :smile:).

The clean solution would be to gather forms and their inputs, instead of just gathering a list of inputs, in [`gather_anchors`](https://github.com/Kozea/WeasyPrint/blob/361afcb4995b43f69f113295b5faa3c1ebf7b75f/weasyprint/anchors.py#L30). We basically need to:
- replace `inputs` by `forms`, something like a dictionary with forms as keys and a list of inputs as values,
- detect if `box` is a form, if it is store it in `forms` and keep the information that it’s the current form,
- detect if `box` is an input, if it is store it in the current form,
- change the `document` to store forms instead of inputs,
- change the rendering code to receive forms instead of inputs, and to correctly draw radio buttons by finding radio button groups.

Gathering forms would also be useful to create PDF forms that send data to servers. But let’s keep that for another issue!
Thank you for this detailed reply. I'll see if I can understand a bit better and maybe I'll have a follow-up question or two. 

> Rendering these inputs is [the easy part](https://github.com/Kozea/WeasyPrint/blob/361afcb4995b43f69f113295b5faa3c1ebf7b75f/weasyprint/pdf/anchors.py#L93) (after spending some time reading the PDF specification, of course 😄).

I noticed in your profile the mention of being a "specification reader" 😉 

@liZe - when I was looking at another Python pdf library, I believe they stated the PDF 1.7 spec is the latest. Before I look closer, I wanted to make sure I'm in the right _"neighborhood"_: 

This is falls in [the Acrobat SDK](https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/index.html) area, correct? 

And this is the document to look at? 

https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf

----

📝 _the reason I asked if because the previous hyperlink used to refer to PDF 1.7 spec, https://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/pdf_reference_1-7.pdf, is now resolving to this page:_

https://opensource.adobe.com/dc-acrobat-sdk-docs/

(cool that there is **now** an opensource.adobe.com subdomain ✨)
> when I was looking at another Python pdf library, I believe they stated the PDF 1.7 spec is the latest. Before I look closer, I wanted to make sure I'm in the right _"neighborhood"_:

There’s actually [PDF 2](https://www.pdfa.org/resource/iso-32000-pdf/) (now available for free), but 1.7 is totally OK and easily available online, so you can use it.

> This is falls in [the Acrobat SDK](https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/index.html) area, correct?

You don’t have to use any of Adobe code, or to follow any of their API.

> And this is the document to look at?

Yes. Section 12.7 is about forms, and section 12.7.4.2.4 is about radio buttons. You’ll see that’s it’s actually a subtype of buttons, really close to checkboxes (that are already implemented in WeasyPrint).
Sorry @liZe - I worded this confusingly. I mainly wanted to confirm that I was reading that the right specification. 

I did find the section on Field Types and Radio Buttons in PDF 1.7. 

It seems like there is some valuable effort to get to the point where radio buttons could be rendered. So I'm going to try to address the first 4 bullets mentioned (regarding gather as form(s) that are keyed from form => input(s)) over here and I'll pull request them back to that forked repository in case anyone is curious to look at it:
- https://github.com/alenards/WeasyPrint


@zopyx - a note about the sample provided here, the weasyprint.pdf is no longer available at the provided link: 

![Screenshot 2023-08-02 at 9 37 56 AM](https://github.com/Kozea/WeasyPrint/assets/78614213/14d63484-e3d4-4b27-8cc3-4f0d3df39bd1)

@liZe - given what you know the source code, focusing these 2 bullets: 

- detect if `box` is a form, if it is store it in `forms` and keep the information that it’s the current form,
- detect if `box` is an input, if it is store it in the current form

Do you have any thoughts about how you'd handle encountering a `box` where `box.is_input()` is `True`, but there is **not** a _current form_? 
> Do you have any thoughts about how you'd handle encountering a `box` where `box.is_input()` is `True`, but there is **not** a _current form_?

If it’s not in a form, then it’s global and will be put with other global inputs.

If we store forms with this structure:

`{<form1>: [<input11>, <input12>], <form2>: [<input21>, <input22>]}`

then we can use `None` as a key to store inputs not in a form.
That makes sense. I wasn't sure of the implications. Thanks @liZe 🙇 

I have a PR nearing _draft_ for "gathering forms" so that they're tracked. It does **_not_** address anything with the radio button groups _yet_. I thought if I sliced it a bit thinner it might help me tune up my Python (I haven't written Python in 3-5 years). 
I might be getting ahead of myself. But do see the "gathering" of inputs within their containing form (and/or global form) as causing any problems with the order they appear within a page of a document? 

I can try to look closer at the pytests maybe for an answer there. 
> But do see the "gathering" of inputs within their containing form (and/or global form) as causing any problems with the order they appear within a page of a document?

I’m not sure to understand your problem, but I don’t see why the order could be a problem.
I'll see if I can get a little example together. 

But if we're putting the inputs into a forms dictionary, we lose the source order when process the dictionary by its' keys. 
I’m not sure that it’s a problem:
1. Why should we care about the order?
2. Dictionaries keep order in Python.

> I'll see if I can get a little example together.

That would help!
I forgot that Python will get your the order that keys were _"put"_ into the dictionary! 

I expected that when we did a `for key in forms: # blah`, we'd get the inputs in whatever order the hash function for the dictionary had encoded them. 

😮‍💨 🤦 Sorry for the confusion. 
👋 excited to see the conversation here and possible solution!

It looks like this thread may have gone a bit stale - @alenards I'm curious if you're still looking into this, otherwise I may take a look as it'd be really nice to have for a project I'm working on.
Hi @okkays / Kevin, 

I haven't been able to spend enough time here. I agree that it is a nice project, and I know that it would be highly valuable. I had my _super_ draft work in branches on the fork I made: https://github.com/alenards/WeasyPrint/branches

I haven't touched it for 9 months, but I was using the feedback from here to "gathering together" the form inputs, and then trying to get all the radio button inputs in their group: 

https://github.com/alenards/WeasyPrint/commits/issue-1831-forms-gathering/

I thought I'd get them "tracked", but I didn't know enough about the encode in the rendering (and trying to wrap my head around the spec, how "children" - aka _kids_ - needed to be written out). 

I'm on a US timezone, seems from your profile you are too. I've been up to _"pair & ponder"_ on this. I just know I can't try to tackle it as a primary contributor right now 😢 
Thanks for the quick response and sharing your work so far! I'll take a look through what you have and spend a bit of time in the code to try and size this up 😄 
What I have committed is very much "sketches"; I don't think it would make it through any level of code review. And, it might help. I think the instructions for getting test suites running locally are up-to-date now 🤞 (or hope they are). 
#2143 is up for this following the outline from https://github.com/Kozea/WeasyPrint/issues/1831#issuecomment-1659757718 😄 
WHOA! 



Patch:
diff --git a/weasyprint/anchors.py b/weasyprint/anchors.py
index e40992005..ab18e53f1 100644
--- a/weasyprint/anchors.py
+++ b/weasyprint/anchors.py
@@ -27,10 +27,11 @@ def rectangle_aabb(matrix, pos_x, pos_y, width, height):
     return box_x1, box_y1, box_x2, box_y2
 
 
-def gather_anchors(box, anchors, links, bookmarks, inputs, parent_matrix=None):
+def gather_anchors(box, anchors, links, bookmarks, forms, parent_matrix=None,
+                   parent_form=None):
     """Gather anchors and other data related to specific positions in PDF.
 
-    Currently finds anchors, links, bookmarks and inputs.
+    Currently finds anchors, links, bookmarks and forms.
 
     """
     # Get box transformation matrix.
@@ -89,6 +90,11 @@ def gather_anchors(box, anchors, links, bookmarks, inputs, parent_matrix=None):
     has_anchor = anchor_name and anchor_name not in anchors
     is_input = box.is_input()
 
+    if box.is_form():
+        parent_form = box.element
+        if parent_form not in forms:
+            forms[parent_form] = []
+
     if has_bookmark or has_link or has_anchor or is_input:
         if is_input:
             pos_x, pos_y = box.content_box_x(), box.content_box_y()
@@ -106,7 +112,7 @@ def gather_anchors(box, anchors, links, bookmarks, inputs, parent_matrix=None):
                 link_type = 'attachment'
             links.append((link_type, target, rectangle, box))
         if is_input:
-            inputs.append((box.element, box.style, rectangle))
+            forms[parent_form].append((box.element, box.style, rectangle))
         if matrix and (has_bookmark or has_anchor):
             pos_x, pos_y = matrix.transform_point(pos_x, pos_y)
         if has_bookmark:
@@ -116,7 +122,7 @@ def gather_anchors(box, anchors, links, bookmarks, inputs, parent_matrix=None):
             anchors[anchor_name] = pos_x, pos_y
 
     for child in box.all_children():
-        gather_anchors(child, anchors, links, bookmarks, inputs, matrix)
+        gather_anchors(child, anchors, links, bookmarks, forms, matrix, parent_form)
 
 
 def make_page_bookmark_tree(page, skipped_levels, last_by_depth,
diff --git a/weasyprint/css/html5_ua.css b/weasyprint/css/html5_ua.css
index c28628081..67f846252 100644
--- a/weasyprint/css/html5_ua.css
+++ b/weasyprint/css/html5_ua.css
@@ -187,8 +187,10 @@ input[value]::before {
   overflow: hidden;
 }
 input::before,
-input[value=""]::before {
-  content: " ";
+input[value=""]::before,
+input[type="checkbox"]::before,
+input[type="radio"]::before {
+  content: "";
 }
 select {
   background: lightgrey;
diff --git a/weasyprint/document.py b/weasyprint/document.py
index 08b7e7f64..ffdab3a35 100644
--- a/weasyprint/document.py
+++ b/weasyprint/document.py
@@ -28,6 +28,7 @@ class Page:
     instantiated directly.
 
     """
+
     def __init__(self, page_box):
         #: The page width, including margins, in CSS pixels.
         self.width = page_box.margin_width()
@@ -67,14 +68,15 @@ def __init__(self, page_box):
         #: ``(x, y)`` point in CSS pixels from the top-left of the page.
         self.anchors = {}
 
-        #: The :obj:`list` of ``(element, attributes, rectangle)`` :obj:`tuples
-        #: <tuple>`. A ``rectangle`` is ``(x, y, width, height)``, in CSS
+        #: The :obj:`dict` mapping form elements to a list
+        #: of ``(element, attributes, rectangle)`` :obj:`tuples <tuple>`.
+        #: A ``rectangle`` is ``(x, y, width, height)``, in CSS
         #: pixels from the top-left of the page. ``atributes`` is a
         #: :obj:`dict` of HTML tag attributes and values.
-        self.inputs = []
+        #: The key ``None`` will contain inputs that are not part of a form.
+        self.forms = {None: []}
 
-        gather_anchors(
-            page_box, self.anchors, self.links, self.bookmarks, self.inputs)
+        gather_anchors(page_box, self.anchors, self.links, self.bookmarks, self.forms)
         self._page_box = page_box
 
     def paint(self, stream, scale=1):
@@ -105,6 +107,7 @@ class DocumentMetadata:
     New attributes may be added in future versions of WeasyPrint.
 
     """
+
     def __init__(self, title=None, authors=None, description=None,
                  keywords=None, generator=None, created=None, modified=None,
                  attachments=None, lang=None, custom=None):
@@ -162,6 +165,7 @@ class DiskCache:
     (i.e. RasterImage instances) are still stored in memory.
 
     """
+
     def __init__(self, folder):
         self._path = Path(folder)
         self._path.mkdir(parents=True, exist_ok=True)
diff --git a/weasyprint/formatting_structure/boxes.py b/weasyprint/formatting_structure/boxes.py
index f755e5762..82fe683cb 100644
--- a/weasyprint/formatting_structure/boxes.py
+++ b/weasyprint/formatting_structure/boxes.py
@@ -336,6 +336,12 @@ def is_input(self):
                 return not isinstance(self, (LineBox, TextBox))
         return False
 
+    def is_form(self):
+        """Return whether this box is a form element."""
+        if self.element is None:
+            return False
+        return self.element.tag == 'form'
+
 
 class ParentBox(Box):
     """A box that has children."""
diff --git a/weasyprint/pdf/__init__.py b/weasyprint/pdf/__init__.py
index 7db96de28..b6f2b2b2d 100644
--- a/weasyprint/pdf/__init__.py
+++ b/weasyprint/pdf/__init__.py
@@ -11,7 +11,7 @@
 from .stream import Stream
 
 from .anchors import (  # isort:skip
-    add_annotations, add_inputs, add_links, add_outlines, resolve_links,
+    add_annotations, add_forms, add_links, add_outlines, resolve_links,
     write_pdf_attachment)
 
 VARIANTS = {
@@ -184,8 +184,8 @@ def generate_pdf(document, target, zoom, **options):
         add_annotations(
             links_and_anchors[0], matrix, document, pdf, pdf_page, annot_files,
             compress)
-        add_inputs(
-            page.inputs, matrix, pdf, pdf_page, resources, stream,
+        add_forms(
+            page.forms, matrix, pdf, pdf_page, resources, stream,
             document.font_config.font_map, compress)
         page.paint(stream, scale)
 
diff --git a/weasyprint/pdf/anchors.py b/weasyprint/pdf/anchors.py
index 9843017da..383a6d539 100644
--- a/weasyprint/pdf/anchors.py
+++ b/weasyprint/pdf/anchors.py
@@ -1,7 +1,9 @@
 """Insert anchors, links, bookmarks and inputs in PDFs."""
 
+import collections
 import io
 import mimetypes
+from base64 import b64encode
 from hashlib import md5
 from os.path import basename
 from urllib.parse import unquote, urlsplit
@@ -91,10 +93,10 @@ def add_outlines(pdf, bookmarks, parent=None):
     return outlines, count
 
 
-def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
-               compress):
+def add_forms(forms, matrix, pdf, page, resources, stream, font_map,
+              compress):
     """Include form inputs in PDF."""
-    if not inputs:
+    if not forms or not any(forms.values()):
         return
 
     if 'Annots' not in page:
@@ -109,20 +111,43 @@ def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
     context = ffi.gc(
         pango.pango_font_map_create_context(font_map),
         gobject.g_object_unref)
-    for i, (element, style, rectangle) in enumerate(inputs):
+    inputs_with_forms = [
+        (form, element, style, rectangle)
+        for form, inputs in forms.items()
+        for element, style, rectangle in inputs
+    ]
+    radio_groups = collections.defaultdict(dict)
+    for i, (form, element, style, rectangle) in enumerate(inputs_with_forms):
         rectangle = (
             *matrix.transform_point(*rectangle[:2]),
             *matrix.transform_point(*rectangle[2:]))
 
         input_type = element.attrib.get('type')
+        input_value = element.attrib.get('value', 'Yes')
         default_name = f'unknown-{page_reference.decode()}-{i}'
         input_name = element.attrib.get('name', default_name)
         # TODO: where does this 0.75 scale come from?
         font_size = style['font_size'] * 0.75
         field_stream = pydyf.Stream(compress=compress)
         field_stream.set_color_rgb(*style['color'][:3])
-        if input_type == 'checkbox':
-            # Checkboxes
+        if input_type in ('radio', 'checkbox'):
+            if input_type == 'radio':
+                if input_name not in radio_groups[form]:
+                    radio_groups[form][input_name] = group = pydyf.Dictionary({
+                        'FT': '/Btn',
+                        'Ff': (1 << (15 - 1)) + (1 << (16 - 1)),  # NoToggle & Radio
+                        'T': pydyf.String(f'{len(radio_groups)}'),
+                        'V': '/Off',
+                        'Kids': pydyf.Array(),
+                    })
+                    pdf.add_object(group)
+                    pdf.catalog['AcroForm']['Fields'].append(group.reference)
+                group = radio_groups[form][input_name]
+                character = 'l'  # Disc character in Dingbats
+            else:
+                character = '4'  # Check character in Dingbats
+
+            # Create stream when input is checked
             width = rectangle[2] - rectangle[0]
             height = rectangle[1] - rectangle[3]
             checked_stream = pydyf.Stream(extra={
@@ -135,17 +160,18 @@ def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
             checked_stream.begin_text()
             checked_stream.set_color_rgb(*style['color'][:3])
             checked_stream.set_font_size('ZaDb', font_size)
-            # Center (let’s assume that Dingbat’s check has a 0.8em size)
-            x = (width - font_size * 0.8) / 2
-            y = (height - font_size * 0.8) / 2
+            # Center (let’s assume that Dingbat’s characters have a 0.75em size)
+            x = (width - font_size * 0.75) / 2
+            y = (height - font_size * 0.75) / 2
             checked_stream.move_text_to(x, y)
-            checked_stream.show_text_string('4')
+            checked_stream.show_text_string(character)
             checked_stream.end_text()
             checked_stream.pop_state()
             pdf.add_object(checked_stream)
 
             checked = 'checked' in element.attrib
             field_stream.set_font_size('ZaDb', font_size)
+            key = b64encode(input_value.encode(), altchars=b"+-").decode()
             field = pydyf.Dictionary({
                 'Type': '/Annot',
                 'Subtype': '/Widget',
@@ -153,14 +179,21 @@ def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
                 'FT': '/Btn',
                 'F': 1 << (3 - 1),  # Print flag
                 'P': page.reference,
-                'T': pydyf.String(input_name),
-                'V': '/Yes' if checked else '/Off',
+                'AS': f'/{key}' if checked else '/Off',
                 'AP': pydyf.Dictionary({'N': pydyf.Dictionary({
-                    'Yes': checked_stream.reference,
-                })}),
-                'AS': '/Yes' if checked else '/Off',
+                    key: checked_stream.reference})}),
+                'MK': pydyf.Dictionary({'CA': pydyf.String(character)}),
                 'DA': pydyf.String(b' '.join(field_stream.stream)),
             })
+            pdf.add_object(field)
+            if input_type == 'radio':
+                field['Parent'] = group.reference
+                if checked:
+                    group['V'] = f'/{key}'
+                group['Kids'].append(field.reference)
+            else:
+                field['T'] = pydyf.String(input_name)
+                field['V'] = field['AS']
         elif element.tag == 'select':
             # Select fields
             font_description = get_font_description(style)
@@ -174,7 +207,7 @@ def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
             selected_values = []
             for option in element:
                 value = pydyf.String(option.attrib.get('value', ''))
-                text = pydyf.String(option.text or "")
+                text = pydyf.String(option.text or '')
                 options.append(pydyf.Array([value, text]))
                 if 'selected' in option.attrib:
                     selected_values.append(value)
@@ -198,6 +231,7 @@ def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
                 field['V'] = (
                     selected_values[-1] if selected_values
                     else pydyf.String(''))
+            pdf.add_object(field)
         else:
             # Text, password, textarea, files, and unknown
             font_description = get_font_description(style)
@@ -231,8 +265,8 @@ def add_inputs(inputs, matrix, pdf, page, resources, stream, font_map,
             maxlength = element.get('maxlength')
             if maxlength and maxlength.isdigit():
                 field['MaxLen'] = element.get('maxlength')
+            pdf.add_object(field)
 
-        pdf.add_object(field)
         page['Annots'].append(field.reference)
         pdf.catalog['AcroForm']['Fields'].append(field.reference)
 


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
index 2e81f0846..0b94ec101 100644
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -510,6 +510,13 @@ def test_partial_pdf_custom_metadata():
     ('<input>', ['/Tx', '/V ()']),
     ('<input value="">', ['/Tx', '/V ()']),
     ('<input type="checkbox">', ['/Btn']),
+    ('<input type="radio">',
+     ['/Btn', '/V /Off', '/AS /Off', '/Ff 49152']),
+    ('<input checked type="radio" name="foo" value="value">',
+     ['/Btn', '/T (1)', '/V /dmFsdWU=', '/AS /dmFsdWU=']),
+    ('<form><input type="radio" name="foo" value="v0"></form>'
+     '<form><input checked type="radio" name="foo" value="v1"></form>',
+     ['/Btn', '/AS /djE=', '/V /djE=', '/AS /Off', '/V /Off']),
     ('<textarea></textarea>', ['/Tx', '/V ()']),
     ('<select><option value="a">A</option></select>', ['/Ch', '/Opt']),
     ('<select>'
@@ -525,7 +532,8 @@ def test_partial_pdf_custom_metadata():
 def test_pdf_inputs(html, fields):
     stdout = _run('--pdf-forms --uncompressed-pdf - -', html.encode())
     assert b'AcroForm' in stdout
-    assert all(field.encode() in stdout for field in fields)
+    for field in fields:
+        assert field.encode() in stdout
     stdout = _run('--uncompressed-pdf - -', html.encode())
     assert b'AcroForm' not in stdout