Repository: robocorp/rpaframework

Issue:
`RPA.PDF.Find Text`: Add `subtext:` strategy and case ignoring switch for an easier text search
Affects [Find Text](https://robocorp.com/docs/libraries/rpa-framework/rpa-pdf/keywords#find-text)

### Context

We currently support the following strategies: `text:`, `regex:` and `coords:` and they are working just fine in most of the PDFs for setting an anchor which matches the locator.

But in too many scenarios (to ignore this Issue), we see text like `Invoice number: 1234` in a text-based PDF and we have no idea what the internal text object box contains as the text value. It might be any of the following cases:
- `Invoice number:`
- `Invoice number`
- `Invoice number: 1234`
- `Invoice` - rarely seen
- `number:` - rarely seen
- `some garbage\n Invoice number: 1234 more garbage\n here\n and there` - mostly seen

And in this scenario, the quickest option to a solution is this locator: `regex:[\s\S]*Invoice number[\s\S]*`, but this seems a little bit too mystic to any average RPA developer.

A cleaner not-so-specific locator would be `subtext:Invoice number`. With an optional `ignore_case=${True}` we can even pass something like `subtext:invoice` and that would still work. (but increasing the chance of a false-positive result -- no worries, developer's call here)

### ToDo
- [x] Add `subtext:` strategy as well.
- [x] Add `ignore_case` param which will work with any text/subtext/regex locator and even the `regexp` param if passed (to find neighbors matching the filter, relative to anchor).


Discussion:

Resolving patch:
diff --git a/packages/pdf/src/RPA/PDF/keywords/finder.py b/packages/pdf/src/RPA/PDF/keywords/finder.py
--- a/packages/pdf/src/RPA/PDF/keywords/finder.py
+++ b/packages/pdf/src/RPA/PDF/keywords/finder.py
@@ -42,6 +42,8 @@ class Match:
 class FinderKeywords(LibraryContext):
     """Keywords for locating elements."""
 
+    RE_FLAGS = re.MULTILINE | re.DOTALL  # default regexp flags
+
     def __init__(self, ctx):
         super().__init__(ctx)
 
@@ -77,6 +79,13 @@ def _log_element(self, elem: Element, prefix: str = ""):
         template = f"{prefix} box %d | bbox %s | text %r"
         self.logger.debug(template, elem.boxid, elem.bbox, elem.text)
 
+    @classmethod
+    def _re_flags(cls, ignore_case: bool) -> int:
+        flags = cls.RE_FLAGS
+        if ignore_case:
+            flags |= re.IGNORECASE
+        return flags
+
     @keyword
     def find_text(
         self,
@@ -85,17 +94,18 @@ def find_text(
         direction: str = "right",
         closest_neighbours: Optional[Union[int, str]] = 1,
         strict: bool = False,
-        regexp: str = None,
+        regexp: Optional[str] = None,
         trim: bool = True,
+        ignore_case: bool = False,
     ) -> List[Match]:
         """Find the closest text elements near the set anchor(s) through `locator`.
 
         The PDF will be parsed automatically before elements can be searched.
 
         :param locator: Element to set anchor to. This can be prefixed with either
-            `text:`, `regex:` or `coords:` to find the anchor by text or coordinates.
-            `text` is assumed if no such prefix is specified. (text search is case
-            insensitive)
+            "text:", "subtext:", "regex:" or "coords:" to find the anchor by text or
+            coordinates. The "text" strategy is assumed if no such prefix is specified.
+            (text search is case-sensitive; use `ignore_case` param for controlling it)
         :param pagenum: Page number where search is performed on, defaults to 1 (first
             page).
         :param direction: In which direction to search for text elements. This can be
@@ -109,6 +119,8 @@ def find_text(
             candidates in range are considered valid neighbours.
         :param trim: Automatically trim leading/trailing whitespace from the text
             elements. (switched on by default)
+        :param ignore_case: Do a case-insensitive search when set to `True`. (affects
+            the passed `locator` and `regexp` filtering)
         :returns: A list of `Match` objects where every match has the following
             attributes: `.anchor` - the matched text with the locator; `.neighbours` -
             a list of adjacent texts found on the specified direction
@@ -154,7 +166,7 @@ def pdf_invoice_parsing():
             closest_neighbours = int(closest_neighbours)
         self.logger.info(
             "Searching for %s neighbour(s) to the %s of %r on page %d using regular "
-            "expression: %s",
+            "expression: %s (case %s)",
             f"closest {closest_neighbours}"
             if closest_neighbours is not None
             else "all",
@@ -162,13 +174,18 @@ def pdf_invoice_parsing():
             locator,
             pagenum,
             regexp,
+            "insensitive" if ignore_case else "sensitive",
+        )
+        self.set_anchor_to_element(
+            locator, trim=trim, pagenum=pagenum, ignore_case=ignore_case
         )
-        self.set_anchor_to_element(locator, trim=trim, pagenum=pagenum)
         if not self.anchor_element:
             self.logger.warning("No anchor(s) set for locator: %s", locator)
             return []
 
-        regexp_compiled = re.compile(regexp) if regexp else None
+        regexp_compiled = None
+        if regexp:
+            regexp_compiled = re.compile(regexp, flags=self._re_flags(ignore_case))
         search_for_candidate = self._get_candidate_search_function(
             direction, regexp_compiled, strict
         )
@@ -208,7 +225,11 @@ def pdf_invoice_parsing():
 
     @keyword
     def set_anchor_to_element(
-        self, locator: str, trim: bool = True, pagenum: Union[int, str] = 1
+        self,
+        locator: str,
+        trim: bool = True,
+        pagenum: Union[int, str] = 1,
+        ignore_case: bool = False,
     ) -> bool:
         """Sets main anchor point in the document for further searches.
 
@@ -216,13 +237,14 @@ def set_anchor_to_element(
         the same time if such are found.
 
         :param locator: Element to set anchor to. This can be prefixed with either
-            `text:`, `regex:` or `coords:` to find the anchor by text or coordinates.
-            `text` is assumed if no such prefix is specified. (text search is case
-            insensitive)
+            "text:", "subtext:", "regex:" or "coords:" to find the anchor by text or
+            coordinates. The "text" strategy is assumed if no such prefix is specified.
+            (text search is case-sensitive; use `ignore_case` param for controlling it)
         :param trim: Automatically trim leading/trailing whitespace from the text
             elements. (switched on by default)
         :param pagenum: Page number where search is performed on, defaults to 1 (first
             page).
+        :param ignore_case: Do a case-insensitive search when set to `True`.
         :returns: True if at least one anchor was found.
 
         **Examples**
@@ -247,7 +269,10 @@ def example_keyword():
         """
         pagenum = int(pagenum)
         self.logger.info(
-            "Trying to set anchor on page %d using locator: %r", pagenum, locator
+            "Trying to set anchor on page %d using locator: %r (case %s)",
+            pagenum,
+            locator,
+            "insensitive" if ignore_case else "sensitive",
         )
         self.ctx.convert(trim=trim, pagenum=pagenum)
         self._anchors.clear()
@@ -256,7 +281,7 @@ def example_keyword():
         pure_locator = locator
         criteria = "text"
         parts = locator.split(":", 1)
-        if len(parts) == 2 and parts[0] in ("coords", "text", "regex"):
+        if len(parts) == 2 and parts[0] in ("coords", "text", "regex", "subtext"):
             criteria = parts[0]
             pure_locator = parts[1]
 
@@ -279,10 +304,17 @@ def example_keyword():
             )
             anchor = TargetObject(bbox=bbox)
             self._anchors.append(anchor)
-        else:
+        else:  # text-based search
             if criteria == "regex":
-                pure_locator = re.compile(pure_locator)
-            anchors = self._find_matching_textboxes(pure_locator, pagenum=pagenum)
+                pure_locator = re.compile(
+                    pure_locator, flags=self._re_flags(ignore_case)
+                )
+            anchors = self._find_matching_textboxes(
+                pure_locator,
+                pagenum=pagenum,
+                is_subtext=criteria == "subtext",
+                ignore_case=ignore_case,
+            )
             self._anchors.extend(anchors)
 
         if self._anchors:
@@ -296,22 +328,34 @@ def _get_textboxes_on_page(self, pagenum: int) -> List[TextBox]:
         return list(page.textboxes.values())
 
     def _find_matching_textboxes(
-        self, locator: Union[str, Pattern], *, pagenum: int
+        self,
+        locator: Union[str, Pattern],
+        *,
+        pagenum: int,
+        is_subtext: bool = False,
+        ignore_case: bool = False,
     ) -> List[TextBox]:
         self.logger.info("Searching for matching text boxes with: %r", locator)
 
         if isinstance(locator, str):
-            lower_locator = locator.lower()
-            matches_anchor = (
-                lambda _anchor: _anchor.text.lower() == lower_locator
-            )  # noqa: E731
+            get_text = lambda string: (  # noqa: E731
+                string.lower() if ignore_case else string
+            )
+            if is_subtext:
+                matches_anchor = lambda _anchor: (  # noqa: E731
+                    get_text(locator) in get_text(_anchor.text)
+                )
+            else:
+                matches_anchor = lambda _anchor: (  # noqa: E731
+                    get_text(_anchor.text) == get_text(locator)
+                )
         else:
             matches_anchor = lambda _anchor: locator.match(_anchor.text)  # noqa: E731
+
         anchors = []
         for anchor in self._get_textboxes_on_page(pagenum):
             if matches_anchor(anchor):
                 anchors.append(anchor)
-
         if anchors:
             self.logger.info("Found %d matches with locator %r", len(anchors), locator)
             for anchor in anchors:
diff --git a/packages/pdf/src/RPA/PDF/keywords/model.py b/packages/pdf/src/RPA/PDF/keywords/model.py
--- a/packages/pdf/src/RPA/PDF/keywords/model.py
+++ b/packages/pdf/src/RPA/PDF/keywords/model.py
@@ -639,8 +639,7 @@ def example_keyword():
             ]
         except KeyError as err:
             raise KeyError(
-                'PDF "%s" does not have any input fields.'
-                % self.active_pdf_document.path
+                f"PDF {active_document.path!r} does not have any input fields."
             ) from err
 
         record_fields = {}
@@ -664,7 +663,7 @@ def example_keyword():
             }
             record_fields[name] = parsed_field
 
-        self.active_pdf_document.fields = record_fields or None
+        active_document.fields = record_fields or None
         return record_fields
 
     @keyword


Test patch:
diff --git a/packages/pdf/tests/python/__init__.py b/packages/pdf/tests/python/__init__.py
--- a/packages/pdf/tests/python/__init__.py
+++ b/packages/pdf/tests/python/__init__.py
@@ -30,6 +30,7 @@ class TestFiles:
     alianz_pdf = RESOURCE_DIR / "alianz.pdf"
     receipt_pdf = RESOURCE_DIR / "receipt.pdf"
     robot_png = RESOURCE_DIR / "robot.png"
+    camelot_table = RESOURCE_DIR / "camelot-table.pdf"
 
 
 @contextmanager
diff --git a/packages/pdf/tests/python/test_finder.py b/packages/pdf/tests/python/test_finder.py
--- a/packages/pdf/tests/python/test_finder.py
+++ b/packages/pdf/tests/python/test_finder.py
@@ -15,7 +15,7 @@
 )
 def test_set_anchor_to_element(library, locator, trim, expected):
     library.open_pdf(TestFiles.invoice_pdf)
-    result = library.set_anchor_to_element(locator, trim=trim)
+    result = library.set_anchor_to_element(locator, trim=trim, ignore_case=True)
 
     assert result is expected
 
@@ -33,7 +33,7 @@ def test_set_anchor_to_element(library, locator, trim, expected):
 )
 def test_find_text_default_right(library, locator, trim, expected):
     library.open_pdf(TestFiles.invoice_pdf)
-    result = library.find_text(locator, trim=trim)
+    result = library.find_text(locator, trim=trim, ignore_case=True)
 
     assert result[0].neighbours[0] == expected
 
@@ -47,7 +47,7 @@ def test_find_text_left(library):
 
 def test_find_text_bottom(library):
     library.open_pdf(TestFiles.invoice_pdf)
-    service = library.find_text("text:service", direction="bottom")
+    service = library.find_text("text:service", direction="bottom", ignore_case=True)
 
     assert "Web Design" in service[0].neighbours[0]
 
@@ -126,3 +126,23 @@ def test_find_text(
     assert re.match(
         expected_neighbour, found_neighbour, flags=re.DOTALL
     ), f"doesn't match pattern {expected_neighbour}"
+
+
+@pytest.mark.parametrize(
+    "locator, ignore_case, expected_anchor",
+    [
+        ("text:Distance \n(mi)", False, "Distance \n(mi)"),
+        ("subtext:Distance", False, "Distance \n(mi)"),
+        ("subtext:name", False, None),
+        ("subtext:name", True, "Cycle \nName"),
+    ],
+)
+def test_find_text_subtext(library, locator, ignore_case, expected_anchor):
+    library.open_pdf(TestFiles.camelot_table)
+    matches = library.find_text(locator, ignore_case=ignore_case)
+    if expected_anchor:
+        assert matches, "no results found"
+        match = matches[0]
+        assert match.anchor == expected_anchor
+    else:
+        assert not matches, "shouldn't find results"
diff --git a/packages/pdf/tests/python/test_model.py b/packages/pdf/tests/python/test_model.py
--- a/packages/pdf/tests/python/test_model.py
+++ b/packages/pdf/tests/python/test_model.py
@@ -6,6 +6,22 @@
 from . import TestFiles, temp_filename  # noqa
 
 
+def _escape_string(value: str) -> str:
+    to_escape = {"."}
+    values = []
+    for char in value:
+        if char in to_escape:
+            char = "\\\\" + oct(ord(char)).replace("o", "")
+        values.append(char)
+    return "".join(values)
+
+
+def assert_field_value(lib, name, value):
+    value = _escape_string(value)
+    content = lib.active_pdf_document.fileobject.read()
+    assert re.search(rf"{name}[^>]+{value}".encode(), content)
+
+
 @pytest.mark.parametrize(
     "trim,text",
     [
@@ -87,56 +103,30 @@ def test_set_field_value_checkbox(library):
         with pytest.raises(KeyError):
             # This output can't retrieve fields after save anymore.
             library.get_input_fields()
-        content = library.active_pdf_document.fileobject.read()
-        assert re.search(rf"{checkbox_name}[^>]+Yes".encode(), content)
+        assert_field_value(library, checkbox_name, "Yes")
 
 
-@pytest.mark.xfail(reason="Known issue: PDF won't show as having fields after saving")
-def test_save_field_values_fields_exist(library):
-    new_number = "12313123"
-    new_date = "01.04.2021"
-
-    with temp_filename() as tmp_file:
-        library.open_pdf(TestFiles.vero_pdf)
-        library.set_field_value("Puhelinnumero", new_number)
-        library.set_field_value("Paivays", new_date)
-        library.save_field_values(output_path=tmp_file)
-        fields = library.get_input_fields(tmp_file)
-
-        assert fields["Puhelinnumero"] == "12313123"
-        assert fields["Paivays"] == "01.04.2021"
-
-
-@pytest.mark.xfail(reason="Known issue: Field values won't show in text body")
-def test_save_field_values_text_exists(library):
-    new_number = "12313123"
-    new_date = "01.04.2021"
+@pytest.mark.parametrize("at_once", [False, True])
+def test_save_field_values_fields_exist(library, at_once):
+    fields = {
+        "Puhelinnumero": "12313123",  # new number
+        "Paivays": "01.04.2021",  # new date
+    }
 
-    with temp_filename() as tmp_file:
+    with temp_filename(suffix="-fields.pdf") as tmp_file:
         library.open_pdf(TestFiles.vero_pdf)
-        library.set_field_value("Puhelinnumero", new_number)
-        library.set_field_value("Paivays", new_date)
-        library.save_field_values(output_path=tmp_file)
-        text = library.get_text_from_pdf(tmp_file)
-
-        assert new_number in text[2]
-        assert new_date in text[2]
-
-
-@pytest.mark.xfail(reason="Known issue: PDF won't show as having fields after saving")
-def test_save_field_values_multiple_updates_in_one_operation(library):
-    new_fields = {
-        "Puhelinnumero": "12313123",
-        "Paivays": "01.04.2021",
-    }
-    with temp_filename() as tmp_file:
-        library.save_field_values(
-            source_path=TestFiles.vero_pdf, output_path=tmp_file, newvals=new_fields
-        )
-        fields = library.get_input_fields(tmp_file)
-
-        assert fields["Puhelinnumero"] == "12313123"
-        assert fields["Paivays"] == "01.04.2021"
+        # Keep non-empty values, because empty fields will fail the saving.
+        library.get_input_fields(replace_none_value=True)
+        if at_once:
+            for name, value in fields.items():
+                library.set_field_value(name, value)
+            library.save_field_values(output_path=tmp_file)
+        else:
+            library.save_field_values(output_path=tmp_file, newvals=fields)
+
+        library.open_pdf(tmp_file)
+        for name, value in fields.items():
+            assert_field_value(library, name, value)
 
 
 def test_dump_pdf_as_xml(library):
diff --git a/packages/pdf/tests/resources/camelot-table.pdf b/packages/pdf/tests/resources/camelot-table.pdf
new file mode 100644
Binary files /dev/null and b/packages/pdf/tests/resources/camelot-table.pdf differ
diff --git a/packages/pdf/tests/robot/test_pdf.robot b/packages/pdf/tests/robot/test_pdf.robot
--- a/packages/pdf/tests/robot/test_pdf.robot
+++ b/packages/pdf/tests/robot/test_pdf.robot
@@ -84,27 +84,28 @@ Extract pages from PDF
     &{item} =    Get Text From PDF    ${WORKDIR}${/}extract.pdf
     RPA Should Contain    ${item}[${1}]    4.5 Omaisuuden hankkimisesta aiheutuneet menot
 
-Get text closest to element on the right (default)
-    Open PDF    ${INVOICE_PDF}
-    ${items} =    Find Text    text:due date
+Get text closest to element on the right
+    [Setup]     Open PDF    ${INVOICE_PDF}
+    ${items} =    Find Text    text:due date    ignore_case=${True}
     Should Be Equal    ${items[0].neighbours}[0]    January 31, 2016
 
 Get text closest to element on the left
-    Open PDF    ${INVOICE_PDF}
+    [Setup]     Open PDF    ${INVOICE_PDF}
     ${items} =    Find Text    text:January 31, 2016    pagenum=1    direction=left
     Should Be Equal    ${items[0].neighbours}[0]    Due Date
 
 Get text closest to element using regexp match for value
-    Open PDF    ${INVOICE_PDF}
+    [Setup]     Open PDF    ${INVOICE_PDF}
     ${items} =    Find Text    text:Hrs/Qty    pagenum=1    direction=bottom    regexp=\\d+[.]\\d+
     Should Be Equal    ${items[0].neighbours}[0]    1.00
 
 Get figures from PDF
+    [Setup]     Open PDF    ${IMAGES_PDF}
+
     # In the PDF with images we get one figure for each page and each figure's `top`
-    # value is found among the ones below.
-    @{expect_tops} =    Create List    ${817}    ${587}
-    Open PDF    ${IMAGES_PDF}
+    #  value is found among the ones below.
     &{figures_by_page} =    Get All Figures
+    @{expect_tops} =    Create List    ${817}    ${587}
     FOR    ${page}    ${figures}    IN    &{figures_by_page}
         Log    On page: ${page}
         FOR    ${figure_id}    ${figure}    IN    &{figures}
@@ -143,19 +144,21 @@ XML Dumping And Parsing
     RPA Should Contain    ${text_dict}[${1}]    test@test.com
 
 Find multiple anchors in multi-page PDF
-    Open PDF    ${BOOK_PDF}
+    [Setup]     Open PDF    ${BOOK_PDF}
+
     @{all_matches} =    Create List
     ${pages} =    Get Number Of Pages
     FOR    ${page}    IN RANGE    1    ${pages + 1}
         ${matches} =    Find Text    regex:.*Python.*    pagenum=${page}    direction=down
         Append To List    ${all_matches}    @{matches}
     END
+
     # First text below first "Python" result.
     RPA Should Contain    ${all_matches[0].neighbours[0]}    Simple, Rapid, Effective, and Scalable
     # Second "Python" result's text.
     RPA Should Contain    ${all_matches[1].anchor}    13. A4. Packaging and Distributing Python Projects
     # Paragraph under the last "Python" match.
-    RPA Should Contain    ${all_matches[7].neighbours[0]}    Flask is another popular framework
+    RPA Should Contain    ${all_matches[8].neighbours[0]}    Flask is another popular framework
 
 Add watermark into PDF
     ${pdf} =    Set Variable    ${WORK_DIR}${/}receipt.pdf
@@ -172,7 +175,7 @@ Figures to Images
     File Should Exist    ${WORK_DIR}${/}Energy-*.bmp
 
 Figure to Image
-    Open Pdf    ${RESOURCE_DIR}${/}sparebin-receipt.pdf
+    [Setup]     Open Pdf    ${RESOURCE_DIR}${/}sparebin-receipt.pdf
     &{figures} =    Get All Figures
     Log Dictionary    ${figures}
     &{figure_dict} =    Get From Dictionary    ${figures}    ${2}    # page 2
@@ -184,3 +187,9 @@ Figure to Image
     ...    file_prefix=robot-
     Log To Console    ${image_file_path}
     File Should Exist    ${WORK_DIR}${/}robot-*.bmp
+
+Get Subtext From Invoice
+    [Setup]     Open PDF    ${INVOICE_PDF}
+
+    ${matches} =    Find Text   subtext:order   ignore_case=${True}
+    Should Be Equal     ${matches[0].neighbours[0]}     12345