Repository: thp/urlwatch

Issue:
Feature request: Extension of regex filtering to extract data
I couldn't find a way, how to use `re.sub` to extract data as a filter in urlwatch. Unless I haven't overseen anything, and there is a way to do it, I would like to open a feature request.

According to #401 I would like to request an extensions of the regex filtering. Instead of replacing matched strings, it would be useful to have a positive regex filter, to extract specific data. An example of this filtering is even in the [documentation](https://urlwatch.readthedocs.io/en/latest/filters.html#using-a-shell-script-as-a-filter). However there it is implemented using shellpipes and grep, but for me it turned out to be a bit unstable.

```
url: https://example.net/shellpipe-grep.txt
filter:
  - shellpipe: "grep -i -o 'price: <span>.*</span>'"
```

IMO better approach would be to have a filter like `re.findall` which goes along with `re.sub`:

```
url: https://example.net/pricelist.html
filter:
  - re.findall: 'price: <span>(.*)</span>'
```

Discussion:
Jamstah: Draft because it needs documentation.
Jamstah: Yeah, I was thinking that but was too lazy to think of a good example :)

I've borrowed something from another test, and added some doc.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2c4288b..93bb7f2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,8 @@ The format mostly follows [Keep a Changelog](http://keepachangelog.com/en/1.0.0/
 - Command line options to enable and disbale jobs (Requested in #813 by gapato, contributed in #820 by jamstah)
 - New option `ignore_incomplete_reads` (Requested in #725 by wschoot, contributed in #787 by wfrisch)
 - New option `wait_for` in browser jobs (Requested in #763 by yuis-ice, contributed in #810 by jamstah)
+- Added tags to jobs and the ability to select them at the command line (#789 by jamstah)
+- New filter `re.findall` (Requested in #804 by f0sh, contributed in #805 by jamstah)
 - Added tags to jobs and the ability to select them at the command line (#789, #824 by jamstah)
 
 ### Changed
diff --git a/docs/source/filters.rst b/docs/source/filters.rst
index 65148d0..c153de8 100644
--- a/docs/source/filters.rst
+++ b/docs/source/filters.rst
@@ -77,6 +77,7 @@ At the moment, the following filters are built-in:
 - **ical2text**: Convert `iCalendar`_ to plaintext
 - **ocr**: Convert text in images to plaintext using Tesseract OCR
 - **re.sub**: Replace text with regular expressions using Python's re.sub
+- **re.findall**: Find all non-overlapping matches using Python's re.findall
 - **reverse**: Reverse input items
 - **sha1sum**: Calculate the SHA-1 checksum of the content
 - **shellpipe**: Filter using a shell command
@@ -485,12 +486,13 @@ Alternatively, ``jq`` can be used for filtering:
    filter:
      - jq: '.[0].name'
 
-Remove or replace text using regular expressions
-------------------------------------------------
+Find, remove or replace text using regular expressions
+------------------------------------------------------
 
-Just like Python’s ``re.sub`` function, there’s the possibility to apply
-a regular expression and either remove of replace the matched text. The
-following example applies the filter 3 times:
+You can use ``re.sub`` and ``re.findall`` to apply regular expressions.
+
+``re.sub`` can be used to remove or replace all non-overlapping instances
+of matched text. The following example applies the filter 3 times:
 
 1. Just specifying a string as the value will replace the matches with
    the empty string.
@@ -499,11 +501,7 @@ following example applies the filter 3 times:
 3. You can use groups (``()``) and back-reference them with ``\1``
    (etc..) to put groups into the replacement string.
 
-All features are described in Python’s
-`re.sub <https://docs.python.org/3/library/re.html#re.sub>`__
-documentation (the ``pattern`` and ``repl`` values are passed to this
-function as-is, with the value of ``repl`` defaulting to the empty
-string).
+``repl`` defaults to the empty string, which will remove matched strings.
 
 .. code:: yaml
 
@@ -517,15 +515,42 @@ string).
            pattern: '</([^>]*)>'
            repl: '<END OF TAG \1>'
 
-If you want to enable certain flags (e.g. ``re.MULTILINE``) in the
-call, this is possible by inserting an "inline flag" documented in
-`flags in re.compile`_, here are some examples:
+``re.findall`` can be used to find all non-overlapping matches of a
+regular expression. Each match is output on its own line. The following
+example applies the filter twice:
+
+1. It uses a group (``()``) and back-reference (``\1``) to extract a
+   date from the input string.
+2. It breaks the numbers in the date out into separate lines.
+
+If ``repl`` is not specified, the full match will be included in the output.
+
+.. code:: yaml
+
+   url: https://example.com/regex-findall.html
+   filter:
+       - re.findall:
+           pattern: 'The next draw is on (\d{4}-\d{2}-\d{2}).'
+           repl: '\1'
+       - re.findall: '\d+'
+
+Note: When using HTML or XML, it is usually better to use CSS selectors or
+XPATH expressions. HTML and XML `cannot be parsed`_ properly using regular
+expressions. If the CSS selector or XPATH cannot provide the targeted
+selection required, using an ``html2text`` filter first then using
+``re.findall`` can be a good pattern.
+
+.. _`cannot be parsed`: https://stackoverflow.com/a/1732454/1047040
+
+If you want to enable flags (e.g. ``re.MULTILINE``) in ``re.sub``
+or ``re.findall`` filters, use an "inline flag", here are some
+examples:
 
 * ``re.MULTILINE``: ``(?m)`` (Makes ``^`` match start-of-line and ``$`` match end-of-line)
 * ``re.DOTALL``: ``(?s)`` (Makes ``.`` also match a newline)
 * ``re.IGNORECASE``: ``(?i)`` (Perform case-insensitive matching)
 
-.. _flags in re.compile: https://docs.python.org/3/library/re.html#re.compile
+.. _full re syntax: https://docs.python.org/3/library/re.html#regular-expression-syntax
 
 This allows you, for example, to remove all leading spaces (only
 space character and tab):
diff --git a/lib/urlwatch/filters.py b/lib/urlwatch/filters.py
index ed21b4c..e4fd25a 100644
--- a/lib/urlwatch/filters.py
+++ b/lib/urlwatch/filters.py
@@ -848,6 +848,26 @@ class RegexSub(FilterBase):
         return re.sub(subfilter['pattern'], subfilter.get('repl', ''), data)
 
 
+class RegexFindall(FilterBase):
+    """Pick out regular expressions using Python's re.findall"""
+
+    __kind__ = 're.findall'
+
+    __supported_subfilters__ = {
+        'pattern': 'Regular expression to search for (required)',
+        'repl': 'Replacement string (default: full match)',
+    }
+
+    __default_subfilter__ = 'pattern'
+
+    def filter(self, data, subfilter):
+        if 'pattern' not in subfilter:
+            raise ValueError('{} needs a pattern'.format(self.__kind__))
+
+        # Default: Replace with full match if no "repl" value is set
+        return "\n".join(match.expand(subfilter.get('repl', '\\g<0>')) for match in re.finditer(subfilter['pattern'], data))
+
+
 class SortFilter(FilterBase):
     """Sort input items"""
 


Test patch:
diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml
index 496c958..2026af7 100644
--- a/.github/workflows/unit-tests.yml
+++ b/.github/workflows/unit-tests.yml
@@ -32,7 +32,8 @@ jobs:
         python -m pip install --upgrade pip
         python -m pip install --requirement requirements.txt
         python -m pip install pycodestyle==2.6.0 pytest
-        sudo apt install -y build-essential libpoppler-cpp-dev pkg-config python3-dev tesseract-ocr
+        sudo apt-get update
+        sudo apt-get install -y build-essential libpoppler-cpp-dev pkg-config python3-dev tesseract-ocr
         python -m pip install pdftotext docutils pygments pytesseract pillow jq
     - name: Test with pytest
       run: pytest -v
diff --git a/lib/urlwatch/tests/data/filter_documentation_testdata.yaml b/lib/urlwatch/tests/data/filter_documentation_testdata.yaml
index cdcf8df..cbaf509 100644
--- a/lib/urlwatch/tests/data/filter_documentation_testdata.yaml
+++ b/lib/urlwatch/tests/data/filter_documentation_testdata.yaml
@@ -285,6 +285,19 @@ https://example.com/regex-substitute.html:
       HEADING 1: Welcome to this webpage<END OF TAG h1>
       <a>Some Link<END OF TAG a>
     <END OF TAG div>
+https://example.com/regex-findall.html:
+  input: |-
+    Welcome to the lottery webpage.
+    The numbers for 2020-07-11 are:
+
+       4, 8, 15, 16, 23 and 42
+
+    The next draw is on 2020-07-13.
+    Thank you for visiting the lottery webpage.
+  output: |-
+    2020
+    07
+    13
 https://example.net/shellpipe-grep.txt:
   input: |-
     <h1>Welcome to our price watching page!</h1>
diff --git a/lib/urlwatch/tests/data/filter_tests.yaml b/lib/urlwatch/tests/data/filter_tests.yaml
index 081cf5e..72629d2 100644
--- a/lib/urlwatch/tests/data/filter_tests.yaml
+++ b/lib/urlwatch/tests/data/filter_tests.yaml
@@ -326,6 +326,32 @@ re_sub_multiline:
     One Line
     
     Another Line
+re_findall:
+  filter:
+    - re.findall: '-[a-z][a-z][a-z]-'
+  data: |-
+    Some-abc-things-def-on-ghi-this-line-and
+    some-jkl-more-mno-here
+  expected_result: |-
+    -abc-
+    -def-
+    -ghi-
+    -jkl-
+    -mno-
+re_findall_repl:
+  filter:
+    - re.findall:
+        pattern: '-([a-z])([a-z])([a-z])-'
+        repl: '\3\2\1'
+  data: |-
+    Some-abc-things-def-on-ghi-this-line-and
+    some-jkl-more-mno-here
+  expected_result: |-
+    cba
+    fed
+    ihg
+    lkj
+    onm
 strip:
   filter: strip
   data: "  The rose is red;   \n\nthe violet's blue.\nSugar is sweet,       \nand so are you.   "