Repository: Pylons/colander

Issue:
Add data URLs to the URL validator
Colander provides a [`colander.url`](https://docs.pylonsproject.org/projects/colander/en/latest/api.html#colander.url) regex that validates only http/s and ftp/s schemes:

https://github.com/Pylons/colander/blob/048fb24eeb6c3df21831413943dbf89d7b5776e4/src/colander/__init__.py#L610-L629

A [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs) would be rejected as invalid. Any interest in adding support for data URLs? Or should this be handled by the user by using the [`colander.Any()`](https://docs.pylonsproject.org/projects/colander/en/latest/api.html#colander.Any) validator with two separate Regex validators?

Also, I think it would be nice if the documentation would say “A Regex validator which…” (minor nit, though).

Discussion:
stevepiercy: Shame on me. @jenstroeger I forgot to ask you to sign `CONTRIBUTORS.txt` in the other PR, but we can do that in this one.
stevepiercy: A rebase on master or merging master into this branch should fix the CI.
stevepiercy: This LGTM. I've requested a review from any maintainer so they can cut a release after approval and merge. Thank you!

@mcdonc @ericof @miohtama @mmerickel @tseaver
jenstroeger: Thank you @stevepiercy. Any thoughts on the `msg` parameters for the `DataURL` class? And should I add the locales, or not bother for now?
stevepiercy: Regarding `msg`, I think when more than one part of the data URL is not valid, then each invalid part should return its error message and that each message should be customizable. Else the user could potentially need to submit their data up to three times, each time returning a different error message.

For locales, see [my comment on the linked issue](https://github.com/Pylons/colander/issues/347#issuecomment-1171655997). I am not sure what to use for an unknown language's translation string. I assume that if the translation has an empty string or `None`, then the default is used, but don't hold me to that. Someone more familiar with the package [`translationstring`](https://github.com/pylons/translationstring) would know.
jenstroeger: Ok on `msg` and I’ll adjust the code accordingly, similar to how `All` handles multiple failures in a single raised exception:

https://github.com/Pylons/colander/blob/a1c525c1f172dc43bd82ba62bcb5521f853a760c/src/colander/__init__.py#L260-L263

Looks like `translationstring` wraps around [`gettext`](https://docs.python.org/3/library/gettext.html) which uses the translation catalogues in [locale/](https://github.com/Pylons/colander/tree/master/src/colander/locale). In that case, missing translation strings simply default (in this case to English) but I’ll take a closer look before touching that part…
jenstroeger: Actually, I think the current implementation is somewhat cumbersome because both MIME type and Base64 data will be handled _twice_ — first by the validator, and then again by the user who unpacks the URL string.

With that in mind it probably makes more sense to implement `DataURL` as a [Type](https://docs.pylonsproject.org/projects/colander/en/latest/api.html#types) (maybe sub-classing `String`) such that
```python
data_url = colander.SchemaNode(
    colander.DataURL(),
)
```
validates the `data_url` and assigns the data to it. Or we leave `data_url` alone and the type _adds_ `data_url_mimetype` and `data_url_data` to the `node` as part of its deserialization?
stevepiercy: > Actually, I think the current implementation is somewhat cumbersome because both MIME type and Base64 data will be handled _twice_ — first by the validator, and then again by the user who unpacks the URL string.
>
> With that in mind it probably makes more sense to implement `DataURL` as a [Type](https://docs.pylonsproject.org/projects/colander/en/latest/api.html#types) (maybe sub-classing `String`) such that
> 
> ```python
> data_url = colander.SchemaNode(
>     colander.DataURL(),
> )
> ```
> 
> validates the `data_url` and assigns the data to it. Or we leave `data_url` alone and the type _adds_ `data_url_mimetype` and `data_url_data` to the `node` as part of its deserialization?

Can you explain why the user needs to unpack the data URL string?

If valid, I would store the data URL as submitted. If not valid, I don't really need to present the part's value that the user entered as an error message, e.g., "Bad MIME type: 'foo/bar'". I would display a single concatenated error message. "Data URL not valid. MIME type not valid. Data not valid."
jenstroeger: > Can you explain why the user needs to unpack the data URL string?

If the data URL is valid then the user will take that URL and unpack the data from it. If the data URL is something like
```
data:image/png;base64,abcde...xyz==
```
then we have two potentially expensive passes over that one URL string:
1. the validator
   - looks up all MIME types
      ```python
       mime not in mimetypes.types_map.values()
       ```
   - decodes the data to make sure it’s valid Base64
       ```python
        base64.standard_b64decode(data)
        ```
2. the user then does the same again to unpack the data from the URL.

If we handle the data URL as a `Type` rather than using a validator I think we might be able to manage both, validation and data unpacking in one go.

Patch:
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
index 80d9573..9b02033 100644
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -143,3 +143,4 @@ Contributors
 - Damian Dimmich, 2020/07/19
 - Keith M Franklin, 2020/07/30
 - Scott Maki, 2020/11/15
+- Jens Troeger, 2022/07/03
diff --git a/docs/api.rst b/docs/api.rst
index eea26b2..c03d845 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -83,6 +83,8 @@ Validators
 
   .. autoclass:: Email
 
+  .. autoclass:: DataURL
+
   .. autofunction:: luhnok
 
   .. attribute:: url
diff --git a/src/colander/__init__.py b/src/colander/__init__.py
index 7b578c7..189d1bd 100644
--- a/src/colander/__init__.py
+++ b/src/colander/__init__.py
@@ -1,9 +1,11 @@
 # coding=utf-8
+import base64
 import copy
 import datetime
 import decimal
 import functools
 import itertools
+import mimetypes
 import pprint
 import re
 import translationstring
@@ -403,6 +405,67 @@ class Email(Regex):
         super(Email, self).__init__(email_regex, msg=msg)
 
 
+# Regex for data URLs, loosely based on MDN:
+# https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs
+DATA_URL_REGEX = (
+    # data: (required)
+    r'^data:'
+    # optional mime type
+    r'([^;]*)?'
+    # optional base64 identifier
+    r'(;base64)?'
+    # actual data follows the comma
+    r',(.*)$'
+)
+
+
+class DataURL(Regex):
+    """A data URL validator. If ``url_msg`` is supplied, it will be the error
+    message to be used when raising :exc:`colander.Invalid` for a syntactically
+    incorrect data URL, defaults to 'Not a data URL'. If, however, the data
+    URL string is syntactically correct but contains an invalid MIME type,
+    ``mimetype_err`` is raised (defaults to 'Invalid MIME type'); for incorrectly
+    encoded Base64 data, ``base64_err`` is raised (defaults to  'Invalid Base64
+    encoded data').
+    """
+
+    _URL_ERR = _("Not a data URL")
+    _MIMETYPE_ERR = _("Invalid MIME type")
+    _BASE64_ERR = _("Invalid Base64 encoded data")
+
+    def __init__(
+        self,
+        url_err=_URL_ERR,
+        mimetype_err=_MIMETYPE_ERR,
+        base64_err=_BASE64_ERR,
+    ):
+        data_url_regex = text_(DATA_URL_REGEX)
+        self.url_err = url_err
+        self.mimetype_err = mimetype_err
+        self.base64_err = base64_err
+        super(DataURL, self).__init__(
+            data_url_regex, msg=url_err, flags=re.IGNORECASE
+        )
+
+    def __call__(self, node, value):
+        match_ = self.match_object.match(value)
+        if match_ is None:
+            raise Invalid(node, self.url_err)
+        excs = []
+        mime, is_base64_data, data = match_.groups()
+        if mime and mime not in mimetypes.types_map.values():
+            excs.append(self.mimetype_err)
+        try:
+            if is_base64_data:
+                base64.standard_b64decode(data)
+        except Exception:
+            excs.append(self.base64_err)
+        if len(excs) == 1:
+            raise Invalid(node, excs[0])
+        elif len(excs) == 2:
+            raise Invalid(node, excs)
+
+
 class Range(object):
     """Validator which succeeds if the value it is passed is greater
     or equal to ``min`` and less than or equal to ``max``.  If ``min``


Test patch:
diff --git a/tests/test_colander.py b/tests/test_colander.py
index 4be0336..9316b8f 100644
--- a/tests/test_colander.py
+++ b/tests/test_colander.py
@@ -506,6 +506,68 @@ class TestEmail(unittest.TestCase):
         self.assertRaises(Invalid, validator, None, 'name1,name2@here.info')
 
 
+class TestDataURL(unittest.TestCase):
+    def _makeOne(self):
+        from colander import DataURL
+
+        return DataURL()
+
+    def test_valid_data_urls(self):
+        validator = self._makeOne()
+        self.assertEqual(validator(None, 'data:,'), None)
+        self.assertEqual(validator(None, 'data:,foo'), None)
+        self.assertEqual(validator(None, 'data:;base64,'), None)
+        self.assertEqual(validator(None, 'data:;base64,Zm9vCg=='), None)
+        self.assertEqual(validator(None, 'data:text/plain,foo'), None)
+        self.assertEqual(
+            validator(None, 'data:text/plain;base64,Zm9vCg=='), None
+        )
+        self.assertEqual(validator(None, 'data:text/plain,foo%20bar'), None)
+        self.assertEqual(validator(None, 'data:text/plain,%F0%9F%A4%93'), None)
+
+    def test_invalid_data_urls(self):
+        validator = self._makeOne()
+        msg = 'Not a data URL'
+        e = invalid_exc(validator, None, '')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'foo')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'data:foo')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'data:;base64')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'data:;base32,')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'data:text/plain;charset=ASCII,foo')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(
+            validator, None, 'data:text/plain;charset=ASCII;base64,Zm9vCg=='
+        )
+        self.assertEqual(e.msg, msg)
+
+    def test_invalid_mimetypes(self):
+        validator = self._makeOne()
+        msg = 'Invalid MIME type'
+        e = invalid_exc(validator, None, 'data:no/mime,foo')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'data:no-mime;base64,Zm9vCg==')
+        self.assertEqual(e.msg, msg)
+
+    def test_invalid_base64_data(self):
+        validator = self._makeOne()
+        msg = 'Invalid Base64 encoded data'
+        e = invalid_exc(validator, None, 'data:;base64,Zm9vCg')
+        self.assertEqual(e.msg, msg)
+        e = invalid_exc(validator, None, 'data:text/plain;base64,Zm*vCg==')
+        self.assertEqual(e.msg, msg)
+
+    def test_invalid_mimetypes_and_base64(self):
+        validator = self._makeOne()
+        msg = ['Invalid MIME type', 'Invalid Base64 encoded data']
+        e = invalid_exc(validator, None, 'data:no/mime;base64,Zm9vCg')
+        self.assertEqual(e.msg, msg)
+
+
 class TestLength(unittest.TestCase):
     def _makeOne(self, **kw):
         from colander import Length