Repository: falconry/falcon
Pull request #1537 (2019-05-07 20:14:07)

feat(media): add URL-encoded form data handler
~~Just dropping a quick sketch from PyCon sprints, this still a bit rough on edges.~~

We could probably polish documentation & recipes etc, but otherwise this is ready to review as in proposal for inclusion in Falcon 3.0 alpha1.

vytas7: Just dropping a quick sketch from PyCon sprints, this still a bit rough on edges.
codecov[bot]: # [Codecov](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=h1) Report
> Merging [#1537](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=desc) into [master](https://codecov.io/gh/falconry/falcon/commit/c90352e829bf021d0bb136279ff1ed2cc348b3fd?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/falconry/falcon/pull/1537/graphs/tree.svg?width=650&token=zOtkfNQ3h6&height=150&src=pr)](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1537   +/-   ##
======================================
  Coverage     100%    100%           
======================================
  Files          38      39    +1     
  Lines        2565    2582   +17     
  Branches      391     392    +1     
======================================
+ Hits         2565    2582   +17
```


| [Impacted Files](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [falcon/media/urlencoded.py](https://codecov.io/gh/falconry/falcon/pull/1537/diff?src=pr&el=tree#diff-ZmFsY29uL21lZGlhL3VybGVuY29kZWQucHk=) | `100% <100%> (ø)` | |
| [falcon/constants.py](https://codecov.io/gh/falconry/falcon/pull/1537/diff?src=pr&el=tree#diff-ZmFsY29uL2NvbnN0YW50cy5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [falcon/media/\_\_init\_\_.py](https://codecov.io/gh/falconry/falcon/pull/1537/diff?src=pr&el=tree#diff-ZmFsY29uL21lZGlhL19faW5pdF9fLnB5) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=footer). Last update [c90352e...6b2b1a0](https://codecov.io/gh/falconry/falcon/pull/1537?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

vytas7: @nZac it is so shockingly simple because we already had that functionality, albeit in a different form, and because it **is** simple :slightly_smiling_face:
Now, `multipart/form-data` is a completely different beast... and a really ugly one. I'm still wading through RFCs, and polishing something that at least could be demoed interface and functionality wise, although full correctness and improved performance are going to be challenging.
ch3pjw: This looks like it would be super-useful for a project I'm working on right now! Thanks!
vytas7: Implements #1580
vytas7: Thanks @kgriffs , merging then!

I'll create a follow-up PR to promote this to default handlers (if we want that?) and to deprecate the old parameter merge approach as per #1580 

Patch:
diff --git a/docs/_newsfragments/urlencoded-form-media-handler.feature.rst b/docs/_newsfragments/urlencoded-form-media-handler.feature.rst
new file mode 100644
index 0000000..9350c37
--- /dev/null
+++ b/docs/_newsfragments/urlencoded-form-media-handler.feature.rst
@@ -0,0 +1,2 @@
+``URLEncodedFormHandler`` was added for handling URL-encoded forms (of content
+type ``application/x-www-form-urlencoded``) as ``Request.media``.
diff --git a/docs/api/media.rst b/docs/api/media.rst
index 90c7d3e..06ec166 100644
--- a/docs/api/media.rst
+++ b/docs/api/media.rst
@@ -67,6 +67,7 @@ middleware. Here is an example of how this can be done:
         def process_request(self, req, resp):
             resp.content_type = req.accept
 
+.. _custom_media_handlers:
 
 Replacing the Default Handlers
 ------------------------------
@@ -121,6 +122,9 @@ Supported Handler Types
 .. autoclass:: falcon.media.MessagePackHandler
     :members:
 
+.. autoclass:: falcon.media.URLEncodedFormHandler
+    :members:
+
 Custom Handler Type
 -------------------
 
@@ -152,6 +156,7 @@ common media types, including the following:
 
     falcon.MEDIA_JSON
     falcon.MEDIA_MSGPACK
+    falcon.MEDIA_URLENCODED
     falcon.MEDIA_YAML
     falcon.MEDIA_XML
     falcon.MEDIA_HTML
diff --git a/docs/user/faq.rst b/docs/user/faq.rst
index 01fe018..24b02c4 100644
--- a/docs/user/faq.rst
+++ b/docs/user/faq.rst
@@ -529,7 +529,34 @@ the form parameters accessible via :attr:`~.Request.params`,
 
     api.req_options.auto_parse_form_urlencoded = True
 
-Alternatively, POSTed form parameters may be read directly from
+Alternatively, :class:`falcon.media.URLEncodedFormHandler` may be
+:ref:`installed <custom_media_handlers>` to handle the
+``application/x-www-form-urlencoded`` content type. The form parameters can
+then be simply accessed as :attr:`Request.media <falcon.Request.media>`:
+
+.. code:: python
+
+    import falcon
+    from falcon import media
+
+
+    handlers = media.Handlers({
+        falcon.MEDIA_JSON: media.JSONHandler(),
+        falcon.MEDIA_URLENCODED: media.URLEncodedFormHandler(),
+        # ... and any other request media handlers you may need
+    })
+
+    api = falcon.API()
+    api.req_options.media_handlers = handlers
+
+.. note::
+   Going forward, the :attr:`~RequestOptions.auto_parse_form_urlencoded` way to
+   access the submitted form may be deprecated in favor of the media handler in
+   further development versions of Falcon 3.0 series;
+   :class:`falcon.media.URLEncodedFormHandler` would then be installed by
+   default.
+
+POSTed form parameters may also be read directly from
 :attr:`~.Request.stream` and parsed via
 :meth:`falcon.uri.parse_query_string` or
 `urllib.parse.parse_qs() <https://docs.python.org/3.6/library/urllib.parse.html#urllib.parse.parse_qs>`_.
diff --git a/falcon/constants.py b/falcon/constants.py
index 1c01939..08db4f0 100644
--- a/falcon/constants.py
+++ b/falcon/constants.py
@@ -51,6 +51,8 @@ MEDIA_JSON = 'application/json'
 # but the use of the 'x-' prefix is discouraged by RFC 6838.
 MEDIA_MSGPACK = 'application/msgpack'
 
+MEDIA_URLENCODED = 'application/x-www-form-urlencoded'
+
 # NOTE(kgriffs): An internet media type for YAML has not been
 # registered. RoR uses 'application/x-yaml', but since use of
 # 'x-' is discouraged by RFC 6838, we don't use it in Falcon.
diff --git a/falcon/media/__init__.py b/falcon/media/__init__.py
index 44cb8d8..c598c8f 100644
--- a/falcon/media/__init__.py
+++ b/falcon/media/__init__.py
@@ -1,5 +1,6 @@
 from .base import BaseHandler  # NOQA
 from .json import JSONHandler  # NOQA
 from .msgpack import MessagePackHandler  # NOQA
+from .urlencoded import URLEncodedFormHandler  # NOQA
 
 from .handlers import Handlers  # NOQA
diff --git a/falcon/media/urlencoded.py b/falcon/media/urlencoded.py
new file mode 100644
index 0000000..95228fd
--- /dev/null
+++ b/falcon/media/urlencoded.py
@@ -0,0 +1,46 @@
+from urllib.parse import urlencode
+
+from falcon.media.base import BaseHandler
+from falcon.util.uri import parse_query_string
+
+
+class URLEncodedFormHandler(BaseHandler):
+    """
+    URL-encoded form data handler.
+
+    This handler parses ``application/x-www-form-urlencoded`` HTML forms to a
+    ``dict`` in a similar way that URL query parameters are parsed.
+
+    Keyword Arguments:
+        keep_blank (bool): Whether to keep empty-string values from the form
+            when deserializing.
+        csv (bool): Whether to split comma-separated form values into list
+            when deserializing.
+    """
+
+    def __init__(self, keep_blank=True, csv=False):
+        self.keep_blank = keep_blank
+        self.csv = csv
+
+    def serialize(self, media, content_type):
+        # NOTE(vytas): Setting doseq to True to mirror the parse_query_string
+        # behaviour.
+        return urlencode(media, doseq=True)
+
+    def deserialize(self, stream, content_type, content_length):
+        body = stream.read()
+
+        # NOTE(kgriffs): According to http://goo.gl/6rlcux the
+        # body should be US-ASCII. Enforcing this also helps
+        # catch malicious input.
+        body = body.decode('ascii')
+
+        # TODO(vytas): We are not short-circuiting here for performance (as
+        #   empty URL-encoded payload should not be a common case), but to work
+        #   around #1600
+        if not body:
+            return {}
+
+        return parse_query_string(body,
+                                  keep_blank=self.keep_blank,
+                                  csv=self.csv)


Test patch:
diff --git a/tests/test_media_urlencoded.py b/tests/test_media_urlencoded.py
new file mode 100644
index 0000000..6f05c2c
--- /dev/null
+++ b/tests/test_media_urlencoded.py
@@ -0,0 +1,70 @@
+import io
+
+import pytest
+
+import falcon
+from falcon import media
+from falcon import testing
+
+
+def test_deserialize_empty_form():
+    handler = media.URLEncodedFormHandler()
+    stream = io.BytesIO(b'')
+    assert handler.deserialize(stream, falcon.MEDIA_URLENCODED, 0) == {}
+
+
+def test_deserialize_invalid_unicode():
+    handler = media.URLEncodedFormHandler()
+    stream = io.BytesIO('spade=♠'.encode())
+    with pytest.raises(UnicodeDecodeError):
+        print(handler.deserialize(stream, falcon.MEDIA_URLENCODED, 9))
+
+
+@pytest.mark.parametrize('data,expected', [
+    ({'hello': 'world'}, 'hello=world'),
+    ({'number': [1, 2]}, 'number=1&number=2'),
+])
+def test_urlencoded_form_handler_serialize(data, expected):
+    handler = media.URLEncodedFormHandler()
+    assert handler.serialize(data, falcon.MEDIA_URLENCODED) == expected
+
+
+class MediaMirror:
+
+    def on_post(self, req, resp):
+        resp.media = req.media
+
+
+@pytest.fixture
+def client():
+    handlers = media.Handlers({
+        falcon.MEDIA_JSON: media.JSONHandler(),
+        falcon.MEDIA_URLENCODED: media.URLEncodedFormHandler(),
+    })
+    api = falcon.API()
+    api.req_options.media_handlers = handlers
+    api.resp_options.media_handlers = handlers
+
+    api.add_route('/media', MediaMirror())
+
+    return testing.TestClient(api)
+
+
+def test_empty_form(client):
+    resp = client.simulate_post(
+        '/media',
+        headers={'Content-Type': 'application/x-www-form-urlencoded'})
+    assert resp.content == b''
+
+
+@pytest.mark.parametrize('body,expected', [
+    ('a=1&b=&c=3', {'a': '1', 'b': '', 'c': '3'}),
+    ('param=undefined', {'param': 'undefined'}),
+    ('color=green&color=black', {'color': ['green', 'black']}),
+])
+def test_urlencoded_form(client, body, expected):
+    resp = client.simulate_post(
+        '/media',
+        body=body,
+        headers={'Content-Type': 'application/x-www-form-urlencoded'})
+    assert resp.json == expected