Repository: falconry/falcon

Issue:
falcon.to_query_str does not properly encode query parameters
# Version details

* Python version tested: 3.7.9
* Falcon version tested: 2.0.0

# Observed Behavior

`falcon.testing.TestClient`'s `simulate_*` methods do not correctly encode query parameters when using the `params` keyword argument.

```python
import urllib.parse

import falcon
import falcon.testing
import falcon.util


class SomeHandler:
    def on_get(self, request, response):
        print("parsed params       ", request.params)


app = falcon.API()
app.add_route("/", SomeHandler())

test_client = falcon.testing.TestClient(app)


# Try a params dict with the TestClient. Note that it does not encode any special characters, and this results in a
# query string that does not decode back to the same parameters.
#
# This is because falcon.testing uses falcon.util.to_query_str, which does not apply any encoding to the supplied query
# params dict.
params = {"foo": "bar", "baz": "boz&something"}
print("original params dict", params)
print("to_query_str        ", falcon.util.to_query_str(params))
response = test_client.simulate_get("/", params=params)
print(response.status_code)
# => original params dict {'foo': 'bar', 'baz': 'boz&something'}
# => to_query_str         ?foo=bar&baz=boz&something
# => parsed params        {'foo': 'bar', 'baz': 'boz', 'something': ''}
# => 200

# Now try manually encoding the params dict ourselves with urllib. Note that the query string is then correctly
# interpreted by falcon.
params = {"foo": "bar", "baz": "boz&something"}
encoded_params = urllib.parse.urlencode(params)
print("encoded params      ", encoded_params)
response = test_client.simulate_get("/", query_string=encoded_params)
print(response.status_code)
# => encoded params       foo=bar&baz=boz%26something
# => parsed params        {'foo': 'bar', 'baz': 'boz&something'}
# => 200
```

# Expected Behavior

I expect a WSGI app testing library that supports a way to specify query parameters as a dict to correctly encode my parameters so that the application interprets them the same way (although it is reasonable to require both sides to agree on how to handle lists as CSV vs repeating a query parameter)

I think that merely documenting this shortcoming is insufficient because the argument specifically exists to allow me to pass in query parameters as a dict.


# Root Cause

This is caused by `falcon.util.misc.to_query_str` not applying any URL encoding to the query parameters as it constructs the query string. `to_query_str` looks like it is currently only used by `falcon.testing.simulate_request`.


Discussion:
Hi :wave:,

Thanks for using Falcon. The large amount of time and effort needed to
maintain the project and develop new features is not sustainable without
the generous financial support of community members like you.

Please consider helping us secure the future of the Falcon framework with a
one-time or recurring donation.

Thank you for your support!

Hi @aetherknight !
Indeed, that's embarrassing.

As you demonstrated, It affects not only the testing interface, but also the public utility function [`to_query_str`](https://falcon.readthedocs.io/en/stable/api/util.html#falcon.to_query_str):
```python
>>> falcon.to_query_str({'foo': 'bar', 'baz': 'bar&bar&baz'})
'?foo=bar&baz=bar&bar&baz'
```

We'll try to rectify this for the impending 3.0 release.

Unless you want to contribute a PR @aetherknight, that would be very welcome too :wink: 
The behaviour the same also for the keys.

```py
>>> falcon.to_query_str({'a&b':'3'})
'?a&b=3'
```

(not sure if that key would be valid in general though)
Hm... :thinking: 
IMHO, while at it, we should also make a breaking change in the function's signature, switching the default value of `comma_delimited_lists` to `False` in order to better match the csv-related default values elsewhere in the framework (in line with https://github.com/falconry/falcon/issues/1730 etc).

Thoughts?

Resolving patch:
diff --git a/falcon/util/misc.py b/falcon/util/misc.py
--- a/falcon/util/misc.py
+++ b/falcon/util/misc.py
@@ -34,6 +34,7 @@
 
 from falcon import status_codes
 from falcon.constants import PYPY
+from falcon.uri import encode_value
 # NOTE(vytas): Hoist `deprecated` here since it is documented as part of the
 # public Falcon interface.
 from .deprecation import deprecated
@@ -230,7 +231,7 @@ def to_query_str(params, comma_delimited_lists=True, prefix=True):
             v = 'false'
         elif isinstance(v, list):
             if comma_delimited_lists:
-                v = ','.join(map(str, v))
+                v = ','.join(map(encode_value, map(str, v)))
             else:
                 for list_value in v:
                     if list_value is True:
@@ -238,15 +239,15 @@ def to_query_str(params, comma_delimited_lists=True, prefix=True):
                     elif list_value is False:
                         list_value = 'false'
                     else:
-                        list_value = str(list_value)
+                        list_value = encode_value(str(list_value))
 
-                    query_str += k + '=' + list_value + '&'
+                    query_str += encode_value(k) + '=' + list_value + '&'
 
                 continue
         else:
-            v = str(v)
+            v = encode_value(str(v))
 
-        query_str += k + '=' + v + '&'
+        query_str += encode_value(k) + '=' + v + '&'
 
     return query_str[:-1]
 


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -211,6 +211,21 @@ def test_pack_query_params_several(self):
 
         assert expected == garbage_out
 
+    @pytest.mark.parametrize('csv', [True, False])
+    @pytest.mark.parametrize('params', [
+        {'a & b': 'a and b', 'b and c': 'b & c'},
+        {'apples and oranges': '🍏 & 🍊'},
+        {'garbage': ['&', '&+&', 'a=1&b=2', 'c=4&'], 'one': '1'},
+        {'&': '&amp;', '™': '&trade;', '&&&': ['&amp;', '&amp;', '&amp;']},
+
+        # NOTE(vytas): Would fail because of https://github.com/falconry/falcon/issues/1872
+        # {'&': '%26', '&&': '%26', '&&&': ['%26', '%2', '%']},
+    ])
+    def test_to_query_str_encoding(self, params, csv):
+        query_str = falcon.to_query_str(params, comma_delimited_lists=csv, prefix=False)
+
+        assert uri.parse_query_string(query_str, csv=csv) == params
+
     def test_uri_encode(self):
         url = 'http://example.com/v1/fizbit/messages?limit=3&echo=true'
         assert uri.encode(url) == url