Repository: getsentry/responses

Issue:
Issues with Unicode in path
I saw that you fixed #153 by handling Unicode characters in the querystring, but there still appear to be issues when said Unicode characters are in the URL path itself. If you pass a mock URL with a Unicode character in it to Responses and then request the same URL via Requests, you get a connection refused error.

[This is the line](https://github.com/getsentry/responses/blob/master/responses.py#L166) where it fails, when I stepped through the debugger.

I'm using Python 2.7.13 and Responses 0.8.0. Please let me know if I need to clarify this issue. Thanks!

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=h1) Report
> Merging [#176](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=desc) into [master](https://codecov.io/gh/getsentry/responses/commit/92b2d4597e03c8dee096329fca2ed7f28c8fefa5?src=pr&el=desc) will **increase** coverage by `0.75%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/getsentry/responses/pull/176/graphs/tree.svg?token=x2tAGR7rxE&width=650&height=150&src=pr)](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #176      +/-   ##
==========================================
+ Coverage   95.06%   95.81%   +0.75%     
==========================================
  Files           1        1              
  Lines         304      311       +7     
==========================================
+ Hits          289      298       +9     
+ Misses         15       13       -2
```


| [Impacted Files](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [responses.py](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=tree#diff-cmVzcG9uc2VzLnB5) | `95.81% <100%> (+0.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=footer). Last update [92b2d45...9a3cfd3](https://codecov.io/gh/getsentry/responses/pull/176?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/responses.py b/responses.py
index c2a3280..1c48310 100644
--- a/responses.py
+++ b/responses.py
@@ -23,8 +23,9 @@ except ImportError:
 
 if six.PY2:
     from urlparse import urlparse, parse_qsl, urlsplit, urlunsplit
+    from urllib import quote
 else:
-    from urllib.parse import urlparse, parse_qsl, urlsplit, urlunsplit
+    from urllib.parse import urlparse, parse_qsl, urlsplit, urlunsplit, quote
 
 if six.PY2:
     try:
@@ -53,6 +54,21 @@ def _is_string(s):
     return isinstance(s, six.string_types)
 
 
+def _has_unicode(s):
+    return any(ord(char) > 128 for char in s)
+
+
+def _clean_unicode(url):
+    if isinstance(url.encode('utf8'), six.string_types):
+        url = url.encode('utf8')
+    chars = list(url)
+    for i, x in enumerate(chars):
+        if ord(x) > 128:
+            chars[i] = quote(x)
+
+    return ''.join(chars)
+
+
 def _is_redirect(response):
     try:
         # 2.0.0 <= requests <= 2.2
@@ -173,22 +189,18 @@ class BaseResponse(object):
             return False
 
         for (a_k, a_v), (b_k, b_v) in zip(url_qsl, other_qsl):
-            if not isinstance(a_k, six.text_type):
-                a_k = a_k.decode('utf-8')
-            if not isinstance(b_k, six.text_type):
-                b_k = b_k.decode('utf-8')
             if a_k != b_k:
                 return False
-            if not isinstance(a_v, six.text_type):
-                a_v = a_v.decode('utf-8')
-            if not isinstance(b_v, six.text_type):
-                b_v = b_v.decode('utf-8')
             if a_v != b_v:
                 return False
         return True
 
     def _url_matches(self, url, other, match_querystring=False):
         if _is_string(url):
+            if _has_unicode(url):
+                url = _clean_unicode(url)
+                if not isinstance(other, six.text_type):
+                    other = other.encode('ascii').decode('utf8')
             if match_querystring:
                 return self._url_matches_strict(url, other)
             else:


Test patch:
diff --git a/test_responses.py b/test_responses.py
index 6965a1c..a7c3ed2 100644
--- a/test_responses.py
+++ b/test_responses.py
@@ -648,7 +648,7 @@ def test_allow_redirects_samehost():
     assert_reset()
 
 
-def test_handles_chinese_url():
+def test_handles_unicode_querystring():
     url = u'http://example.com/test?type=2&ie=utf8&query=汉字'
 
     @responses.activate
@@ -663,6 +663,21 @@ def test_handles_chinese_url():
     assert_reset()
 
 
+def test_handles_unicode_url():
+    url = u'https://hi.wikipedia.org/wiki/दिलवाले_दुल्हनिया_ले_जाएंगे'
+
+    @responses.activate
+    def run():
+        responses.add(responses.GET, url, body='test')
+
+        resp = requests.get(url)
+
+        assert_response(resp, 'test')
+
+    run()
+    assert_reset()
+
+
 def test_headers():
     @responses.activate
     def run():