Repository: scrapy/w3lib

Issue:
`canonicalize_url` use of `safe_url_string` breaks when an encoded hash character is encountered
`canonicalize_url` will decode all percent encoded elements in a string.

if a hash `#` is present as a percent-encoded entity (`%23`) it will be decoded... however it shouldn't be as it is a rfc delimiter and fundamentally changes the URL structure -- turning the subsequent characters into a fragment.   one of the effects is that a canonical url with a safely encoded hash will point to another url; another is that running the canonical on the output will return a different url.

example:

	>>> import w3lib.url
	>>> url = "https://example.com/path/to/foo%20bar%3a%20biz%20%2376%2c%20bang%202017#bash"
	>>> canonical = w3lib.url.canonicalize_url(url)
	>>> print canonical
	https://example.com/path/to/foo%20bar:%20biz%20#76,%20bang%202017
	>>> canonical2 = w3lib.url.canonicalize_url(canonical)
	>>> print canonical2
	https://example.com/path/to/foo%20bar:%20biz%20

what is presented as a fragment in "canonical": `#76,%20bang%202017` is part of the valid url - not a fragment - and is discarded when `canonicalize_url` is run again.

references:
* https://github.com/scrapy/w3lib/issues/5


Comments:

Patch:
diff --git a/w3lib/url.py b/w3lib/url.py
index 5feddab..a41446c 100644
--- a/w3lib/url.py
+++ b/w3lib/url.py
@@ -538,6 +538,8 @@ def canonicalize_url(
     # UTF-8 can handle all Unicode characters,
     # so we should be covered regarding URL normalization,
     # if not for proper URL expected by remote website.
+    if isinstance(url, str):
+        url = url.strip()
     try:
         scheme, netloc, path, params, query, fragment = _safe_ParseResult(
             parse_url(url), encoding=encoding or "utf8"


Test patch:
diff --git a/tests/test_url.py b/tests/test_url.py
index c7079c6..2356c2c 100644
--- a/tests/test_url.py
+++ b/tests/test_url.py
@@ -876,6 +876,12 @@ class CanonicalizeUrlTest(unittest.TestCase):
             "http://www.example.com/a%A3do?q=r%E9sum%E9",
         )
 
+        url = "https://example.com/a%23b%2cc#bash"
+        canonical = canonicalize_url(url)
+        # %23 is not accidentally interpreted as a URL fragment separator
+        self.assertEqual(canonical, "https://example.com/a%23b,c")
+        self.assertEqual(canonical, canonicalize_url(canonical))
+
     def test_normalize_percent_encoding_in_query_arguments(self):
         self.assertEqual(
             canonicalize_url("http://www.example.com/do?k=b%a3"),
@@ -1085,6 +1091,17 @@ class CanonicalizeUrlTest(unittest.TestCase):
             "http://www.example.com/path/to/%23/foo/bar?url=http%3A%2F%2Fwww.example.com%2F%2Fpath%2Fto%2F%23%2Fbar%2Ffoo#frag",
         )
 
+    def test_strip_spaces(self):
+        self.assertEqual(
+            canonicalize_url(" https://example.com"), "https://example.com/"
+        )
+        self.assertEqual(
+            canonicalize_url("https://example.com "), "https://example.com/"
+        )
+        self.assertEqual(
+            canonicalize_url(" https://example.com "), "https://example.com/"
+        )
+
 
 class DataURITests(unittest.TestCase):
     def test_default_mediatype_charset(self):