Repository: scrapy/scrapy

Issue:
Unicode Link Extractor
When using the following to extract all of the links from a response:
```
self.link_extractor = LinkExtractor()
...
links = self.link_extractor.extract_links(response)
```

On rare occasions, the following error is thrown:

```
2016-05-25 12:13:55,432 [root] [ERROR]  Error on http://detroit.curbed.com/2016/5/5/11605132/tiny-house-designer-show, traceback: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1203, in mainLoop
    self.runUntilCurrent()
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback
    self._startRunCallbacks(result)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/var/www/html/DomainCrawler/DomainCrawler/spiders/hybrid_spider.py", line 223, in parse
    items.extend(self._extract_requests(response))
  File "/var/www/html/DomainCrawler/DomainCrawler/spiders/hybrid_spider.py", line 477, in _extract_requests
    links = self.link_extractor.extract_links(response)
  File "/usr/local/lib/python2.7/site-packages/scrapy/linkextractors/lxmlhtml.py", line 111, in extract_links
    all_links.extend(self._process_links(links))
  File "/usr/local/lib/python2.7/site-packages/scrapy/linkextractors/__init__.py", line 103, in _process_links
    link.url = canonicalize_url(urlparse(link.url))
  File "/usr/local/lib/python2.7/site-packages/scrapy/utils/url.py", line 85, in canonicalize_url
    parse_url(url), encoding=encoding)
  File "/usr/local/lib/python2.7/site-packages/scrapy/utils/url.py", line 46, in _safe_ParseResult
    to_native_str(parts.netloc.encode('idna')),
  File "/usr/local/lib/python2.7/encodings/idna.py", line 164, in encode
    result.append(ToASCII(label))
  File "/usr/local/lib/python2.7/encodings/idna.py", line 73, in ToASCII
    raise UnicodeError("label empty or too long")
exceptions.UnicodeError: label empty or too long
```

I was able to find some information concerning the error from [here](http://stackoverflow.com/questions/25103126/label-empty-or-too-long-python-urllib2).
My question is: What is the best way to handle this? Even if there is one bad link in the response, I'd want all of the other good links to be extracted.

Discussion:

Patch:
diff --git a/scrapy/utils/url.py b/scrapy/utils/url.py
index c80fc6e70..406eb5843 100644
--- a/scrapy/utils/url.py
+++ b/scrapy/utils/url.py
@@ -41,9 +41,16 @@ def url_has_any_extension(url, extensions):
 
 
 def _safe_ParseResult(parts, encoding='utf8', path_encoding='utf8'):
+    # IDNA encoding can fail for too long labels (>63 characters)
+    # or missing labels (e.g. http://.example.com)
+    try:
+        netloc = parts.netloc.encode('idna')
+    except UnicodeError:
+        netloc = parts.netloc
+
     return (
         to_native_str(parts.scheme),
-        to_native_str(parts.netloc.encode('idna')),
+        to_native_str(netloc),
 
         # default encoding for path component SHOULD be UTF-8
         quote(to_bytes(parts.path, path_encoding), _safe_chars),


Test patch:
diff --git a/tests/test_utils_url.py b/tests/test_utils_url.py
index 1fc3a3510..b4819874d 100644
--- a/tests/test_utils_url.py
+++ b/tests/test_utils_url.py
@@ -265,6 +265,20 @@ class CanonicalizeUrlTest(unittest.TestCase):
             # without encoding, already canonicalized URL is canonicalized identically
             self.assertEqual(canonicalize_url(canonicalized), canonicalized)
 
+    def test_canonicalize_url_idna_exceptions(self):
+        # missing DNS label
+        self.assertEqual(
+            canonicalize_url(u"http://.example.com/résumé?q=résumé"),
+            "http://.example.com/r%C3%A9sum%C3%A9?q=r%C3%A9sum%C3%A9")
+
+        # DNS label too long
+        self.assertEqual(
+            canonicalize_url(
+                u"http://www.{label}.com/résumé?q=résumé".format(
+                    label=u"example"*11)),
+            "http://www.{label}.com/r%C3%A9sum%C3%A9?q=r%C3%A9sum%C3%A9".format(
+                    label=u"example"*11))
+
 
 class AddHttpIfNoScheme(unittest.TestCase):