Repository: scrapy/scrapy

Issue:
Exception in LxmLinkExtractor.extract_links 'charmap' codec can't encode character 
```
Stacktrace (most recent call last):

  File "scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "scrapy/spidermiddlewares/offsite.py", line 28, in process_spider_output
    for x in result:
  File "scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "scrapy/spidermiddlewares/offsite.py", line 28, in process_spider_output
    for x in result:
  File "scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "scrapy/spidermiddlewares/depth.py", line 54, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "scrapy/spiders/crawl.py", line 69, in _parse_response
    for requests_or_item in iterate_spider_output(cb_res):
  File "ex_link_crawl/spiders/external_link_spider.py", line 45, in parse_obj
    for link in LxmlLinkExtractor(allow=(), deny=self.allowed_domains).extract_links(response):
  File "scrapy/linkextractors/lxmlhtml.py", line 108, in extract_links
    links = self._extract_links(doc, response.url, response.encoding, base_url)
  File "scrapy/linkextractors/__init__.py", line 103, in _extract_links
    return self.link_extractor._extract_links(*args, **kwargs)
  File "scrapy/linkextractors/lxmlhtml.py", line 57, in _extract_links
    url = url.encode(response_encoding)
  File "encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)

```

My use of extractor is following:

```
def parse_obj(self, response):
        if not isinstance(response, HtmlResponse):
            return
        for link in LxmlLinkExtractor(allow=(), deny=self.allowed_domains).extract_links(response):
            if not link.nofollow:
                yield LinkCrawlItem(domain=link.url)
```

HTML entity causes UnicodeEncodeError in LxmlLinkExtractor
A page containing &amp;#9001; causes with error in LxmlLinkExtractor.link_extractor()

```
          File "/[path]/spiders/search_spider.py", line 112, in parse
            self.link_extractor.extract_links(response) if
          File "/[path to virtualenv]/local/lib/python2.7/site-packages/scrapy/contrib/linkextractors/lxmlhtml.py", line 107, in extract_links
            links = self._extract_links(doc, response.url, response.encoding, base_url)
          File "/[path to virtualenv]/local/lib/python2.7/site-packages/scrapy/linkextractor.py", line 94, in _extract_links
            return self.link_extractor._extract_links(*args, **kwargs)
          File "/[path to virtualenv]/local/lib/python2.7/site-packages/scrapy/contrib/linkextractors/lxmlhtml.py", line 57, in _extract_links
            url = url.encode(response_encoding)
          File "/[path to virtualenv]/lib/python2.7/encodings/cp1252.py", line 12, in encode
            return codecs.charmap_encode(input,errors,encoding_table)
        exceptions.UnicodeEncodeError: 'charmap' codec can't encode character u'\u2329' in position 87: character maps to <undefined>
```

The encoding is correct and works for displaying the page in a browser, but fails as above in Scrapy.

Add encoding to Link object + re-enable HTML entities links tests
Add `encoding` parameter and attribute to `Link` object.
That's the only way I found to properly account for encoding information when building requests from link extractors.

Supersedes #1880 
Should fix #1403



Discussion:
There is a test for it, but it is disabled for LxmlLinkExtractor. I've marked it as xfail here: https://github.com/scrapy/scrapy/pull/1461; apparently it passes in Python 3.

It passes in python 3 because `to_native_str` at  https://github.com/scrapy/scrapy/blob/8dc400c66d2001a28cd3b717355e5e09a5da7b18/scrapy/linkextractors/lxmlhtml.py#L59
doesn't encode the link to the response's encoding.

It's a case where lxml manages to extract a unicode link with symbols that the response's encoding doesn't support. But urls need to be all ascii.
How does it get properly encoded before fetching it in python3?
I tried `to_unicode(url, response_encoding)` and the test passes both on 2 and 3
but on python2 `Link` expects bytes so it encodes unicode urls to utf8 with a warning.
Why bytes in python2 and unicode in python3?
If encoding is meant to be handled by the downloader, shouldn't all be unicode?
If such abstraction was not meant at all, shouldn't it all be bytes?
This all looks like there's an abstraction on the handling of url encoding
that is handled in the extractor in python2 and elsewhere in python3.

> Why bytes in python2 and unicode in python3?

As I recall, the main reason is that in Python 2 urllib/urlparse work on bytes, while in Python 3 they work on unicode. Also, we can't make urls unicode in Python 2 because it is inconvenient and backwards incompatible, and urls as bytes are hard to work with in Python 3 (urllib.parse doesn't work with bytes, you have to pass original encoding everywhere if you work with bytes, selectors and link extractors return unicode).

Non-ascii URL encoding is tricker than one may think: domain should be encoded using IDNA, path should be encoded to UTF8 and then escaped to ASCII, query should be encoded to response encoding and then escaped to ASCII - this is how browsers work. 

We haven't really solved that. In the Python 3 port we decided that URLs should be bytes in Python 2 and unicode in Python 3 and updated all code accordingly, marking 'hard' test cases as xfail to unblock further porting.

I don't understand how urlparse doesn't work with bytes in In python2:
`urlparse.urlparse(u'http://\N{SNOWFLAKE}.com/\N{SNOWMAN}')`
`ParseResult(scheme=u'http', netloc=u'\u2744.com', path=u'/\u2603', params='', query='', fragment='')`

Regarding the backwards compatibility,
where exactly is it dropped by a transition to unicode urls?

Regarding the final encoding of the url before downloading,
requiring links to be encodable to lesser encodings
shouldn't be a necessary step
because the path can be encoded to utf8 + %escapes later.

> I don't understand how urlparse doesn't work with bytes in In python2

I think we've seen other functions from urllib and urlparse modules which don't work correctly; sorry, no concrete examples right now.

> Regarding the backwards compatibility, where exactly is it dropped by a transition to unicode urls?

Besides obvious data type differences (other stuff may be unexpectingly promoted to unicode) the difference is that with urls-as-bytes users should encode them, but with urls-as-unicode scrapy encodes them, and the result may be different.

> requiring links to be encodable to lesser encodings shouldn't be a necessary step 

Sorry, I didn't get what you mean; could you please clarify it?

> because the path can be encoded to utf8 + %escapes later.

The problem is that here `/path?query` path should be encoded to utf8 and then %-escaped (even if an URL is extracted from a non-utf8 page), but query should be encoded to response encoding and then %-escaped (even if path was encoded to utf8).

One such example: http://stackoverflow.com/questions/16614695/python-urlparse-parse-qs-unicode-url

another example: `urllib.quote(u'привет')`

> > requiring links to be encodable to lesser encodings shouldn't be a necessary step
> 
> Sorry, I didn't get what you mean; could you please clarify it?

In the link extractor, unicode objects are encoded with `url.encode(response_encoding)`
then in the downloader they are decoded again, I suppose with `url.decode(response_encoding)`
and the path finally to utf8, after parsing, `parsed_url.path.encode('utf8')`

Unless I misunderstood and what you explained about unicode urls is not done in scrapy
and the request is done with a path encoded to the responses's encoding.

I have a workaroud:

```
cleaned_response = response.replace(
    body=response.body_as_unicode().encode('utf-32'),
    encoding='utf-32'
)
```

and then pass cleaned_response to link_extactor()

I hope it means a fix will be straightforward.

I am using Scrapy 0.24.4, by the way.

Can you give us a snippet to reproduce it?

Here is a snippet to reproduce:
file: snowman.html

``` html
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=Windows-1251"/>
  </head>
  <body>
    <a title="unicode snowman" href="./&#9731;">
      &#x0441;&#x043d;&#x0435;&#x0436;&#x0435;&#x043d;
      &#x0447;&#x043e;&#x0432;&#x0435;&#x043a;
    </a>
  </body>
</html>
```

~~run: `scrapy shell ~/snowman.html`~~
The above doesn't work (probably some bug `utils.url`),
on 1.1 neither `scrapy shell file:///home/digenis/snowman.html` does.
Only
`scrapy shell file://localhost/home/digenis/snowman.html`

``` python
from scrapy.linkextractors import LinkExtractor
LinkExtractor().extract_links(response)
```

 `UnicodeEncodeError: 'charmap' codec can't encode character u'\u2603' in position 24: character maps to <undefined>`

lxml decodes the entity inside the href attribute
and since`Windows-1251` doesn't have the :snowman: character
it can't be decoded "back" to cp1251.

As @kmike suggests in https://github.com/scrapy/scrapy/issues/1403#issuecomment-149194449
the path part needs to be encoded to utf8, as browsers do.
It should need to be able to be encoded in the response's encoding

This issue is fixed in #1949 where Link objects are created with Unicode URLs (and [not converted to native str anymore](https://github.com/scrapy/scrapy/pull/1949/files#diff-952f1c77a1cfacb08eed58f08daa8870L59), which chokes in Python 2),
leaving the encoding to the Request' init (when safe_url_string is called, path encoded as UTF-8 and query/fragment as response/document encoding) or when canonicalize_url (on by default) kicks in

cf. `test_restrict_xpaths_with_html_entities`
If you open this HTML in a browser

```
<html>

<head>
  <meta http-equiv="content-type" content="text/html;charset=iso8859-15" />
</head>

<body>
  <p><a href="http://www.example.com/&hearts;/you?c=&euro;">text</a></p>
</body>

</html>
```

you'll see `http://www.example.com/%E2%99%A5/you?c=%A4` being sent when clicked

## [Current coverage](https://codecov.io/gh/scrapy/scrapy/pull/1949?src=pr) is **81.65%**

> Merging [#1949](https://codecov.io/gh/scrapy/scrapy/pull/1949?src=pr) into [master](https://codecov.io/gh/scrapy/scrapy/branch/master?src=pr) will decrease coverage by **-0.00%**

``` diff
@@             master      #1949   diff @@
==========================================
  Files           161        161          
  Lines          8669       8669          
  Methods           0          0          
  Messages          0          0          
  Branches       1269       1269          
==========================================
- Hits           7080       7078     -2   
  Misses         1233       1233          
- Partials        356        358     +2   
```
1. 2 files (not in diff) in `scrapy/core` were modified. [more](https://codecov.io/gh/scrapy/scrapy/commit/a4dbf7e936b2565475808a1fc46c78b4f9f5c4e4/changes?src=pr#7363726170792F636F7265) 
   - Partials `+2` 
   - Hits `-2`

[![Sunburst](https://codecov.io/gh/scrapy/scrapy/pull/1949/graphs/sunburst.svg?src=pr&size=200)](https://codecov.io/gh/scrapy/scrapy/pull/1949?src=pr)

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by a4dbf7e

@kmike , with `safe_url_string` removed from link extraction (https://github.com/scrapy/scrapy/pull/1949/commits/54f1c2419e96c832932a9125f719b791bdd2694c), for Python2 Unicode URL input case in `Link` init, one actually needs UTF8 for `to_bytes`, otherwise `test_restrict_xpaths_with_html_entities` even fails ("iso8859-15" can only encode part of the Unicode characters)

What happens with `to_bytes(url, encoding='utf8')` in `Link.__init__()` is illustrated below:

```
Py2 warning: u'http://example.org/\u2665/you?c=\u20ac'
--> 'http://example.org/\xe2\x99\xa5/you?c=\xe2\x82\xac'

(link extraction's) canonicalize_url() gets this:
('http://example.org/\xe2\x99\xa5/you?c=\xe2\x82\xac', 'iso8859-15')

which goes through _safe_ParseResult(),
after parse_url('http://example.org/\xe2\x99\xa5/you?c=\xe2\x82\xac', None)

(None actually saves the whole thing, decoding UTF8)

the rest is fine after that:
ParseResult(scheme=u'http', netloc=u'example.org', path=u'/\u2665/you', params='', query=u'c=\u20ac', fragment='')

_safe_ParseResult --> 'http', 'example.org', '/%E2%99%A5/you', '', 'c=%A4', ''
...
```


Resolving patch:
diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.py
--- a/scrapy/linkextractors/lxmlhtml.py
+++ b/scrapy/linkextractors/lxmlhtml.py
@@ -5,11 +5,11 @@
 
 import lxml.etree as etree
 from w3lib.html import strip_html5_whitespace
-from w3lib.url import canonicalize_url
+from w3lib.url import canonicalize_url, safe_url_string
 
 from scrapy.link import Link
 from scrapy.utils.misc import arg_to_iter, rel_has_nofollow
-from scrapy.utils.python import unique as unique_list, to_unicode
+from scrapy.utils.python import unique as unique_list
 from scrapy.utils.response import get_base_url
 from scrapy.linkextractors import FilteringLinkExtractor
 
@@ -66,7 +66,7 @@ def _extract_links(self, selector, response_url, response_encoding, base_url):
                 url = self.process_attr(attr_val)
                 if url is None:
                     continue
-            url = to_unicode(url, encoding=response_encoding)
+            url = safe_url_string(url, encoding=response_encoding)
             # to fix relative links after process_value
             url = urljoin(response_url, url)
             link = Link(url, _collect_string_content(el) or u'',


Test patch:
diff --git a/tests/test_linkextractors.py b/tests/test_linkextractors.py
--- a/tests/test_linkextractors.py
+++ b/tests/test_linkextractors.py
@@ -2,8 +2,6 @@
 import unittest
 from warnings import catch_warnings
 
-import pytest
-
 from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import HtmlResponse, XmlResponse
 from scrapy.link import Link
@@ -16,7 +14,6 @@
 class Base:
     class LinkExtractorTestCase(unittest.TestCase):
         extractor_cls = None
-        escapes_whitespace = False
 
         def setUp(self):
             body = get_testdata('link_extractor', 'linkextractor.html')
@@ -30,10 +27,7 @@ def test_urls_type(self):
 
         def test_extract_all_links(self):
             lx = self.extractor_cls()
-            if self.escapes_whitespace:
-                page4_url = 'http://example.com/page%204.html'
-            else:
-                page4_url = 'http://example.com/page 4.html'
+            page4_url = 'http://example.com/page%204.html'
 
             self.assertEqual([link for link in lx.extract_links(self.response)], [
                 Link(url='http://example.com/sample1.html', text=u''),
@@ -214,7 +208,7 @@ def test_restrict_xpaths_with_html_entities(self):
             response = HtmlResponse("http://example.org/somepage/index.html", body=html, encoding='iso8859-15')
             links = self.extractor_cls(restrict_xpaths='//p').extract_links(response)
             self.assertEqual(links,
-                             [Link(url='http://example.org/%E2%99%A5/you?c=%E2%82%AC', text=u'text')])
+                             [Link(url='http://example.org/%E2%99%A5/you?c=%A4', text=u'text')])
 
         def test_restrict_xpaths_concat_in_handle_data(self):
             """html entities cause SGMLParser to call handle_data hook twice"""
@@ -310,10 +304,7 @@ def test_base_url_with_restrict_xpaths(self):
 
         def test_attrs(self):
             lx = self.extractor_cls(attrs="href")
-            if self.escapes_whitespace:
-                page4_url = 'http://example.com/page%204.html'
-            else:
-                page4_url = 'http://example.com/page 4.html'
+            page4_url = 'http://example.com/page%204.html'
 
             self.assertEqual(lx.extract_links(self.response), [
                 Link(url='http://example.com/sample1.html', text=u''),
@@ -506,7 +497,6 @@ def test_link_restrict_text(self):
             Link(url='http://example.org/item2.html', text=u'Pic of a dog', nofollow=False),
         ])
 
-    @pytest.mark.xfail
     def test_restrict_xpaths_with_html_entities(self):
         super(LxmlLinkExtractorTestCase, self).test_restrict_xpaths_with_html_entities()