Repository: scrapy/scrapy

Issue:
Request serialization should fail for non-picklable objects
The Pickle-based disk queues silently serialize requests that shouldn't be serialized in Python<=3.5. I found this problem when dumping a request with an `ItemLoader` object in its `meta` dict. Python 3.6 fails in [this line](https://github.com/scrapy/scrapy/blob/1.4/scrapy/squeues.py#L27) with `TypeError: can't pickle HtmlElement objects`, because the loader contains a `Selector`, which in turns contains an `HtmlElement` object.

I tested this using the https://github.com/scrapinghub/scrapinghub-stack-scrapy repository, and found that `pickle.loads(pickle.dumps(selector))` doesn't fail, but generates a broken object.

#### Python 2.7, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3)
```
root@04bfc6cf84cd:/# scrapy version -v
Scrapy    : 1.3.3
lxml      : 3.7.2.0
libxml2   : 2.9.3
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 16.6.0
Python    : 2.7.14 (default, Dec 12 2017, 16:55:09) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10
root@04bfc6cf84cd:/# scrapy shell "http://example.org"
2017-12-29 16:49:27 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
(...)
>>> from six.moves import cPickle as pickle
>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))
>>> response.selector.css('a')
[<Selector xpath=u'descendant-or-self::a' data=u'<a href="http://www.iana.org/domains/exa'>]
>>> s2.css('a')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/parsel/selector.py", line 227, in css
    return self.xpath(self._css2xpath(query))
  File "/usr/local/lib/python2.7/site-packages/parsel/selector.py", line 203, in xpath
    **kwargs)
  File "src/lxml/lxml.etree.pyx", line 1584, in lxml.etree._Element.xpath (src/lxml/lxml.etree.c:59349)
  File "src/lxml/xpath.pxi", line 257, in lxml.etree.XPathElementEvaluator.__init__ (src/lxml/lxml.etree.c:170478)
  File "src/lxml/apihelpers.pxi", line 19, in lxml.etree._assertValidNode (src/lxml/lxml.etree.c:16482)
AssertionError: invalid Element proxy at 140144569743064
```


#### Python 3.5, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3-py3)
```
root@1945e2154919:/# scrapy version -v
Scrapy    : 1.3.3
lxml      : 3.7.2.0
libxml2   : 2.9.3
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 16.6.0
Python    : 3.5.4 (default, Dec 12 2017, 16:43:39) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10
root@1945e2154919:/# scrapy shell "http://example.org"
2017-12-29 16:52:37 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
(...)
>>> from six.moves import cPickle as pickle
>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))
>>> response.selector.css('a')
[<Selector xpath='descendant-or-self::a' data='<a href="http://www.iana.org/domains/exa'>]
>>> s2.css('a')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/parsel/selector.py", line 227, in css
    return self.xpath(self._css2xpath(query))
  File "/usr/local/lib/python3.5/site-packages/parsel/selector.py", line 203, in xpath
    **kwargs)
  File "src/lxml/lxml.etree.pyx", line 1584, in lxml.etree._Element.xpath (src/lxml/lxml.etree.c:59349)
  File "src/lxml/xpath.pxi", line 257, in lxml.etree.XPathElementEvaluator.__init__ (src/lxml/lxml.etree.c:170478)
  File "src/lxml/apihelpers.pxi", line 19, in lxml.etree._assertValidNode (src/lxml/lxml.etree.c:16482)
AssertionError: invalid Element proxy at 139862544625976
```


#### Python 3.6, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3-py3)
```
root@43e690443ca7:/# scrapy version -v
Scrapy    : 1.3.3
lxml      : 3.7.2.0
libxml2   : 2.9.3
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 16.6.0
Python    : 3.6.4 (default, Dec 21 2017, 01:35:12) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10
root@43e690443ca7:/# scrapy shell "http://example.org"
2017-12-29 16:54:49 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
(...)
>>> from six.moves import cPickle as pickle
>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: can't pickle HtmlElement objects
```


Discussion:

Resolving patch:
diff --git a/scrapy/squeues.py b/scrapy/squeues.py
--- a/scrapy/squeues.py
+++ b/scrapy/squeues.py
@@ -25,9 +25,10 @@ def pop(self):
 def _pickle_serialize(obj):
     try:
         return pickle.dumps(obj, protocol=2)
-    # Python>=3.5 raises AttributeError here while
-    # Python<=3.4 raises pickle.PicklingError
-    except (pickle.PicklingError, AttributeError) as e:
+    # Python <= 3.4 raises pickle.PicklingError here while
+    # 3.5 <= Python < 3.6 raises AttributeError and
+    # Python >= 3.6 raises TypeError
+    except (pickle.PicklingError, AttributeError, TypeError) as e:
         raise ValueError(str(e))
 
 PickleFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -71,7 +71,7 @@ def has_environment_marker_platform_impl_support():
         'pyOpenSSL',
         'cssselect>=0.9',
         'six>=1.5.2',
-        'parsel>=1.1',
+        'parsel>=1.4',
         'PyDispatcher>=2.0.5',
         'service_identity',
     ],


Test patch:
diff --git a/tests/test_squeues.py b/tests/test_squeues.py
--- a/tests/test_squeues.py
+++ b/tests/test_squeues.py
@@ -5,6 +5,7 @@
 from scrapy.item import Item, Field
 from scrapy.http import Request
 from scrapy.loader import ItemLoader
+from scrapy.selector import Selector
 
 class TestItem(Item):
     name = Field()
@@ -17,20 +18,22 @@ class TestLoader(ItemLoader):
     name_out = staticmethod(_test_procesor)
 
 def nonserializable_object_test(self):
+    q = self.queue()
     try:
         pickle.dumps(lambda x: x)
     except Exception:
         # Trigger Twisted bug #7989
         import twisted.persisted.styles  # NOQA
-        q = self.queue()
         self.assertRaises(ValueError, q.push, lambda x: x)
     else:
         # Use a different unpickleable object
         class A(object): pass
         a = A()
         a.__reduce__ = a.__reduce_ex__ = None
-        q = self.queue()
         self.assertRaises(ValueError, q.push, a)
+    # Selectors should fail (lxml.html.HtmlElement objects can't be pickled)
+    sel = Selector(text='<html><body><p>some text</p></body></html>')
+    self.assertRaises(ValueError, q.push, sel)
 
 class MarshalFifoDiskQueueTest(t.FifoDiskQueueTest):