internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: scrapy/scrapy

Issue:
Change the scrapy short description
The Scrapy short description says:

> Scrapy, a fast high-level screen scraping and web crawling framework.

I think would be better:

> Scrapy, a fast high-level web crawling and screen scraping framework.

Because it highlights first its difference with simple screen scraping tools (i.e. Nokogiri. Mechanize, etc).

Screen scraping can be done even with curl and grep, but I don't think you could do web crawling with such simple tools.

Perhaps this can be an alternative:

> Scrapy, a fast and scalable web crawling and screen scraping framework.

Also the term "web data mining" can be a good fit for Scrapy (along with Scrapely and similar tools) and help to shape its roadmap.



Discussion:
@kmike You been working on the docs, is this still useful? Or should we close it?

I think that @darkrho suggestions are useful. Both look fine for me:

> Scrapy, a fast high-level web crawling and screen scraping framework.
> Scrapy, a fast and scalable web crawling and screen scraping framework.

+1 to change.

There is a problem with existing tutorial: it doesn't really show "web crawling" part of Scrapy. It only demonstrates how to scrape a static list of URLs - curl can also do that. But that's another issue :)

I was discussing doc changes with @eliasdorneles, and it became clear "screen scraping" is confusing. Wikipedia [article](http://en.wikipedia.org/wiki/Data_scraping#Screen_scraping) thinks it is something related to OCR, and the term is not generally well-known. I haven't heard it until I started to work with Scrapy :) 

What about using "data extraction" instead?
Thoughts?

Web Scraping is a better term than Screen Scraping for what Scrapy does. However, I think with "web crawling and data extraction" we also have it covered.

@pablohoffman made a good point: if we move scrapy.selectors to a separate library (https://github.com/scrapy/scrapy/pull/1007) scrapy itself won't be focused on data extraction. So +1 to Web Scraping.


Resolving patch:
diff --git a/scrapy/__init__.py b/scrapy/__init__.py
--- a/scrapy/__init__.py
+++ b/scrapy/__init__.py
@@ -1,5 +1,5 @@
 """
-Scrapy - a web crawling and screen scraping framework written for Python
+Scrapy - a web crawling and web scraping framework written for Python
 """
 
 __all__ = ['__version__', 'version_info', 'optional_features', 'twisted_version',
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,7 @@
     name='Scrapy',
     version=version,
     url='http://scrapy.org',
-    description='A high-level Web Crawling and Screen Scraping framework',
+    description='A high-level Web Crawling and Web Scraping framework',
     long_description=open('README.rst').read(),
     author='Scrapy developers',
     maintainer='Pablo Hoffman',


Test patch:
Repository: scrapy/scrapy

Issue:
Deprecate htmlparser link extractor
Let's add a deprecation warning like we did for SGML Link extractor



Discussion:

Resolving patch:
diff --git a/scrapy/linkextractors/htmlparser.py b/scrapy/linkextractors/htmlparser.py
--- a/scrapy/linkextractors/htmlparser.py
+++ b/scrapy/linkextractors/htmlparser.py
@@ -2,6 +2,7 @@
 HTMLParser-based link extractor
 """
 
+import warnings
 from HTMLParser import HTMLParser
 from six.moves.urllib.parse import urljoin
 
@@ -9,12 +10,20 @@
 
 from scrapy.link import Link
 from scrapy.utils.python import unique as unique_list
+from scrapy.exceptions import ScrapyDeprecationWarning
+
 
 class HtmlParserLinkExtractor(HTMLParser):
 
     def __init__(self, tag="a", attr="href", process=None, unique=False):
         HTMLParser.__init__(self)
 
+        warnings.warn(
+            "HtmlParserLinkExtractor is deprecated and will be removed in "
+            "future releases. Please use scrapy.linkextractors.LinkExtractor",
+            ScrapyDeprecationWarning, stacklevel=2,
+        )
+
         self.scan_tag = tag if callable(tag) else lambda t: t == tag
         self.scan_attr = attr if callable(attr) else lambda a: a == attr
         self.process_attr = process if callable(process) else lambda v: v
diff --git a/scrapy/linkextractors/sgml.py b/scrapy/linkextractors/sgml.py
--- a/scrapy/linkextractors/sgml.py
+++ b/scrapy/linkextractors/sgml.py
@@ -21,7 +21,7 @@ def __init__(self, tag="a", attr="href", unique=False, process_value=None):
         warnings.warn(
             "BaseSgmlLinkExtractor is deprecated and will be removed in future releases. "
             "Please use scrapy.linkextractors.LinkExtractor",
-            ScrapyDeprecationWarning
+            ScrapyDeprecationWarning, stacklevel=2,
         )
         SGMLParser.__init__(self)
         self.scan_tag = tag if callable(tag) else lambda t: t == tag
@@ -104,7 +104,7 @@ def __init__(self, allow=(), deny=(), allow_domains=(), deny_domains=(), restric
         warnings.warn(
             "SgmlLinkExtractor is deprecated and will be removed in future releases. "
             "Please use scrapy.linkextractors.LinkExtractor",
-            ScrapyDeprecationWarning
+            ScrapyDeprecationWarning, stacklevel=2,
         )
 
         tags, attrs = set(arg_to_iter(tags)), set(arg_to_iter(attrs))


Test patch:
Repository: scrapy/scrapy

Issue:
[MRG+1] 0 delay inside utils/defer.py->defer_succeed makes reactor not to be async enough
This will make scrapy to be more async by letting the reactor spend extra cycles to accomplish its needs.



Discussion:

Resolving patch:
diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py
--- a/scrapy/utils/defer.py
+++ b/scrapy/utils/defer.py
@@ -8,19 +8,25 @@
 from scrapy.exceptions import IgnoreRequest
 
 def defer_fail(_failure):
-    """Same as twisted.internet.defer.fail, but delay calling errback until
+    """Same as twisted.internet.defer.fail but delay calling errback until
     next reactor loop
+
+    It delays by 100ms so reactor has a chance to go trough readers and writers
+    before attending pending delayed calls, so do not set delay to zero.
     """
     d = defer.Deferred()
-    reactor.callLater(0, d.errback, _failure)
+    reactor.callLater(0.1, d.errback, _failure)
     return d
 
 def defer_succeed(result):
-    """Same as twsited.internet.defer.succed, but delay calling callback until
+    """Same as twisted.internet.defer.succeed but delay calling callback until
     next reactor loop
+
+    It delays by 100ms so reactor has a chance to go trough readers and writers
+    before attending pending delayed calls, so do not set delay to zero.
     """
     d = defer.Deferred()
-    reactor.callLater(0, d.callback, result)
+    reactor.callLater(0.1, d.callback, result)
     return d
 
 def defer_result(result):


Test patch:
Repository: scrapy/scrapy

Issue:
to stop CrawlerProcess started with stop_after_crawl=False two Ctrl-Cs are required
When CrawlerProcess is started with `stop_after_crawl=True` (default) and user presses Ctrl-C the following happens:
1. Ctrl-C signal handler is changed to "unclean shutown" so that second Ctrl-C will definitely stop the reactor;
2. `self.stop` method is called which asks all crawlers to stop;
3. when all crawlers are stopped a Deferred returned from `self.join()` executes its callback. This callback is set to "stop reactor", so the reactor stops.

But with `crawler_process.start(stop_after_crawl=False)` there is no step (3) because `self.join` is not used. It means that to stop CrawlerProcess user have to send signal (press Ctrl-C) twice. The first Ctrl-C only asks all current crawlers to stop.

I think it is better to change it - a signal should mean "graceful shutdown" regardless of `stop_after_crawl` option.



Discussion:

Resolving patch:
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -216,7 +216,7 @@ def _signal_shutdown(self, signum, _):
         signame = signal_names[signum]
         logger.info("Received %(signame)s, shutting down gracefully. Send again to force ",
                     {'signame': signame})
-        reactor.callFromThread(self.stop)
+        reactor.callFromThread(self._graceful_stop_reactor)
 
     def _signal_kill(self, signum, _):
         install_shutdown_handlers(signal.SIG_IGN)
@@ -242,16 +242,30 @@ def start(self, stop_after_crawl=True):
             # Don't start the reactor if the deferreds are already fired
             if d.called:
                 return
-            d.addBoth(lambda _: self._stop_reactor())
+            d.addBoth(self._stop_reactor)
 
-        cache_size = self.settings.getint('DNSCACHE_SIZE') if self.settings.getbool('DNSCACHE_ENABLED') else 0
-        reactor.installResolver(CachingThreadedResolver(reactor, cache_size,
-                                                            self.settings.getfloat('DNS_TIMEOUT')))
+        reactor.installResolver(self._get_dns_resolver())
         tp = reactor.getThreadPool()
         tp.adjustPoolsize(maxthreads=self.settings.getint('REACTOR_THREADPOOL_MAXSIZE'))
         reactor.addSystemEventTrigger('before', 'shutdown', self.stop)
         reactor.run(installSignalHandlers=False)  # blocking call
 
+    def _get_dns_resolver(self):
+        if self.settings.getbool('DNSCACHE_ENABLED'):
+            cache_size = self.settings.getint('DNSCACHE_SIZE')
+        else:
+            cache_size = 0
+        return CachingThreadedResolver(
+            reactor=reactor,
+            cache_size=cache_size,
+            timeout=self.settings.getfloat('DNS_TIMEOUT')
+        )
+
+    def _graceful_stop_reactor(self):
+        d = self.stop()
+        d.addBoth(self._stop_reactor)
+        return d
+
     def _stop_reactor(self, _=None):
         try:
             reactor.stop()


Test patch:
Repository: scrapy/scrapy

Issue:
HTTP_PROXY variable with username and empty password not supported
Scrapy doesn't support proxy authentication when the password is empty when using the HTTP_PROXY environment variable to supply the proxy argument.



Discussion:
For example, scrapy does not support Crawlera's API key like this:
`export http_proxy=http://<API key>:@paygo.crawlera.com:8010`


Resolving patch:
diff --git a/scrapy/downloadermiddlewares/httpproxy.py b/scrapy/downloadermiddlewares/httpproxy.py
--- a/scrapy/downloadermiddlewares/httpproxy.py
+++ b/scrapy/downloadermiddlewares/httpproxy.py
@@ -25,7 +25,7 @@ def _get_proxy(self, url, orig_type):
         proxy_type, user, password, hostport = _parse_proxy(url)
         proxy_url = urlunparse((proxy_type or orig_type, hostport, '', '', '', ''))
 
-        if user and password:
+        if user:
             user_pass = '%s:%s' % (unquote(user), unquote(password))
             creds = base64.b64encode(user_pass).strip()
         else:


Test patch:
diff --git a/tests/test_downloadermiddleware_httpproxy.py b/tests/test_downloadermiddleware_httpproxy.py
--- a/tests/test_downloadermiddleware_httpproxy.py
+++ b/tests/test_downloadermiddleware_httpproxy.py
@@ -54,6 +54,14 @@ def test_proxy_auth(self):
         self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
         self.assertEquals(req.headers.get('Proxy-Authorization'), 'Basic dXNlcjpwYXNz')
 
+    def test_proxy_auth_empty_passwd(self):
+        os.environ['http_proxy'] = 'https://user:@proxy:3128'
+        mw = HttpProxyMiddleware()
+        req = Request('http://scrapytest.org')
+        assert mw.process_request(req, spider) is None
+        self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
+        self.assertEquals(req.headers.get('Proxy-Authorization'), 'Basic dXNlcjo=')
+
     def test_proxy_already_seted(self):
         os.environ['http_proxy'] = http_proxy = 'https://proxy.for.http:3128'
         mw = HttpProxyMiddleware()
Repository: scrapy/scrapy

Issue:
signals docs are confusing
It seems it is not explained how to connect a callback to a singnal anywhere in Scrapy docs.

http://doc.scrapy.org/en/latest/topics/signals.html tells:

> You can connect to signals (or send your own) through the [Signals API](http://doc.scrapy.org/en/latest/topics/api.html#topics-api-signals).

But if you follow this link you get docs for scrapy.signalmanager.SignalManager - that's fine, but it is not explained where to get a SignalManager instance from.

There is an example in Extension docs (http://doc.scrapy.org/en/latest/topics/extensions.html#sample-extension), but

a) this is just an example;
b) it is not explained that crawler.signals is a SignalManager instance;
c) this example is neither in Signals docs nor in SignalManager docs.

There is also a bit of information here: http://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.Crawler.signals, but

a) it is not linked to neither from Signal docs nor from SignalManager, so you can't find it if you don't know about it already;
b) it is not explained that crawler.signals is the only way to access signals.

So in the end users may get some luck connecting signals if they start from Crawler docs, but almost no luck if they start from Signals docs.



Discussion:
Hey !
This is an excellent opportunity to get to know the signals API better. Expect a PR soon :)


Resolving patch:
diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -1,4 +1,4 @@
-"""Helper functions which doesn't fit anywhere else"""
+"""Helper functions which don't fit anywhere else"""
 import re
 import hashlib
 from importlib import import_module


Test patch:
Repository: scrapy/scrapy

Issue:
KeyError in robotstxt middleware
I'm getting these errors in robots.txt middleware:

```
2016-01-27 16:18:21 [scrapy.core.scraper] ERROR: Error downloading <GET http://yellowpages.co.th>
Traceback (most recent call last):
  File "/Users/kmike/envs/scraping/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/Users/kmike/svn/scrapy/scrapy/downloadermiddlewares/robotstxt.py", line 65, in robot_parser
    if isinstance(self._parsers[netloc], Deferred):
KeyError: 'yellowpages.co.th'
```

It looks like https://github.com/scrapy/scrapy/pull/1473 caused it (I can't get this issue in Scrapy 1.0.4, but it present in Scrapy master). It happens when page failed to download and HTTP cache is enabled. I haven't debugged it further.



Discussion:
Marking it for 1.1 because it is regression since 1.0.x.


Resolving patch:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -101,4 +101,6 @@ def _parse_robots(self, response, netloc):
         rp_dfd.callback(rp)
 
     def _robots_error(self, failure, netloc):
-        self._parsers.pop(netloc).callback(None)
+        rp_dfd = self._parsers[netloc]
+        self._parsers[netloc] = None
+        rp_dfd.callback(None)


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -123,6 +123,18 @@ def return_failure(request, spider):
         deferred.addCallback(lambda _: self.assertTrue(middleware._logerror.called))
         return deferred
 
+    def test_robotstxt_immediate_error(self):
+        self.crawler.settings.set('ROBOTSTXT_OBEY', True)
+        err = error.DNSLookupError('Robotstxt address not found')
+        def immediate_failure(request, spider):
+            deferred = Deferred()
+            deferred.errback(failure.Failure(err))
+            return deferred
+        self.crawler.engine.download.side_effect = immediate_failure
+
+        middleware = RobotsTxtMiddleware(self.crawler)
+        return self.assertNotIgnored(Request('http://site.local'), middleware)
+
     def test_ignore_robotstxt_request(self):
         self.crawler.settings.set('ROBOTSTXT_OBEY', True)
         def ignore_request(request, spider):
Repository: scrapy/scrapy

Issue:
scrapy/xlib/pydispatch no longer in master
Hi there, I'm not sure if this is an issue or a design change. I just updated to the latest scrapy and my project has failed because it doesn't find the module scrapy.xlib.pydispatch. It was working fine some days ago and it looks like the code was removed from the release.



Discussion:
I investigated a bit further and it seems the problem is that the release Scrapy 1.1.0rc1 doesn't include that package while the previous release (Scrapy 1.0.5) does.
Is this something expected? it will break some working code referencing that package like in my case.

Hey, @lagenar, thanks for the report, it seems we missed this in the release notes.

The code was previously using a bundled pydispatch library, but now it uses [PyDispatcher](https://pypi.python.org/pypi/PyDispatcher) which is Python 3 compatible.

In case your project was using pydispatch imports to connect signals, you can add a `from_crawler` method to your spider to do that:

```
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
    spider = super(MySpider, cls).from_crawler(crawler, *args, **kwargs)
    crawler.signals.connect(spider.spider_opened, signals.spider_opened)
    return spider
```

@dangra @kmike do you think it's worth to add a fallback for these imports with a DeprecationWarning or should we just document this in the release notes (since it isn't part of the core API)?

@eliasdorneles I'm afraid yes, we should add a fallback with a big deprecation warning.

+1

Okay, cool, will send a PR later.


Resolving patch:
diff --git a/scrapy/xlib/pydispatch.py b/scrapy/xlib/pydispatch.py
new file mode 100644
--- /dev/null
+++ b/scrapy/xlib/pydispatch.py
@@ -0,0 +1,19 @@
+from __future__ import absolute_import
+
+import warnings
+from scrapy.exceptions import ScrapyDeprecationWarning
+
+from pydispatch import (
+    dispatcher,
+    errors,
+    robust,
+    robustapply,
+    saferef,
+)
+
+warnings.warn("Importing from scrapy.xlib.pydispatch is deprecated and will"
+              " no longer be supported in future Scrapy versions."
+              " If you just want to connect signals use the from_crawler class method,"
+              " otherwise import pydispatch directly if needed."
+              " See: https://github.com/scrapy/scrapy/issues/1762",
+              ScrapyDeprecationWarning, stacklevel=2)


Test patch:
diff --git a/tests/test_pydispatch_deprecated.py b/tests/test_pydispatch_deprecated.py
new file mode 100644
--- /dev/null
+++ b/tests/test_pydispatch_deprecated.py
@@ -0,0 +1,12 @@
+import unittest
+import warnings
+from six.moves import reload_module
+
+
+class DeprecatedPydispatchTest(unittest.TestCase):
+    def test_import_xlib_pydispatch_show_warning(self):
+        with warnings.catch_warnings(record=True) as w:
+            from scrapy.xlib import pydispatch
+            reload_module(pydispatch)
+        self.assertIn('Importing from scrapy.xlib.pydispatch is deprecated',
+                      str(w[0].message))
Repository: scrapy/scrapy

Issue:
Add a sample middleware to startproject's template
It will be nice to have a middleware template inside the template project to serve as an example for people that want to use it.



Discussion:

Resolving patch:
diff --git a/scrapy/commands/startproject.py b/scrapy/commands/startproject.py
--- a/scrapy/commands/startproject.py
+++ b/scrapy/commands/startproject.py
@@ -17,6 +17,7 @@
     ('${project_name}', 'settings.py.tmpl'),
     ('${project_name}', 'items.py.tmpl'),
     ('${project_name}', 'pipelines.py.tmpl'),
+    ('${project_name}', 'middlewares.py.tmpl'),
 )
 
 IGNORE = ignore_patterns('*.pyc', '.svn')


Test patch:
Repository: scrapy/scrapy

Issue:
scrapy view <url> raise exc in v1.3.0
````
(py35) wingyiu@mbp101:~$scrapy view http://www.scrapy.org
2017-01-19 22:13:54 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: scrapybot)
2017-01-19 22:13:54 [scrapy.utils.log] INFO: Overridden settings: {}
Traceback (most recent call last):
  File "/Users/user/venv/py35/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/Users/user/venv/py35/lib/python3.5/site-packages/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/user/venv/py35/lib/python3.5/site-packages/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/Users/user/venv/py35/lib/python3.5/site-packages/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/Users/user/venv/py35/lib/python3.5/site-packages/scrapy/commands/fetch.py", line 58, in run
    if not opts.no_redirect:
AttributeError: 'Values' object has no attribute 'no_redirect'
````



Discussion:
Ouch! Thanks for reporting @wingyiu .
Looks like a regression in 1.3.0 because of https://github.com/scrapy/scrapy/pull/2410

Resolving patch:
diff --git a/scrapy/commands/view.py b/scrapy/commands/view.py
--- a/scrapy/commands/view.py
+++ b/scrapy/commands/view.py
@@ -11,9 +11,8 @@ def long_desc(self):
             "contents in a browser"
 
     def add_options(self, parser):
-        ScrapyCommand.add_options(self, parser)
-        parser.add_option("--spider", dest="spider",
-            help="use this spider")
+        super(Command, self).add_options(parser)
+        parser.remove_option("--headers")
 
     def _print_response(self, response, opts):
         open_in_browser(response)


Test patch:
Repository: scrapy/scrapy

Issue:
Disk queues don't preserve Request class
When a Request subclass (e.g. FormRequest) is sent to a disk queue a bare Request is what you get back. 

This is inconvenient for scrapy-splash: Splash requests all have Splash URL as request.url, but for logging it is nice to display the requested URL, not only Splash URL. In scrapy-splash this is implemented by changing `__repr__` in a Request subclass, but it works only when request is kept in memory.



Discussion:

Resolving patch:
diff --git a/scrapy/utils/reqser.py b/scrapy/utils/reqser.py
--- a/scrapy/utils/reqser.py
+++ b/scrapy/utils/reqser.py
@@ -5,6 +5,7 @@
 
 from scrapy.http import Request
 from scrapy.utils.python import to_unicode, to_native_str
+from scrapy.utils.misc import load_object
 
 
 def request_to_dict(request, spider=None):
@@ -32,6 +33,8 @@ def request_to_dict(request, spider=None):
         'priority': request.priority,
         'dont_filter': request.dont_filter,
     }
+    if type(request) is not Request:
+        d['_class'] = request.__module__ + '.' + request.__class__.__name__
     return d
 
 
@@ -47,7 +50,8 @@ def request_from_dict(d, spider=None):
     eb = d['errback']
     if eb and spider:
         eb = _get_method(spider, eb)
-    return Request(
+    request_cls = load_object(d['_class']) if '_class' in d else Request
+    return request_cls(
         url=to_native_str(d['url']),
         callback=cb,
         errback=eb,


Test patch:
diff --git a/tests/test_utils_reqser.py b/tests/test_utils_reqser.py
--- a/tests/test_utils_reqser.py
+++ b/tests/test_utils_reqser.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 import unittest
 
-from scrapy.http import Request
+from scrapy.http import Request, FormRequest
 from scrapy.spiders import Spider
 from scrapy.utils.reqser import request_to_dict, request_from_dict
 
@@ -42,6 +42,7 @@ def _assert_serializes_ok(self, request, spider=None):
         self._assert_same_request(request, request2)
 
     def _assert_same_request(self, r1, r2):
+        self.assertEqual(r1.__class__, r2.__class__)
         self.assertEqual(r1.url, r2.url)
         self.assertEqual(r1.callback, r2.callback)
         self.assertEqual(r1.errback, r2.errback)
@@ -54,6 +55,12 @@ def _assert_same_request(self, r1, r2):
         self.assertEqual(r1.priority, r2.priority)
         self.assertEqual(r1.dont_filter, r2.dont_filter)
 
+    def test_request_class(self):
+        r = FormRequest("http://www.example.com")
+        self._assert_serializes_ok(r, spider=self.spider)
+        r = CustomRequest("http://www.example.com")
+        self._assert_serializes_ok(r, spider=self.spider)
+
     def test_callback_serialization(self):
         r = Request("http://www.example.com", callback=self.spider.parse_item,
                     errback=self.spider.handle_error)
@@ -77,3 +84,7 @@ def parse_item(self, response):
 
     def handle_error(self, failure):
         pass
+
+
+class CustomRequest(Request):
+    pass