Repository: scrapy/scrapy

Issue:
Give `BaseDupeFilter` access to spider-object
I am in a situation where a single item gets defined over a sequence of multiple pages, passing values between the particular callbacks using the `meta`-dict. I believe this is a common approach among scrapy-users.

However, it feels like this approach is difficult to get right. With the default implementation of `RFPDupefilter`, my callback-chain is teared apart quite easy, as fingerprints don't take the meta-dict into account. The corresponding requests are thrown away, the information in the meta-dict which made this request unique is lost.

I have currently implemented by own meta-aware DupeFilter, but I am still facing the problem that it lacks access to the specific spider in use - and only the Spider really knows the meta-attributes that make a request unique. I could now take it a step further and implement my own scheduler, but I'm afraid that all these custom extensions make my code very brittle wrt future versions of scrapy.


Discussion:
I had a different problem that I needed to change the `RFPDupefilter` class because the HTTP cache didn't cache wasn't working the way I want it. I want it to cache every single `Request` so then I could do a replay of the same scrape using cache `Request` instances with their `Response` pair, and any new `Request` or `Response` instances that my scraper  might yield on the consecutive run to ignore completely.

Here is what you need to do:

1. Change the dupefilter class inside your settings.py;
   * DUPEFILTER_CLASS='yourown.dupefilters.RFPDupeFilter'

2. Subclass `scrapy.dupefilters.RFPDupefilter` class from `yourown.dupefilters.RFPDupefilter`.
3. Replace  the `request_fingerprint()` method with your own inside `yourown.dupefilters.RFPDupefilter`. 
4. Optional you can modify a version of [request_fingerprint](https://github.com/scrapy/scrapy/utils/request.py#L19) to your own function.

That is basically the gist, and I'm positive you can get a working version in 20-30 mins , perhaps a full 1 hour.
As already mentioned, I have a custom Dupefilter with its own fingerprint mechanism up and running. My problem is that without also touching the scheduler and extending it, my fingerprint will never have access to the spider.
@Chratho I think your problem could be fixed really easily by changing a few lines. Look here inside the [_next_request_from_scheduler()](https://github.com/scrapy/scrapy/blob/master/scrapy/core/engine.py#L149) function, there is nothing that prevents it from passing the spider as a second argument. In fact, this function is only being called ONCE inside the whole Scrapy codebase. Even the [_next_request_from_scheduler()](https://github.com/scrapy/scrapy/blob/master/scrapy/core/engine.py#L147)  method has a `Spider` instance being passed inside the `spider` argument.

The change required to make this happen would be to allow `next_request()` from the `Scheduler` class to have a `Spider` instance passed as an argument.

@redapple would you be okay if I submit a PR for this. 
Another way to solve the problem is to pass a list of meta keys which should be considered for uniqueness check; a custom dupefilter may look at this list and compute fingerprint accordingly. See also: https://github.com/scrapy/scrapy/issues/900

If we're to make spider available to dupefilter we should add from_crawler support to dupefilters in addition to from_settings, this is how it is solved for all other Scrapy components.
@kmike I still believe there is no possibly reason why spider cannot be available to the Scheduler. If you check my post above, it will be a very small change and it will satisfy the @Chratho request. 

Why does a Scheduler class needs to rely on a dupefilter ? That is the implementation of the default Scheduler class by Scrapy. 
However, looking at the code , the Scheduler` class just needs to implement `from_crawler()`, `enqueues_request()`, and `next_request()`. You have to think beyond FIFO or LIFO @kmike, some people might want to implement a different logic with a custom Scheduler class. They might not want a dupefilter class, but a different mechanism to schedule and dispatch `Request` instances. 

I also believe there should be more docs about how to extend the `Scheduler` class, a good example would be [Frontera](https://github.com/scrapinghub/frontera/blob/master/frontera/contrib/scrapy/schedulers/frontier.py#L74-L181) that actually did implement their own `Scheduler` class.

@IAlwaysBeCoding I'm not sure I follow, sorry! Spider is available to Scheduler in from_crawler method which is called when Scheduler is created; a Scheduler subclass can store spider as an attribute. But this doesn't solve @Chratho's problem, as he heeds spider in dupefilter, not necessarily in Scheduler.

Adding 'scheduler' argument to next_request is a backwards incompatible and could break existing custom schedulers. It also doesn't provide any new features, as this argument is unused by default, and a Scheduler subclass can already access spider by overriding from_crawler method (as a bonus point, access is not limited to next_request method if spider is saved in from_crawler method).
@kmike, I see I made an error. The spider is in fact available to the `Scheduler` class through the `from_crawler` class method. It is on the `crawler.spider` attribute. You are right sorry, about that.

Also, I guess that would give backwards incompatibility. Never mind, I guess the correct answer for @Chratho would be to grab the spider instance from `crawler.spider` when is passed to the `from_crawler` class method.

@kmike thanks for the clarification. You are 100% right!
Sorry for the delay. 

Yes, the solution suggested by @kmike would definitely solve my problem. As it also seems like a natural fit to the design of scrapy I would love to see its implementation in an upcoming release.

Resolving patch:
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -4,7 +4,7 @@
 from os.path import join, exists
 
 from scrapy.utils.reqser import request_to_dict, request_from_dict
-from scrapy.utils.misc import load_object
+from scrapy.utils.misc import load_object, create_instance
 from scrapy.utils.job import job_dir
 
 logger = logging.getLogger(__name__)
@@ -26,7 +26,7 @@ def __init__(self, dupefilter, jobdir=None, dqclass=None, mqclass=None,
     def from_crawler(cls, crawler):
         settings = crawler.settings
         dupefilter_cls = load_object(settings['DUPEFILTER_CLASS'])
-        dupefilter = dupefilter_cls.from_settings(settings)
+        dupefilter = create_instance(dupefilter_cls, settings, crawler)
         pqclass = load_object(settings['SCHEDULER_PRIORITY_QUEUE'])
         dqclass = load_object(settings['SCHEDULER_DISK_QUEUE'])
         mqclass = load_object(settings['SCHEDULER_MEMORY_QUEUE'])


Test patch:
diff --git a/tests/test_dupefilters.py b/tests/test_dupefilters.py
--- a/tests/test_dupefilters.py
+++ b/tests/test_dupefilters.py
@@ -5,11 +5,60 @@
 
 from scrapy.dupefilters import RFPDupeFilter
 from scrapy.http import Request
+from scrapy.core.scheduler import Scheduler
 from scrapy.utils.python import to_bytes
+from scrapy.utils.job import job_dir
+from scrapy.utils.test import get_crawler
+
+
+class FromCrawlerRFPDupeFilter(RFPDupeFilter):
+
+    @classmethod
+    def from_crawler(cls, crawler):
+        debug = crawler.settings.getbool('DUPEFILTER_DEBUG')
+        df = cls(job_dir(crawler.settings), debug)
+        df.method = 'from_crawler'
+        return df
+
+
+class FromSettingsRFPDupeFilter(RFPDupeFilter):
+
+    @classmethod
+    def from_settings(cls, settings):
+        debug = settings.getbool('DUPEFILTER_DEBUG')
+        df = cls(job_dir(settings), debug)
+        df.method = 'from_settings'
+        return df
+
+
+class DirectDupeFilter(object):
+    method = 'n/a'
 
 
 class RFPDupeFilterTest(unittest.TestCase):
 
+    def test_df_from_crawler_scheduler(self):
+        settings = {'DUPEFILTER_DEBUG': True,
+                    'DUPEFILTER_CLASS': __name__  + '.FromCrawlerRFPDupeFilter'}
+        crawler = get_crawler(settings_dict=settings)
+        scheduler = Scheduler.from_crawler(crawler)
+        self.assertTrue(scheduler.df.debug)
+        self.assertEqual(scheduler.df.method, 'from_crawler')
+
+    def test_df_from_settings_scheduler(self):
+        settings = {'DUPEFILTER_DEBUG': True,
+                    'DUPEFILTER_CLASS': __name__  + '.FromSettingsRFPDupeFilter'}
+        crawler = get_crawler(settings_dict=settings)
+        scheduler = Scheduler.from_crawler(crawler)
+        self.assertTrue(scheduler.df.debug)
+        self.assertEqual(scheduler.df.method, 'from_settings')
+
+    def test_df_direct_scheduler(self):
+        settings = {'DUPEFILTER_CLASS': __name__  + '.DirectDupeFilter'}
+        crawler = get_crawler(settings_dict=settings)
+        scheduler = Scheduler.from_crawler(crawler)
+        self.assertEqual(scheduler.df.method, 'n/a')
+
     def test_filter(self):
         dupefilter = RFPDupeFilter()
         dupefilter.open()