Package: scrapy-dynamic-spiders
Scrapy Dynamic SpidersScrapy configures its Crawler objects using class variables associated with each Spider class. Most of these can be
meaningfully changed using the Spider's constructor, or even with other Spider class methods. However, some class
variables, most notably custom_settings, are accessedbeforeSpider instantiation. Crawl rules are similarly accessed
before spider instantiation, though a protected method does allow for rule re-compilation with somewhat unpredictable
results. In certain cases, this can lead to the creation of large numbers of subclasses made solely to handle slightly
different websites or use cases.Dynamic Spiders offers a simple, extensible API for generating spider subclasses based on existing ones, as well as
an API for running crawls in a synchronous environment (usingCrochet).Use CasesMutate your pipeline without putting a million boolean flags in your codeRapidly generate large numbers of spider subclasses based on external dataRun crawls from within a blocking event loopRun crawls tailored to user inputThe APIsWhat follows is a brief overview of the provided APIs.SpiderClsFactoryclassscrapy_dynamic_spiders.factories.SpiderClsFactory(custom_settings=None,settings_ow=False)This is the basic SpiderClsFactory, which can handle Spider, XmlFeedSpider, CsvFeedSpider, and SitemapSpider subclasses.
It allows for dynamic definition of custom_settings, which is the only thing in these generic spider classes which
have effects before instantiation.Attributescustom_settings- a custom_settings dictionary.settings_ow- determines how custom_settings are handled. IfTrue, the provided settings overwrite any other
custom settings native to the template class. IfFalse, the provided settings are merged with a copy of the native
settings, with preference for provided settings.Methodsconstruct_spiderself, (spidercls) -> type- construct_spider returns a subclass of the templatespiderclswith
itscustom settingsclass variable customized based on the Factory's current attributes.ExtensionYou should subclass SpiderFactory if you wish to customize class variables besidescustom_settings. This is done in
theconstruct_spidermethod, which you will thus need to overwrite. There is one relevant private method which you
may wish to use._construct_custom_settings(self, spidercls) -> dict- returns a newcustom_settingsdictionary based off of
existingcustom_settingsand the Factory's current attributes.CrawlSpiderClsFactoryInherits from SpiderClsFactory.classscrapy_dynamic_spiders.factories.CrawlSpiderClsFactory(custom_settings=None,settings_ow=False,extractor_configs=None,rule_configs=None,rule_ow:=False)This subclass handles rule construction for CrawlSpiders and CrawlSpider subclasses. It uses two lists of dictionaries,extractor_configsandrule_configsto create rules dynamically. Eachextractor_configdictionary provides
kwargs to a LinkExtractor, whose corresponding Rule is provided keyword arguments by the dictionary at the matching
index position inrule_configs. If there are fewerextractor_configsthanrule_configs, the lastextractor_configis used for all extrarule_configs.Attributesextractor_configs- a list of dictionaries. Each dictionary should contain kwargs for a LinkExtractor object.rule_configs- a list of dictionaries. Each dictionary should contain kwargs for a Rule object.rule_ow- ifTrue, any exiting rules are replaced by the dynamically generated rules. Iffalse, new rules
are appended to a copy of the existing rule list.ExtensionExtending this subclass should be similar to extending SpiderFactory. There is one private method which you may wish
to use._construct_rule_list(self, spidercls) -> List[Rule]- returns a new list of Rule objects based on those in the
template spidercls and the Factory's current attributes.SpiderWranglerclassscrapy_dynamic_spiders.wranglers.SpiderWrangler(settings,spidercls=None,clsfactory=None,gen_spiders=True)This class allows you to run scrapy crawls sequentially, within a synchronous context. It does this through a crochet-
managed Twisted Reactor. Because of this, it executes crawls using a composited CrawlerRunner object, rather than a
CrawlerProcess object. SpiderWranglers offer built-in support for generating spider subclasses using a SpiderClsFactory,
though this behavior is optional.Parameterssettings- a Scrapy Settings object. Required to initialize the composited CrawlerRunner.Attributesgen_spiders- IfTrue, crawls made using thestart_crawlmethod will use a spider subclass generated using
the composited SpiderClsFactory. IfFalse, crawls will use the Spider class referenced in the Wrangler'sspiderclsattribute. Attempting to crawl withgen_spiders=Truewithout aclsfactoryraises an error.spidercls- The Spider class which will be used for crawling or generating subclasses for crawling.clsfactory- A SpiderClsFactory or subclass instance.Methodsstart_crawl(self, *args, **kwargs) -> Deferred- Initiates a crawl which blocks in a synchronous context. Unlike
Scrapy's Core API, this allows you to perform multiple crawls in sequence without having to manually manipulate your
Reactor.*argsand**kwargsare passed to the Spider's constructor. The Spider class used is eitherspiderclsor a dynamically-generated subclass of it.ExtensionExtension of SpiderWrangler is generally straightforward. However, modifying the behavior ofstart_crawlrequires
extra attention.start_crawlis decorated using crochet'swait_fordecorator. To modify the timeout specified by
this decorator, you will need to override the existingstart_crawlmethod. To simplify this, the primary functionality
ofstart_crawlis encapusalated in the private method_start_crawl. Changing the behavior of the decorator thus
requires overridingstart_crawlwith a new instance of thewait_fordecorator, and calling_start_crawlin
the body of the function._start_crawl(self, *args, **kwargs) -> Deferred- Wraps the actual functionality ofstart_crawl. Return the value
return by this function, or a mutation of it, when overridingstart_crawl.TestsRun tests like so.python-munitteststestsKnown IssuesRunning tests results in unclosed sockets. Currently investigating.Changelogv1.0.0a1Initial alpha release