Package: scrapy-loader-upkeep
OverviewThis improves over the built-inItemLoaderofScrapyby adding features
that focuses on themaintainabilityof the spider over time.This allows developers to keep track of how often parsers are being used on a
crawl, allowing to safely remove obsolete css/xpath fallback rules.MotivationScrapy supports adding multiple css/xpath rules in itsItemLoaderby default
in order to provide a convenient way for developers to keep up with site changes.However, some sites change layouts more often than others, while some perform
A/B tests for weeks/months where developers need to accommodate those changes.These fallback css/xpath rules gets obsolete quickly and fills up the project
with potentially dead code, posing a threat to the spiders’ long term maintenance.Original idea proposal:https://github.com/scrapy/scrapy/issues/3795Usagefromscrapy_loader_upkeepimportItemLoaderclassSiteItemLoader(ItemLoader):passUsing it inside a spider callback would look like:defparse(self,response):loader=SiteItemLoader(response=response,stats=self.crawler.stats)Nothing would change in the usage of thisItemLoaderexcept for the part on
injecting stat dependency to it, which is necessary to keep track of the usage
of the parser rules.This only works for the followingItemLoadermethods:add_css()replace_css()add_xpath()replace_xpath()Basic Spider ExampleThis is taken from theexamples/directory.$scrapycrawlquotestoscrape_simple_has_missingThis should output in the stats:2019-06-1614:32:32[scrapy.statscollectors]INFO:DumpingScrapystats:{...'parser/QuotesItemLoader/author/css/1':10,'parser/QuotesItemLoader/quote/css/1/missing':10,'parser/QuotesItemLoader/quote/css/2':10...}In this example, we could see that the1st cssrule for thequotefield
has had instances of not being matched at all during the scrape.New FeatureAs with the example above, we’re limited only to the positional context of when
theadd_css(),add_xpath(), etc were called during the execution.There will be cases where developers will be maintaining a large spider with a
lot of different parsers to handle varying layouts in the site. It would make
sense to have a better context to what a parser does or is for.A new optionalnameparameter is supported to provide more context around a
given parser. This supports the two (2) main types of creating fallback parsers:multiple callsloader.add_css('NAME','h1::text',name='Name from h1')loader.add_css('NAME','meta[value="title"]::attr(content)',name="Name from meta tag")would result in something like:{...'parser/QuotesItemLoader/NAME/css/1/Name from h1':8,'parser/QuotesItemLoader/NAME/css/1/Name from h1/missing':2,'parser/QuotesItemLoader/NAME/css/2/Name from meta tag':7,'parser/QuotesItemLoader/NAME/css/2/Name from meta tag/missing':3,...}grouped parsers in a single callloader.add_css('NAME',['h1::text','meta[value="title"]::attr(content)',],name='NAMEs at the main content')loader.add_css('NAME',['footer .name::text','div.page-end span.name::text',],name='NAMEs at the bottom of the page')would result in something like:{...'parser/QuotesItemLoader/NAME/css/1/NAMEs at the main content':8,'parser/QuotesItemLoader/NAME/css/1/NAMEs at the main content/missing':2,'parser/QuotesItemLoader/NAME/css/2/NAMEs at the main content':7,'parser/QuotesItemLoader/NAME/css/2/NAMEs at the main content/missing':3,'parser/QuotesItemLoader/NAME/css/3/NAMEs at the bottom of the page':8,'parser/QuotesItemLoader/NAME/css/3/NAMEs at the bottom of the page/missing':2,'parser/QuotesItemLoader/NAME/css/4/NAMEs at the bottom of the page':7,'parser/QuotesItemLoader/NAME/css/4/NAMEs at the bottom of the page/missing':3,...}The latter is useful in grouping fallback parsers together if they are quite
related in terms of layout/arrangement in the page.RequirementsPython 3.6+