Package: scrapy-test
scrapy-testScrapy test is a validation/test framework for validating scrapy results.
This framework is capable of testing scrapy crawl and stats output.Seeexampleproject for hackernews crawler with full test suite.Philosophy and Architecturescrapy-testtries to replicatescrapy.Itemdefinition but instead of defining fields it defines test for every field.Tests are callables that either returns a failure message if some condition is met.
Example item specification:class MyItem(Item):
    name = Field()
    url = Field()

class TestMyItem(ItemSpec):
    item_cls = MyItem

    # define tests
    name_test = Match('some-regex-pattern')
    url_test = lamda v: 'bad url' if 'cat' in v else ''

    # define coverage
    url_cov = 100  # 100% - every item should have url fieldscrapy-testalso supports stats output validation. When scrapy finished crawling it outputs various stats like error count etc.StatSpeccan be defined to validate these stats:class MyStats(StatsSpec):
    spider_cls = MySpder1, MySpider2
    # or multiple spiders
    validation = {  #stat_name_pattern : tests
        'item_scraped_count': MoreThan(1),
        'downloader/response_status_count/50\d': LessThan(1),
    }
    # required stat keys
    required = ['stat_pattern.+']Finallyscrapy-testdetermines failure by asserting if there are any messages generated by either stat ir item specifications (exit code 1 and 0 respectively).UsageSetuptest.pymodule should be created in spider directory.For example creatingtest.pyscrapy-test-example/
├── example
│   ├── __init__.py
│   └── test.py
└── scrapy.cfgAdd test file config toscrapy.cfg:[settings]default=example.settings[test]root=example.testDefineItemSpecfor item field validation:fromscrapytest.testsimportMatch,Equal,Type,MoreThan,Map,Len,RequiredclassTestPost(ItemSpec):# defining item that is being covereditem_cls=PostItem# defining field teststitle_test=Match('.{5,}')points_test=Type(int),MoreThan(0)author_test=Type(str),Match('.{3}')comments_test=Type(list),Required()# also supports methods!defurl_test(self,value:str):ifnotvalue.startswith('http'):returnf'Invalid url:{value}'return''ItemSpecclass should contain attributes that end in_testthese attributes have be callables (functions, methods etc.) that return message(s) if failure is encountered. See theurl_testexample above.DefineStatSpecfor crawl stats validation:classTestStats(StatsSpec):# stat pattern: test functionsvalidate={# this is default'log_count/ERROR$':LessThan(1),'item_scraped_count':MoreThan(1),'finish_reason':Match('finished'),}# these stats shoudl be requiredrequired=['some_cool_stat']StatsSpecshould containvalidateattribute withpattern: testsdictionary.DefineSpiderclasses:fromproject.spidersimportHackernewsSpiderclassTestHackernewsSpider(HackernewsSpider):test_urls=["https://news.ycombinator.com/item?id=19187417",]defstart_requests(self):forurlinself.test_urls:yieldRequest(url,self.parse_submission)This spider should extend your production spider that simply crawls the urls without doing discovery. Alternatively you can also not extend anything for live testing.Running$scrapy-test--helpUsage:scrapy-test[OPTIONS][SPIDER_NAME]runscrapy-testtestsandoutputmessagesandappropriateexitcode(1forfailed,0forpassed)Options:--cacheenableHTTPCACHE_ENABLEDsettingforthisrun--helpShowthismessageandexit.To run the tests use cli command:$ scrapy-test <spider_name>Spider name can be skipped for running all spidersNotificationsscrapy-testsupports notification hooks on either test failure or success:--notify-on-error TEXT    send notification on failure, choice from:
                            ['slack']
  --notify-on-all TEXT      send notification on failure or success, choice
                            from: ['slack']
  --notify-on-success TEXT  send notification on success, choice from:
                            ['slack']Rightscrapy-testoffers these notifiers:* Slack - to configure slack notification follow slack [incoming webhooks](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) app and supply these settings in `scrapy.cfg`:

    slack_url = https://hooks.slack.com/services/AAA/BBB/CCC
    # where the message goes to
    slack_channel = #cats
    # bot's name
    slack_username = bender
    # bot's avatar
    slack_icon_emoji = :bender:
    # maintainer will be mentioned on error
    slack_maintainer = @bernard