Package: scrapy-testmaster
Scrapy TestMasterOverviewScrapy TestMaster is an automatic test-generation, test-execution and general debugging tool for Scrapy spiders.As far as I am aware, Scrapy TestMaster is the most comprehensive tool yet for the automated debugging and testing of Scrapy spiders. It represents a strict extension of the capabilities of its most influential predecessor, Scrapy Autounit (https://github.com/scrapinghub/scrapy-autounit), with further features that make it more useful for both on-the-fly debugging and more robust monitoring of the kind that might be required in an enterprise context. Whereas other Scrapy testing libraries are useful only for crawler-validation subsequent to changes in the crawler code, this library offers further tools to test for changes in the targeted webpages themselves. More precisely, its capabilities include the following:Testing your Scrapy functions against specific requests on the fly, using an extended version of the Scrapy parse command (https://docs.scrapy.org/en/latest/topics/commands.html#std-command-parse) that can take any number of urls (testmaster parse).Using the above command to automatically generate testcases and unit tests.Generating automatic testcases and unit tests as you run your spider viascrapy crawl.Merging the debugging and test-generation processes by designing meta rules to determine whether a fixture (testcase) should be written based on the quality of the parsed results.Specifying requests in a config file to be downloaded and evaluated upon the next execution (allows FormRequests, requests with cookies, requests using proxies, etc).Designing custom testing/validation rules down to a callback-specific level so as to check the correctness of output in a very fine-grained manner.Updating and validation of existing tests via a re-parsing of the original HTML source with the current spider code.Updating and validation of existing tests via an additional re-downloading of the HTML source using the same request.Apart from the wider range of capabilities, the feature I most want to emphasise about this library is that it gives you the ability tosynthesise the processes of debugging and testcase-generation. First you set up the custom logic for validating the output of your spider/s, then you callscrapy crawlortestmaster parse. If the results are acceptable, testcases are written; otherwise, you get an informative error message.For more info on how to use this library, seeWhat is the Use Case for this Library.AcknowledgementsThe structure of my project was strongly influenced by Scrapy Autounit. Thescrapy parsecommand is, of course, another major influence, the code for which bears a very strong resemblance totestmaster parse. Read the License for more.Installationpip install scrapy-testmasterWhat it looks likeThis is what your project will look like after callingscrapy crawlortestmaster parsewith TestMasterMiddleware enabled.my_project

├── my_project
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders
│       ├── __init__.py
│       └── my_spider.py
├── testmaster
│   ├── __init__.py
│   └── tests
│       ├── __init__.py
│       └── my_spider
│           ├── __init__.py
│           └── my_callback
│               ├── __init__.py
│               ├── config.py
│               ├── fixture1.bin
│               ├── fixture2.bin
│               ├── test_fixtures.py
│               ├── view.json 
└── scrapy.cfgAnalogous to thesettings.pymodule in every Scrapy project,config.pyis for specifying the custom logic and rules that your test-results need to pass for the given callback directory in which the file resides. It also allows one to write down requests to be tested the next time you executetestmaster parse.view.jsonis simply a convenience for representing what's currently in your fixtures: it is an ordered list of the requests that generated the current fixtures for the callback in question, plus some basic summary stats. It is updated if the fixtures are changed.I'm considering changing this structure so that thetestmasterfolder is eliminated and thetestsfolder is placed within themy_projectsubfolder (the folder containingsettings.pyandspiderswhich can have a different name from the outer folder).UsageBasicTo begin validating output or generating tests withtestmaster parseorscrapy crawl, setTESTMASTER_ENABLED = Trueinsettings.py, then add the spider middleware to yourSPIDER_MIDDLEWARESsetting (no specific order required):SPIDER_MIDDLEWARES={'scrapy_testmaster.TestMasterMiddleware':950}Generating testsMake sure you enable Scrapy Testmaster in your settings:TESTMASTER_ENABLED=TrueTo generate your fixtures, you have three basic options:Scrapy Testmaster will generate them for you while running your spider as usual.$ scrapy crawl my_spiderAutomatically generate testcases for specified urls against a specified callback usingtestmaster parse, an extended version of the Scrapyparsecommand chiefly distinguished by its permission of multi-url inputs.$ testmaster parse "url1|url2|url3" --spider=my_spider -c my_callback --meta='{"x":"y"}' ...Specify requests to be later executed in python-dict format within theconfig.pyfile for the spider and callback in question (e.g."url":"", "headers":{..}, "_class":"scrapy.http.request.form.FormRequest"}). These will get executed whenever you calltestmaster updatewith the appropriate spider, callback args and options (seetestmaster update), and lead to new fixtures being written subject to the usual conditions.$ testmaster update my_spider -c my_callback --newThe first two of these commands will automatically generate a directorytestmasterin your project root dir if none exists, containing all the generated tests/fixtures for the spider you just ran, plus two more files (config.pyandview.json). (See the directory tree example above.)Other specific tasksSetting up testing environment for given spider/callback before making any requests$ testmaster establish my_spider -c my_callback(If no callback is specified, every single callback in the spider will get its own directory with aconfig.pyfile.)Generating new tests for specific urls on the fly$ testmaster parse "url1|url2|url3|..." --spider=my_spider -c my_callback --meta='{"x":"y"}' ...Testing output against custom rules on the flyFirst, set up your rules in the appropriateconfig.pyfile, then run:$ testmaster parse "url1|url2|url3|..." --spider=my_spider -c my_callback --meta='{"x":"y"}' ...If the results generated by such a command are all fine according to your custom rules, the fixtures/tests will be written (assuming you haven't reached the fixtures cap you set). Otherwise, you will get an error message about the failure mode. This is an example of how you can merge the debugging and test-generation processes.Updating the results of tests in response to changes in spider code (i.e. re-parsing the downloaded source within the fixture)$ testmaster update my_spider -c my_callback --fixture n(If no fixture is specified, every fixture is changed.)Remaking the entire fixture using the original request to check for a website change (i.e. re-downloading the response)$ testmaster update my_spider -c my_callback --dynamicTriggering the download of complex requests (e.g. FormRequests) described in a config.py fileDefine a request in Python-dict within a config.py file for the relevant callback, and then call:$ testmaster update my_spider -c my_callback --newIf the results of the request triggered by this command pass your custom rules, and there is space in the fixtures, a new fixture will be written for this request and response.Clearing some fixtures that are surplus to requirements (and re-ordering the rest within the directory)$ testmaster clear my_spider my_callback 1,4,8,10Running testsTo run your tests statically, you can useunittestregular commands.Test all$ python -m unittestTest a specific spider$ python -m unittest discover -s testmaster.tests.my_spiderTest a specific callback$ python -m unittest discover -s testmaster.tests.my_spider.my_callbackTest a specific fixture$ python -m unittest testmaster.tests.my_spider.my_callback.test_fixture2It's worth stating that all of the commands in this library apart fromestablish,inspectandclearhave a debugging/testing purpose. Theseunittestcommands are just useful to test your code against existing fixtures without changing them in any way.Important CaveatsAs long asTESTMASTER_ENABLEDis on, each time you run a spider usingscrapy crawl, existing tests/fixtures will be over-written, if the results of the requests being made pass your custom rules. However, if you run a specific callback usingtestmaster parse, this over-writing will not apply - fixtures will be added (within the limit you have set byTESTMASTER_MAX_FIXTURES_PER_CALLBACK).There are a few lines of code in this library that rely on the assumption that you haven't named your spider file differently from the name attribute of the spider itself. So keep these names aligned if you want assurance that everything will always work! (If you always usescrapy genspiderand don't later edit the file name or spider name, there will, of course, be no problem.)Running thescrapy parsecommand (as opposed to thetestmaster parsecommand) with the TestMasterMiddleware enabled will not work properly - the middleware will try and fail to interact with the responses.This package works best with base Scrapy spiders, rather than e.g. CrawlSpiders or SiteMapSpiders, at least when runningscrapy crawl(as opposed to usingtestmaster parseand sending the results to one of the explicitly written callbacks in your spider code). For example, in the case of CrawlSpiders, it will write folders for callbacks called_callbackand_parse_response(the underlying callbacks of the CrawlSpider code). I might try to alter this if there is demand.Just in case this happens to be relevant, using as keys in your request.meta any of the strings '_parse', '_update' or '_fixture' may lead to unexpected behaviour with the middleware enabled.What's the deal with the fixtures?The fixtures are essentially just test cases for your spider. Thefixture%d.binfiles store in binary format a big JSON dict containing the spider name that generated the fixture, the full details of the request, the entire downloaded response body, the result (i.e. a list of items and/or requests corresponding to the generator produced by the callback code), details about the middlewares in play, the spider settings, the spider args, and the Python version. This library offers you two ways of using these fixtures to run your tests, once they've been written:Run a static test which parses the response using your updated code for the same callback (to check that you have not broken anything by comparing against the results in the fixture).Run a dynamic test which first downloads a new response using the request info encoded in the fixture, and then parses the response using your code for the callback (to check that the website has not changed).Project/Spider SettingsN.B.You will notice that there is heavy overlap here with the settings in config.py. So you can set custom validation rules at any level, although any settings specified to a contrary, non-default value in aconfig.pyfile will take precedence for that particular callback.TESTMASTER_ENABLEDSet this toTrueorFalseto enable or disable unit test generation when callingscrapy crawl,testmaster parseor when runningtestmaster updatewith the--dynamicor--newoptions. The other commands are completely unaffected (e.g. you can run static updates with or without this enabled).TESTMASTER_MAX_FIXTURES_PER_CALLBACKSets the maximum number of fixtures to store per callback.Minimum: 10Default: 10TESTMASTER_IGNORE_SPIDER_ARGSIfTrue, testing your fixtures will ignore explicitly checking for any new attributes you have added to your spider__init__function. By default, if you add a new attribute after you write a test, re-running that test will cause it to fail.TESTMASTER_SKIPPED_FIELDSSets a list of fields to be skipped from testing your callbacks' items. It's useful to bypass fields that return a different value on each run.For example if you have a field that is always set todatetime.now()in your spider, you probably want to add that field to this list to be skipped on tests. Otherwise you'll get a different value when you're generating your fixtures than when you're running your tests, making your tests fail.Default: []TESTMASTER_REQUEST_SKIPPED_FIELDSSets a list of request fields to be skipped when running your tests.Similar to TESTMASTER_SKIPPED_FIELDS but applied to requests instead of items.Default: []TESTMASTER_EXCLUDED_HEADERSSets a list of headers to exclude from requests recording.For security reasons, Testmaster already excludesAuthorizationandProxy-Authorizationheaders by default, if you want to include them in your fixtures andview.jsonseeTESTMASTER_INCLUDED_AUTH_HEADERS.Default: []TESTMASTER_INCLUDED_AUTH_HEADERSIf you want to includeAuthorizationorProxy-Authorizationheaders in your fixtures andview.json, add one or both of them to this list.
This also applies to a Scrapy-Splash API key which would normally reside insplash_headersunder theAuthorizationkey. Put 'Authorization' in this list to include such a key.Default: []TESTMASTER_INCLUDED_SETTINGSSets a list of settings names to be recorded in the generated test case.Default: []TESTMASTER_EXTRA_PATHThis is an extra string element to add to the test path and name between the spider name and callback name. You can use this to separate tests from the same spider with different configurations. This is respected by all methods of creating directories for spiders + callbacks, i.e.testmaster establish,testmaster parseandscrapy crawl.   It is also respected bytestmaster updatewhen it's working out what fixtures you want to update.Default: NoneThe following 3 settings are the main way to implement custom testing behaviour. They have two utilities:ex-anteandex-post.ex-ante: if the rules established by these settings are violated while Testmaster is evaluating the results of a request that has not yet become a fixture, the fixture will not be written.ex-post: if any of them are violated when you're updating an existing fixture (i.e. parsing the pre-downloaded response with new code), you know that you have made a bad change to your code.TESTMASTER_OBLIGATE_ITEM_FIELDSInsert here any field names which you intend to exist in every "item" (as opposed to "request") object outputted by the tests within this project.
You can set this to a non-default value but override for specific spiders + callbacks by tweaking the corresponding field in the relevant local config.py file/s.Default: []TESTMASTER_PRIMARY_ITEM_FIELDSInsert here any field names which you intend to be non-empty in every "item" object outputted by the tests within this project.
It is not necessary to duplicate a field name across this and the above setting; just put the field in this list.
You can set this to a non-default value but override for specific spiders + callbacks by tweaking the corresponding field in the relevant local config.py file/s.Default: []TESTMASTER_PATH_TO_RULES_FILEInsert here a relative path (relative from the project root, minus ".py" extension) to a Python file containing at least one of the following two classes:class RequestRules(object)andclass ItemRules(object). Within these classes, you can devise any number of functions, given whatever names you like, that take one changeable argument each: a "request" in the former case and an "item" in the latter. These functions are intended to contain one or more "assertion" statements.All items will be tested against your item rules, and equivalently for requests.Default: NoneNote: Remember that you can always apply any of these settings per spider including them in your spider'scustom_settingsclass attribute - seehttps://docs.scrapy.org/en/latest/topics/settings.html#settings-per-spider.Callback-specific settings (in config.py)Be apprised that any list-type options are not combined across the global value and the equivalent local value in the config.py file. That is to say, if you want to add one more element to one of these options like ...EXCLUDED_HEADERS or ...INCLUDED_SETTINGS at the callback level, you can't just fill the list in the config.py file with that single additional element; you must include all of the elements. (This already follows from the rule I stated earlier and which I'll restate again: if any edits away from the default have been made at the local level for a given setting, those edits dominate.) As for custom rules, they work as follows: the global custom rules are tried after the local rules are tried. So you can't override the global custom rules with the local rules.MAX_FIXTURESEquivalent to global setting.SKIPPED_FIELDSEquivalent to global setting.REQUEST_SKIPPED_FIELDSEquivalent to global setting.EXCLUDED_HEADERSEquivalent to global setting.INCLUDED_AUTH_HEADERSEquivalent to global setting.INCLUDED_SETTINGSEquivalent to global setting.OBLIGATE_ITEM_FIELDSEquivalent to global setting.PRIMARY_ITEM_FIELDSEquivalent to global setting.REQUESTS_TO_ADDThis is for storing complex requests in Python dict format that will be executed the next time you run any command oftestmaster updatewith args indicating the relevant callback, and using either of the options--dynamicor--new(seetestmaster update). It allows you to use all the standard request args as keys plus the "_class" key for specifying a FormRequest or a SplashRequest. The motivation behind it is the difficulty/impossibility involved in trying to do the same thing for special/complex requests usingtestmaster parseon the command-line. Obviously, you can also specify simple requests here if you like.Example of request format with special requests:[
    {
        "url": "https://examplewebsite011235811.com/info",  
        "headers": {"referer":"...", "content_type": "..."},
        "method": "POST",
        "_class": "scrapy.http.request.form.FormRequest",
        "body": "user=blah&password=blah",
        "meta": {"x": "y"}
    },
    {
    	"url": "https://...-splash.scrapinghub.com/render.html", 
	"method": "POST", 
	"headers": {...}, 
	"body": "{...}", 
	"meta": {"splash": {"endpoint": "render.html", "splash_headers": {'Authorization': INSERT_API_KEY}, "args": {"url": "..."}}},
	"dont_filter": false,
	"_class": "scrapy_splash.request.SplashRequest"
    }
]Default: []All of these keys are optional except "url" and "meta". The callback is inferred from the location of the config.py file. You can get a sense of what the format for a given request needs to look like by runningscrapy crawlfor that spider and looking atview.jsonfor the relevant callback. The only caveat is that JSON "nulls" need to be converted to Python "None", and same for JSON "true"/"false" (to "True"/"False").When this request is triggered, it is treated as if you had runtestmaster parsewith the same info. So its response and results can then become part of a new fixture if all your validation rules are passed and there is space.Currently, you're going to have to delete these requests manually from theconfig.pyfile after they've been added to make sure they're not triggered again next time you executetestmaster update.OtherAll config.py files will include by default two classes:class ItemRules(object)andclass RequestRules(object). You can define any number of methods for these classes that take args {self,item} and {self,request} respectively. The logic for such methods was explained previously in the precis forTESTMASTER_PATH_TO_RULES_FILE.Format of view.json{"1": {"request": {"url": "...", "callback": "...", "errback": null, "method": "GET", "headers": {...}, "body": "", 
"cookies": {}, "meta": {...}, "_encoding": "utf-8", "priority": 0, "dont_filter": false, "flags": [], "cb_kwargs": 
{}}, "num_items": 0, "num_requests": 1}, "2": {"request": {...}}}Command line interfacetestmaster parse: makes a number of command-line specified requests and automatically generates testcases (if conditions meet)testmaster establish: generates a directory with aconfig.pyfile for every callback specifiedtestmaster update: updates fixtures to test code changes or with a view to guarding against website changestestmaster inspect: inspects fixtures returning a JSON objecttestmaster clear: clears the specified fixtures and re-arranges the rest to restore linearityN.B.testmaster parse,testmaster establishandscrapy crawl my_spider, when called with the middlewares enabled,  automatically generate the basic testmaster project skeleton if it doesn't exist already (i.e.testmaster/tests). The former two do not overwrite anything, butscrapy crawlwill. (Unlike in Scrapy Autounit,scrapy crawldoesn't nuke the entire test directory for the spider in question but instead just immediately overwrites existing fixtures. And likeparseandupdate, it will pay attention to your rules and settings in any existingconfig.pyfiles)testmaster parseThis is just likescrapy parse(https://docs.scrapy.org/en/latest/topics/commands.html#std-command-parse) but with greater powers: a greater diversity of requests can be specified and multiple urls can be inputted. Furthermore, if you enableTestMasterMiddleware, then the requests triggered by this command will be used to create new testcases, assuming the results pass any custom rules you set down and you haven't reached the max fixtures limit.testmaster parsetakes exactly the same arguments/options asscrapy parse, plus three additional options: headers, cookies, and--homepage(to visit the website homepage to pick up one or more session token/s before the primary requests are kicked off). The url argument remains the same except that many urls can be inserted at once, separated by|, i.e."https://www.exampledomain.com/page1|https://www.exampledomain.com/page2|https://exampledomain.com/page2". (This used to be comma-separation, and was changed as of version 1.0.) The depth argument is (of course) re-calibrated for the input plurality.Likescrapy crawl, this command will not write any tests automatically unlessTestMasterMiddlewareis enabled. Unlikescrapy crawl, this command will never generate tests/fixtures that overwrite existing ones; it is additive only (though also beholden to *TESTMASTER_MAX_FIXTURES_PER_CALLBACK).One miscellaneous thing to note is that if you have a set value forTESTMASTER_EXTRA_PATHin your settings, this command will observe this value, i.e. it will create/update the directory 'testmaster/tests/[extra_path]/spider_name/callback_name'.Example:$ testmaster parse "url1|url2|url3" --spider=my_spider -c my_callback --meta='{"x":"y"}' --homepage -d 2...Because of thehomepagearg, this will first try to grab a homepage value from thestart_urlsfield formy_spider(it will move on if it fails). It will pick up cookies from the homepage if possible, then use these to make the requests to the specified urls. Because of the--depth=2arg, it will make one further request for each of the three requests that lead to another request inmy_callback. For each of the requests generated, a fixture will be written subject to the usual conditions and/or custom rules, and (as with the Scrapy parse command) the scraped items will be printed to the terminal for perusal.Why doesn'tscrapy parsetake multiple urls?I suspect its developers figured that if the requests need to be made in a similar manner across different webpages, those webpages cannot nontrivially differ. But I've found in my own scraping efforts that this is an incorrect assumption in a large number of cases. For example, it is common that one's callback code accounts differently for pages with no data versus non-empty pages, or pages with a data field X differently from pages without, and the requests to these two classes of page can be made in the same way (in the simplest case, with default headers and no cookies). Certainly, there is still an infinity of websites out there where most/all of the pages within the domain can be accessed without page-specific headers or parameters. So it's very useful to have a command which allows you to test several differing cases with one call.testmaster establishThis command is simply for setting up the directory structure for a given spider or for a specific spider + callback without having to make any requests. It is not strictly necessary to use this command becausescrapy crawlandtestmaster parsewill set up the directory structure on their own when they generate tests. Its raison d'etre is the case where it is desirable to define custom behaviour for debugging (for processing the testcases before they become inscribed fixtures) before making any requests.
Below is an example of this command and the result.$ testmaster establish my_spider -c my_callback├── testmaster
│   ├── __init__.py
│   └── tests
│       ├── __init__.py
│       └── my_spider
│           ├── __init__.py
│           └── my_callback
│               ├── __init__.py
│               ├── config.pyWithout the callback argument, it creates multiple subdirectories for each callback in the spider (identified using nothing more intelligent than regex, i.e. matches functions with args including "response" keyword). Example:├── testmaster
│   ├── __init__.py
│   └── tests
│       ├── __init__.py
│       └── my_spider
│           ├── __init__.py
│           ├── my_callback
│           │   ├── __init__.py
│           │   └── config.py
│           └── my_callback1
│               ├── __init__.py
│               └── config.pyThis command will observe theTESTMASTER_EXTRA_PATHin your settings.testmaster inspectTo inspect a fixture's data, you need to pass the spider, callback and fixture name to the command:$ testmaster inspect my_spider my_callback fixture3The fixture can also be passed as a number indicating which fixture number to inspect like this:$ testmaster inspect my_spider my_callback 3Extracted Data (unchanged from Scrapy Autounit)This command returns a JSON object that can be parsed with tools likejqto inspect specific blocks of data.The top-level keys of the JSON output are:spider_nameThe name of the spider.requestThe original request that triggered the callback.responseThe response obtained from the original request and passed to the callback.resultThe callback's output such as items and requests.middlewaresThe relevant middlewares to replicate when running the tests.settingsThe settings explicitly recorded by theTESTMASTER_INCLUDED_SETTINGSsetting.spider_argsThe arguments passed to the spider in the crawl.python_versionIndicates if the fixture was recorded in python 2 or 3.Then for example, to inspect a fixture's specific request we can do the following:$ testmaster inspect my_spider my_callback 4 | jq '.request'testmaster updateYou can update your fixtures to match your latest changes in a particular callback to avoid running the whole spider.This updates the results for all the fixtures for a specific callback:$ testmaster update my_spider -c my_callbackThis updates the results for all the fixtures for a spider:$ testmaster update my_spiderThis completely replaces all the fixtures for a specific callback, by re-downloading the response using the original request. (It also downloads, and potentially creates a fixture out of, any requests specified in the "REQUESTS_TO_ADD" field.)$ testmaster update my_spider -c my_callback --dynamicThis completely replaces all the fixtures for a spider:$ testmaster update my_spider --dynamicConversely, you can specify a particular fixture to update with-for--fixture:$ testmaster update my_spider -c my_callback --fixture 4(And of course, this can be paired with--dynamicto replace the fixture.)This updates all the results for all the fixtures for a spider, and downloads the requests in theREQUESTS_TO_ADDfield inconfig.pyfor the specified callback.$ testmaster update my_spider -c my_callback --new(As before, this command can be executed for all callbacks, rather than just one, if desired.)As shown by the above, any call oftestmaster updatewith--dynamicor--newoptions towards a particular callback will automatically trigger the execution of any requests set down in the REQUESTS_TO_ADD list within the relevant config.py file. If these requests are successful according to your rules, and there is space in the fixtures buffer for that particular callback, these requests will result in corresponding fixtures being added. (This is the only way to trigger the execution of these requests.)testmaster updatewill refuse to write its updates if the results fail any of your custom rules or configuration options. So you don't have to worry about your fixtures being overwritten with junk. This means you can use this command to check the correctness of changes to your code in a more fine-grained way than the Scrapy Autounit library enables.CaveatsIf you have used the 'extra path' setting to set up two or more classes of test for a single spider (perhaps because that spider has multiple distinct configurations), thentestmaster updatewill only update any fixtures that can be found using the value for thisextra pathsetting in settings.py at the moment you execute the command. So to update all the fixtures for that spider, across all its configurations, you have to repeatedly edit the extra path value in settings.py and calltestmaster update my_spiderfor every distinct configuration/extra path. If this is a common situation for people to find themselves on and they find this inconvenient, let me know and I will add the feature that you can specify an extra path on the command-line.testmaster clearThis allows you to clear out old fixtures that you no longer need. This may be the case, for example, if the request to which the fixture corresponds no longer works for the website in question (so that there is no hope of fixing it viatestmaster update ... --dynamic).All arguments are obligatory. Example usage:$ testmaster clear my_spider my_callback 5,10,11This will cause fixtures 5, 10, 11 to disappear from the callback specified. Then the others will be renamed, so that fixture 6 becomes fixture 5, 7 becomes 6 and so on. Finally, theview.jsonfile is updated to match this re-numbering.What is the Use Case for this Library?The idea behind this project is to provide a set of robust, effective testing and debugging tools for large Scrapy codebases. Here is how I see this library being used in this high-maintenance/enterprise context:As the developer is programming the spider, she debugs her code, and refines her selectors, by callingtestmaster parseon various urls. She thereby automatically generates tests/fixtures for every callback each time these requests lead to results that pass any custom rules she has set down (so the debugging and test-generation process is neatly entwined). She can then supplement these fixtures, if she requires, by further calls oftestmaster parse. Alternatively, if she is satisfied with her code, she can just runscrapy crawlto generate a set of fresh fixtures. With these fixtures in place, she can feel secure with every change she makes to her code by running the appropriate tests usingpython -m unittest .... Once this initial development phase is all done --- she has a number of solid test-cases in place for each callback, and her spider code is working --- she can now boost the likelihood of continuous, effective deployment of her spider by runningupdate --dynamic ...on appropriate callbacks at regular intervals to re-download the relevant test pages. By regularly re-downloading the same requests and testing the output, she will remain alert to any changes to the target website which affect the viability of the code.Of course, the casual Scrapy user will not be concerned with running regular dynamic test updates, but there is no penalty for using only a subset of the capabilities!The Example ProjectThe example project contains two Scrapy spiders, one called "olympus" which crawls this basic website:http://olympus.realpython.org/profiles, and another called "shopify" which can grab products from the public API endpoint of Shopify websites (and by default gets products from the top 5 Shopify websites according to this website (https://www.shopistores.com/top-500-most-successful-shopify-stores/) as of 8:15 AM UTC, October 13 2020).The fixtures that exist in this project currently were generated according to the following algorithm:Alterexample/settings.pyas displayed.Create ageneric_rules.pyfile at the head of the project with rules as displayed.Calltestmaster establish olympusto generate a fixture folder andconfig.pyfile for each callback.Calltestmaster establish shopifyto do the same for that spider.Altertestmaster/tests/olympus/parse_login/config.pyas displayed.Calltestmaster update olympus --newso as to create a fixture for the FormRequest described in REQUESTS_TO_ADD in that config file.Altertestmaster/tests/olympus/parse_info/config.pyso as to enforce a primary field constraint (as displayed).Runscrapy crawl olympusso as to automatically generate fixtures forparseandparse_info, validated againstgeneric_rules.pyand (for the items inparse_info) validated against the local config file.Alter theshopifyconfig file so as to change the SKIPPED_FIELDS and create another custom item rule to validate the prices for the discounted items in the data.Runscrapy crawl shopifyuntil 10 fixtures are generated, validated bygeneric_rules.pyand the local config file.Possible things to play around with yourself:Inspect the results of the fixtures.Change the generic rules, global settings and/or the local configurations, and then run an update for a given spider and/or callbacks to test whether the fixtures are updated or whether an error message is displayed.Runtestmaster update spider_name --dynamicin order to re-download the original requests for one of the spiders.Expand the number of fixtures for theshopifyspider in the local config file and runscrapy crawl shopifyso as to generate more fixtures.Delete the fixture andview.jsonfile inolympus/parse_info, edit the body field in the request described in the REQUESTS_TO_ADD list in the local config file, and then runtestmaster update olympus parse_login --newso as to see whether the edited request-dict still logs in successfully.Architecture/LogicI will use theparsecommand to illustrate how this package does its automatic
validation and unittest generation with the middlewares switched on. The logic
involved in theparsecommand is more complicated than the others, so if you
can understand this, you can understand everything. Hopefully,
this will be useful for anyone who desires to fix any bugs in this library.After the command-line args are parsed, theparsecommand fires up the
Scrapy engine to run the specified/inferred spider with the one-off requests specified.Via the logic inparse.py, a one-offstart_requestsfunction is kicked
off which returns the list of requests initially specified inparse.py(request pre-processing happens viaprepare_requestinparse.py, which
sets a custom callback for each request).The requests may then undergo further processing via any
active request-processing subroutines in any activeDownloader Middlewaresin your project (just as if you had generated these requests via a standard
crawl).A response is downloaded and the response is processed as usual by the
variousDownloader Middlewarescomponents which deal with responses (just
as if you had generated the response via a standard crawl).The response is fed to theprocess_spider_inputfunction inTestMasterMiddleware(a standard spider middleware). This function prepares some of the data necessary to
create the testcase file and places it in the response meta object.The response arrives at thecallbackfunction embedded in theprepare_requestfunction inparse.py, which is playing the role of the
spider callback. The response is parsed inside this function by the correct
callback and the processed results are stored inside the response meta object.The response and result (but in this case only the outputted requests are
included in the result and not
the items, because of a quirk with theparselogic) arrive at theprocess_spider_outputfunction inTestMasterMiddleware. The total results, items included, are obtained from the
response meta object where they were stored as described in (6.). These results are validated by the
various custom settings and rules you have established, whether at the project-level
or at the config-specific level, or a combination of both.[Still withinprocess_spider_output] If the results are
valid, and there is room in the fixtures, the fixture is written andview.jsonis updated. If not, an exception is raised and the next request/response
is dealt with.Withinparse.py, global variables have been storing the collected results
as you have processed the 1 or more requests you kicked off initially. When
there are no more requests to be triggered, these results are printed
together (regardless of the failure or success of the results according to
your validation rules).