Package: scrapy-wayback
scrapy-waybackScrapy middleware with wayback machine support for more robust scrapers.Dependencies :globe_with_meridians:Python 3.7Scrapy 2.4.0Wayback 0.3.0Installation :inbox_tray:This is a python package hosted on pypi, so to install simply run the following command:pip install scrapy-waybackSettingsWAYBACK_MACHINE_FALLBACK_ENABLED (Optional)Whether falling back to wayback machine after a failed request is enabled (defaults to true).Meta field to enable/disable this per request is:wayback_machine_fallback_enabledWAYBACK_MACHINE_PROXY_ENABLED (Optional)Whether proxying to wayback machine before a request is made is enabled (defaults to false).Meta field to enable/disable this per request is:wayback_machine_proxy_enabledWAYBACK_MACHINE_PROXY_FALLTHROUGH_ENABLED (Optional)Whether when proxying to wayback machine and an error occurs, that the request should continue to the original URL as per normal (defaults to true). Note that this will not have an effect if the wayback machine proxy is not enabled first.Meta field to enable/disable this per request is:wayback_machine_proxy_fallthrough_enabledUsage example :eyes:In order to use this plugin simply add the following settings and substitute your variables:DOWNLOADER_MIDDLEWARES={"waybackmiddleware.middleware.WaybackMachineDownloaderMiddleware":630}This will immediately allow you begin using the wayback machine as a fallback when one of your requests fail. In order to use it as a proxy you can add the following to your settings:WAYBACK_MACHINE_PROXY_ENABLED=TrueThis will make every request hit the wayback machine for a response first, before hitting the original server. If you want to avoid hitting the original server entirely, put the following in your settings (as well as the above):WAYBACK_MACHINE_PROXY_FALLTHROUGH_ENABLED=FalseThis will ensure that your scraper never hits the original servers, just what has been recorded by the wayback machine.Whenever you receive a response from the wayback machine middleware, it will use the classWaybackMachineResponse. It subclassesscrapy.http.HtmlResponseso you can use it like a normal response, however it has some other goodies:defparse(self,response):whileTrue:ifresponseisNone:returnprint(f"Response{response.request.url}at{response.timestamp.isoformat()}")response=response.earlier_response()This will allow you to go through the history one by one to get the earlier snapshots of the page. If you are interested in the response that the wayback middleware recovered, use theoriginal_responseattribute.In order to perform a request that will yield the whole archived contents of a site, you can do the following:importscrapyfromwaybackmiddleware.requestimportWaybackMachineRequestfromwaybackmiddleware.responseimportWaybackMachineResponseclassArchiveScraper(scrapy.Spider):defstart_requests():yieldWaybackMachineRequest("http://www.walmart.com")defparse(self,response):print(f"Archive of{response.url}at{response.timestamp}")ifisinstance(response,WaybackMachineResponse):next_response=response.earlier_response()ifnext_responseisnotNone:yieldnext_response.request_for_response(self.parse)This will send all archived contents ofwalmart.comto theparsecallback (called multiple times).License :memo:The project is available under theMIT License.