Package: scrapy-puppeteer-client
Scrapy-puppeteer-clientThis package aims to manage Chrome browser withPuppeteerfromScrapyspiders.
This allows to scrape sites that require JS to function properly and to make the scraper more similar to humans.
It is a client library forscrapy-puppeteer-service.⚠️ This repository is under development.This project is under development. Use it at your own risk.InstallationUsing pip (master branch):$pipinstallscrapy-puppeteer-clientConfigurationYou should havescrapy-puppeteer-servicestarted.
Then add its URL tosettings.pyand enable puppeteer downloader middleware:DOWNLOADER_MIDDLEWARES={'scrapypuppeteer.middleware.PuppeteerServiceDownloaderMiddleware':1042}PUPPETEER_SERVICE_URL='http://localhost:3000'Basic usageUsescrapypuppeteer.PuppeteerRequestinstead ofscrapy.Requestto render URLs with Puppeteer:importscrapyfromscrapypuppeteerimportPuppeteerRequestclassMySpider(scrapy.Spider):...defstart_requests(self):yieldPuppeteerRequest('https://exapmle.com',callback=self.parse)defparse(self,response):links=response.css(...)...Advanced usagePuppeteerRequest's first argument is a browser action.
Avalable actions are defined inscrapypuppeteer.actionsmodule as subclasses ofPuppeteerServiceAction.
Passing a URL into request is a shortcut forGoTo(url)action.Here is the list of available actions:GoTo(url, options)- navigate to URLGoForward(options)- navigate forward in historyGoBack(options)- navigate back in historyClick(selector, click_options, wait_options)- click on element on pageScroll(selector, wait_options)- scroll pageScreenshot(options)- take screenshotRecaptchaSolver(solve_recaptcha)- find or solve recaptcha on pageCustomJsAction(js_function)- evaluate JS function on pageAvailable options essentially mirrorservicemethod parameters, which in turn mirror puppeteer API functions to some extent.
Seescrapypuppeteer.actionsmodule for details.You may passclose_page=Falseoption to a request to retain browser tab and its state after request's completion.
Then useresponse.followto continue interacting with the same tab:importscrapyfromscrapypuppeteerimportPuppeteerRequestfromscrapypuppeteer.actionsimportClickclassMySpider(scrapy.Spider):...defstart_requests(self):yieldPuppeteerRequest('https://exapmle.com',close_page=False,callback=self.parse)defparse(self,response):...# parse and yield some items...next_page_selector='button.next-page-or-smth'ifresponse.css(next_page_selector):yieldresponse.follow(Click(next_page_selector,wait_options={'selectorOrTimeout':3000}),# wait 3 secondsclose_page=False,callback=self.parse)On your first request service will create new incognito browser context and new page in it.
Their ids will be in returned in response object ascontext_idandpage_idattributes.
Following such response means passing context and page ids to next request.
You also may specify requests context and page ids directly.Once your spider is closed, middleware will take care of closing all used browser contexts.One may customize whichPuppeteerRequest's headers will be sent to remote website by the service
viainclude_headersattribute in request or globally withPUPPETEER_INCLUDE_HEADERSsetting.
Available values are True (all headers), False (no headers) or list of header names.
By default, only cookies are sent.You would also like to send meta with your request. By default, you are not allowed to do this
in order to sustain backward compatibility. You can change this behaviour by settingPUPPETEER_INCLUDE_METAto True.Automatic recaptcha solvingEnable PuppeteerRecaptchaDownloaderMiddleware to automatically solve recaptcha during scraping. We do not recommend
to use RecaptchaSolver action when the middleware works.DOWNLOADER_MIDDLEWARES={'scrapypuppeteer.middleware.PuppeteerRecaptchaDownloaderMiddleware':1041,'scrapypuppeteer.middleware.PuppeteerServiceDownloaderMiddleware':1042}Note that the number of RecaptchaMiddleware has to be lower than ServiceMiddleware's.
You must provide some settings to use the middleware:PUPPETEER_INCLUDE_META=True# Essential to send metaRECAPTCHA_ACTIVATION=True# Enables the middlewareRECAPTCHA_SOLVING=False# Automatic recaptcha solvingRECAPTCHA_SUBMIT_SELECTORS={# Selectors for "submit recaptcha" button'www.google.com/recaptcha/api2/demo':'',# No selectors needed}If you set RECAPTCHA_SOLVING to False the middleware will try to find captcha
and will notify you about number of found captchas on the page.If you don't want the middleware to work on specific request you may provide special meta key:'dont_recaptcha': True.
In this case RecaptchaMiddleware will just skip the request.TODOskeleton that could handle goto, click, scroll, and actionsheaders and cookies managementproxy support for puppeteererror handling for requestshar support