Package: gerapy-playwright
Gerapy PlaywrightThis is a package for supporting Playwright in Scrapy, also this
package is a module inGerapy.NoteThis package does not work on Windows, please check thisissue, please use this Package on Linux/Mac currently.Installationpip3installgerapy-playwright
playwrightinstallUsageYou can usePlaywrightRequestto specify a request which uses playwright to render.For example:yieldPlaywrightRequest(detail_url,callback=self.parse_detail)And you also need to enablePlaywrightMiddlewareinDOWNLOADER_MIDDLEWARES:DOWNLOADER_MIDDLEWARES={'gerapy_playwright.downloadermiddlewares.PlaywrightMiddleware':543,}Congratulate, you've finished the all of the required configuration.If you run the Spider again, Playwright will be started to render every
web page which you configured the request as PlaywrightRequest.SettingsGerapyPlaywright provides some optional settings.ConcurrencyYou can directly use Scrapy's setting to set Concurrency of Playwright,
for example:CONCURRENT_REQUESTS=3Pretend as Real BrowserSome website will detect WebDriver or Headless, GerapyPlaywright can
pretend Chromium by inject scripts. This is enabled by default.You can close it if website does not detect WebDriver to speed up:GERAPY_PLAYWRIGHT_PRETEND=FalseAlso you can usepretendattribute inPlaywrightRequestto overwrite this
configuration.Logging LevelBy default, Playwright will log all the debug messages, so GerapyPlaywright
configured the logging level of Playwright to WARNING.If you want to see more logs from Playwright, you can change the this setting:importloggingGERAPY_PLAYWRIGHT_LOGGING_LEVEL=logging.DEBUGDownload TimeoutPlaywright may take some time to render the required web page, you can also change this setting, default is30s:# playwright timeoutGERAPY_PLAYWRIGHT_DOWNLOAD_TIMEOUT=30HeadlessBy default, Playwright is running inHeadlessmode, you can also
change it toFalseas you need, default isTrue:GERAPY_PLAYWRIGHT_HEADLESS=FalseWindow SizeYou can also set the width and height of Playwright window:GERAPY_PLAYWRIGHT_WINDOW_WIDTH=1400GERAPY_PLAYWRIGHT_WINDOW_HEIGHT=700Default is 1400, 700.ProxyYou can set a proxy channel via below this config:GERAPY_PLAYWRIGHT_PROXY='http://tps254.kdlapi.com:15818'GERAPY_PLAYWRIGHT_PROXY_CREDENTIAL={'username':'xxx','password':'xxxx'}ScreenshotYou can get screenshot of loaded page, you can passscreenshotargs toPlaywrightRequestas dict:Below are the supported args:type(str): Specify screenshot type, can be eitherjpegorpng. Defaults topng.quality(int): The quality of the image, between 0-100. Not applicable topngimage.full_page(bool): When true, take a screenshot of the full scrollable page. Defaults toFalse.clip(dict): An object which specifies clipping region of the page. This option should have the following fields:x(int): x-coordinate of top-left corner of clip area.y(int): y-coordinate of top-left corner of clip area.width(int): width of clipping area.height(int): height of clipping area.omit_background(bool): Hide default white background and allow capturing screenshot with transparency.timeout(str): Maximum time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout.Check more fromhttps://playwright.dev/python/docs/api/class-page#page-screenshot。For example:yieldPlaywrightRequest(start_url,callback=self.parse_index,wait_for='.item .name',screenshot={'type':'png','full_page':True})then you can get screenshot result inresponse.meta['screenshot']:Simplest save it to file:defparse_index(self,response):withopen('screenshot.png','wb')asf:f.write(response.meta['screenshot'].getbuffer())If you want to enable screenshot for all requests, you can configure it byGERAPY_PLAYWRIGHT_SCREENSHOT.For example:GERAPY_PLAYWRIGHT_SCREENSHOT={'type':'png','full_page':True}PlaywrightRequestPlaywrightRequestprovide args which can override global settings above.url: request urlcallback: callbackwait_until: one of "load", "domcontentloaded", "networkidle"
seehttps://playwright.dev/python/docs/api/class-page#page-wait-for-load-state, default isdomcontentloadedwait_for: wait for some element to load, also supports dictscript: script to executeactions: actions defined for execution of Page objectproxy: use proxy for this time, likehttp://x.x.x.x:xproxy_credential: the proxy credential, like{'username': 'xxxx', 'password': 'xxxx'}sleep: time to sleep after loaded, overrideGERAPY_PLAYWRIGHT_SLEEPtimeout: load timeout, overrideGERAPY_PLAYWRIGHT_DOWNLOAD_TIMEOUTignore_resource_types: ignored resource types, overrideGERAPY_PLAYWRIGHT_IGNORE_RESOURCE_TYPESpretend: pretend as normal browser, overrideGERAPY_PLAYWRIGHT_PRETENDscreenshot: ignored resource types, seehttps://playwright.dev/python/docs/api/class-page#page-screenshot,
overrideGERAPY_PLAYWRIGHT_SCREENSHOTFor example, you can configure PlaywrightRequest as:fromgerapy_playwrightimportPlaywrightRequestdefparse(self,response):yieldPlaywrightRequest(url,callback=self.parse_detail,wait_until='domcontentloaded',wait_for='title',script='() => { return {name: "Germey"} }',sleep=2)Then Playwright will:wait for document to loadwait for title to loadexecuteconsole.log(document)scriptsleep for 2sreturn the rendered web page content, get fromresponse.meta['screenshot']return the script executed result, get fromresponse.meta['script_result']For waiting mechanism controlled by JavaScript, you can use await inscript, for example:js='''async () => {await new Promise(resolve => setTimeout(resolve, 10000));return {'name': 'Germey'}}'''yieldPlaywrightRequest(url,callback=self.parse,script=js)Then you can get the script result fromresponse.meta['script_result'], result is{'name': 'Germey'}.If you think the JavaScript is wired to write, you can use actions argument to define a function to executePythonbased functions, for example:asyncdefexecute_actions(page):awaitpage.evaluate('() => { document.title = "Hello World"; }')return1yieldPlaywrightRequest(url,callback=self.parse,actions=execute_actions)Then you can get the actions result fromresponse.meta['actions_result'], result is1.Also you can define proxy and proxy_credential for each Reqest, for example:yieldPlaywrightRequest(self.base_url,callback=self.parse_index,priority=10,proxy='http://tps254.kdlapi.com:15818',proxy_credential={'username':'xxxx','password':'xxxx'})proxyandproxy_credentialwill override the settingsGERAPY_PLAYWRIGHT_PROXYandGERAPY_PLAYWRIGHT_PROXY_CREDENTIAL.ExampleFor more detail, please seeexample.Also you can directly run with Docker:dockerrungermey/gerapy-playwright-exampleOutputs:2021-12-2716:54:14[scrapy.utils.log]INFO:Scrapy2.2.0started(bot:example)2021-12-2716:54:14[scrapy.utils.log]INFO:Versions:lxml4.7.1.0,libxml22.9.12,cssselect1.1.0,parsel1.6.0,w3lib1.22.0,Twisted21.7.0,Python3.7.9(default,Aug312020,07:22:35)-[Clang10.0.0],pyOpenSSL21.0.0(OpenSSL1.1.1l24Aug2021),cryptography35.0.0,PlatformDarwin-21.1.0-x86_64-i386-64bit2021-12-2716:54:14[scrapy.utils.log]DEBUG:Usingreactor:twisted.internet.asyncioreactor.AsyncioSelectorReactor2021-12-2716:54:14[scrapy.crawler]INFO:Overriddensettings:{'BOT_NAME':'example','CONCURRENT_REQUESTS':1,'NEWSPIDER_MODULE':'example.spiders','RETRY_HTTP_CODES':[403,500,502,503,504],'SPIDER_MODULES':['example.spiders']}2021-12-2716:54:14[scrapy.extensions.telnet]INFO:TelnetPassword:e931b241390ad06a2021-12-2716:54:14[scrapy.middleware]INFO:Enabledextensions:['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage','scrapy.extensions.logstats.LogStats']2021-12-2716:54:14[gerapy.playwright]INFO:playwrightlibrariesalreadyinstalled2021-12-2716:54:14[scrapy.middleware]INFO:Enableddownloadermiddlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','gerapy_playwright.downloadermiddlewares.PlaywrightMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']2021-12-2716:54:14[scrapy.middleware]INFO:Enabledspidermiddlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']2021-12-2716:54:14[scrapy.middleware]INFO:Enableditempipelines:[]2021-12-2716:54:14[scrapy.core.engine]INFO:Spideropened2021-12-2716:54:14[scrapy.extensions.logstats]INFO:Crawled0pages(at0pages/min),scraped0items(at0items/min)2021-12-2716:54:14[scrapy.extensions.telnet]INFO:Telnetconsolelisteningon127.0.0.1:60232021-12-2716:54:14[example.spiders.movie]DEBUG:starturlhttps://antispider1.scrape.center/page/12021-12-2716:54:14[gerapy.playwright]DEBUG:processingrequest<GEThttps://antispider1.scrape.center/page/1>2021-12-2716:54:14[gerapy.playwright]DEBUG:playwright_meta{'wait_until':'domcontentloaded','wait_for':'.item','script':None,'actions':None,'sleep':None,'proxy':None,'proxy_credential':None,'pretend':None,'timeout':None,'screenshot':None}2021-12-2716:54:14[gerapy.playwright]DEBUG:setoptions{'headless':False}cookies[]2021-12-2716:54:16[gerapy.playwright]DEBUG:PRETEND_SCRIPTSisrun2021-12-2716:54:16[gerapy.playwright]DEBUG:timeout102021-12-2716:54:16[gerapy.playwright]DEBUG:crawlinghttps://antispider1.scrape.center/page/12021-12-2716:54:16[gerapy.playwright]DEBUG:requesthttps://antispider1.scrape.center/page/1withoptions{'url':'https://antispider1.scrape.center/page/1','wait_until':'domcontentloaded'}2021-12-2716:54:18[gerapy.playwright]DEBUG:waitingfor.item2021-12-2716:54:18[gerapy.playwright]DEBUG:sleepfor1s2021-12-2716:54:19[gerapy.playwright]DEBUG:takingscreenshotusingargs{'type':'png','full_page':True}2021-12-2716:54:19[gerapy.playwright]DEBUG:closeplaywright2021-12-2716:54:20[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://antispider1.scrape.center/page/1>(referer:None)2021-12-2716:54:20[example.spiders.movie]DEBUG:starturlhttps://antispider1.scrape.center/page/22021-12-2716:54:20[gerapy.playwright]DEBUG:processingrequest<GEThttps://antispider1.scrape.center/page/2>2021-12-2716:54:20[gerapy.playwright]DEBUG:playwright_meta{'wait_until':'domcontentloaded','wait_for':'.item','script':None,'actions':None,'sleep':None,'proxy':None,'proxy_credential':None,'pretend':None,'timeout':None,'screenshot':None}2021-12-2716:54:20[gerapy.playwright]DEBUG:setoptions{'headless':False}2021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/12021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/22021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/32021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/42021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/52021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/62021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/72021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/82021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/92021-12-2716:54:20[example.spiders.movie]INFO:detailurlhttps://antispider1.scrape.center/detail/10
cookies[]2021-12-2716:54:21[gerapy.playwright]DEBUG:PRETEND_SCRIPTSisrun2021-12-2716:54:21[gerapy.playwright]DEBUG:timeout102021-12-2716:54:21[gerapy.playwright]DEBUG:crawlinghttps://antispider1.scrape.center/page/22021-12-2716:54:21[gerapy.playwright]DEBUG:requesthttps://antispider1.scrape.center/page/2withoptions{'url':'https://antispider1.scrape.center/page/2','wait_until':'domcontentloaded'}2021-12-2716:54:23[gerapy.playwright]DEBUG:waitingfor.item2021-12-2716:54:24[gerapy.playwright]DEBUG:sleepfor1s2021-12-2716:54:25[gerapy.playwright]DEBUG:takingscreenshotusingargs{'type':'png','full_page':True}2021-12-2716:54:25[gerapy.playwright]DEBUG:closeplaywright2021-12-2716:54:25[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://antispider1.scrape.center/page/2>(referer:None)2021-12-2716:54:25[gerapy.playwright]DEBUG:processingrequest<GEThttps://antispider1.scrape.center/detail/10>2021-12-2716:54:25[gerapy.playwright]DEBUG:playwright_meta{'wait_until':'domcontentloaded','wait_for':'.item','script':None,'actions':None,'sleep':None,'proxy':None,'proxy_credential':None,'pretend':None,'timeout':None,'screenshot':None}2021-12-2716:54:25[gerapy.playwright]DEBUG:setoptions{'headless':False}...