Package: scrapyer-gerapy-pyppeteer
Gerapy PyppeteerThis is a package for supporting pyppeteer in Scrapy, also this
package is a module inGerapy.Installationpip3installgerapy-pyppeteerUsageYou can usePyppeteerRequestto specify a request which uses pyppeteer to render.For example:yieldPyppeteerRequest(detail_url,callback=self.parse_detail)And you also need to enablePyppeteerMiddlewareinDOWNLOADER_MIDDLEWARES:DOWNLOADER_MIDDLEWARES={'gerapy_pyppeteer.downloadermiddlewares.PyppeteerMiddleware':543,}Congratulate, you've finished the all of the required configuration.If you run the Spider again, Pyppeteer will be started to render every
web page which you configured the request as PyppeteerRequest.SettingsGerapyPyppeteer provides some optional settings.ConcurrencyYou can directly use Scrapy's setting to set Concurrency of Pyppeteer,
for example:CONCURRENT_REQUESTS=3Pretend as Real BrowserSome website will detect WebDriver or Headless, GerapyPyppeteer can
pretend Chromium by inject scripts. This is enabled by default.You can close it if website does not detect WebDriver to speed up:GERAPY_PYPPETEER_PRETEND=FalseAlso you can usepretendattribute inPyppeteerRequestto overwrite this
configuration.Logging LevelBy default, Pyppeteer will log all the debug messages, so GerapyPyppeteer
configured the logging level of Pyppeteer to WARNING.If you want to see more logs from Pyppeteer, you can change the this setting:importloggingGERAPY_PYPPETEER_LOGGING_LEVEL=logging.DEBUGDownload TimeoutPyppeteer may take some time to render the required web page, you can also change this setting, default is30s:# pyppeteer timeoutGERAPY_PYPPETEER_DOWNLOAD_TIMEOUT=30HeadlessBy default, Pyppeteer is running inHeadlessmode, you can also
change it toFalseas you need, default isTrue:GERAPY_PYPPETEER_HEADLESS=FalseWindow SizeYou can also set the width and height of Pyppeteer window:GERAPY_PYPPETEER_WINDOW_WIDTH=1400GERAPY_PYPPETEER_WINDOW_HEIGHT=700Default is 1400, 700.Pyppeteer ArgsYou can also change the args of Pyppeteer, such asdumpio,devtools, etc.Optional settings and their default values:GERAPY_PYPPETEER_DUMPIO=FalseGERAPY_PYPPETEER_DEVTOOLS=FalseGERAPY_PYPPETEER_EXECUTABLE_PATH=NoneGERAPY_PYPPETEER_DISABLE_EXTENSIONS=TrueGERAPY_PYPPETEER_HIDE_SCROLLBARS=TrueGERAPY_PYPPETEER_MUTE_AUDIO=TrueGERAPY_PYPPETEER_NO_SANDBOX=TrueGERAPY_PYPPETEER_DISABLE_SETUID_SANDBOX=TrueGERAPY_PYPPETEER_DISABLE_GPU=TrueDisable loading of specific resource typeYou can disable the loading of specific resource type to
decrease the loading time of web page. You can configure
the disabled resource types usingGERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES:GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES=[]For example, if you want to disable the loading of css and javascript,
you can set as below:GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES=['stylesheet','script']All of the optional resource type list:document: the Original HTML documentstylesheet: CSS filesscript: JavaScript filesimage: Imagesmedia: Media files such as audios or videosfont: Fonts filestexttrack: Text Track filesxhr: Ajax Requestsfetch: Fetch Requestseventsource: Event Sourcewebsocket: Websocketmanifest: Manifest filesother: Other filesScreenshotYou can get screenshot of loaded page, you can passscreenshotargs toPyppeteerRequestas dict:type(str): Specify screenshot type, can be eitherjpegorpng. Defaults topng.quality(int): The quality of the image, between 0-100. Not applicable topngimage.fullPage(bool): When true, take a screenshot of the full scrollable page. Defaults toFalse.clip(dict): An object which specifies clipping region of the page. This option should have the following fields:x(int): x-coordinate of top-left corner of clip area.y(int): y-coordinate of top-left corner of clip area.width(int): width of clipping area.height(int): height of clipping area.omitBackground(bool): Hide default white background and allow capturing screenshot with transparency.encoding(str): The encoding of the image, can be eitherbase64orbinary. Defaults tobinary. If binary it will returnBytesIOobject.For example:yieldPyppeteerRequest(start_url,callback=self.parse_index,wait_for='.item .name',screenshot={'type':'png','fullPage':True})then you can get screenshot result inresponse.meta['screenshot']:Simplest save it to file:defparse_index(self,response):withopen('screenshot.png','wb')asf:f.write(response.meta['screenshot'].getbuffer())If you want to enable screenshot for all requests, you can configure it byGERAPY_PYPPETEER_SCREENSHOT.For example:GERAPY_PYPPETEER_SCREENSHOT={'type':'png','fullPage':True}PyppeteerRequestPyppeteerRequestprovide args which can override global settings above.url: request urlcallback: callbackone of "load", "domcontentloaded", "networkidle0", "networkidle2".
seehttps://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goto, default isdomcontentloadedwait_for: wait for some element to load, also supports dictscript: script to executeproxy: use proxy for this time, likehttp://x.x.x.x:xsleep: time to sleep after loaded, overrideGERAPY_PYPPETEER_SLEEPtimeout: load timeout, overrideGERAPY_PYPPETEER_DOWNLOAD_TIMEOUTignore_resource_types: ignored resource types, overrideGERAPY_PYPPETEER_IGNORE_RESOURCE_TYPESpretend: pretend as normal browser, overrideGERAPY_PYPPETEER_PRETENDscreenshot: ignored resource types, seehttps://miyakogi.github.io/pyppeteer/_modules/pyppeteer/page.html#Page.screenshot,
overrideGERAPY_PYPPETEER_SCREENSHOTFor example, you can configure PyppeteerRequest as:fromgerapy_pyppeteerimportPyppeteerRequestdefparse(self,response):yieldPyppeteerRequest(url,callback=self.parse_detail,wait_until='domcontentloaded',wait_for='title',script='() => { return {name: "Germey"} }',sleep=2)Then Pyppeteer will:wait for document to loadwait for title to loadexecuteconsole.log(document)scriptsleep for 2sreturn the rendered web page content, get fromresponse.meta['screenshot']return the script executed result, get fromresponse.meta['script_result']For waiting mechanism controlled by JavaScript, you can use await inscript, for example:js='''async () => {await new Promise(resolve => setTimeout(resolve, 10000));return {'name': 'Germey'}}'''yieldPyppeteerRequest(url,callback=self.parse,script=js)Then you can get the script result fromresponse.meta['script_result'], result is{'name': 'Germey'}.If you think the JavaScript is wired to write, you can use actions argument to define a function to executePythonbased functions, for example:asyncdefexecute_actions(page:Page):awaitpage.evaluate('() => { document.title = "Hello World"; }')return1yieldPyppeteerRequest(url,callback=self.parse,actions=execute_actions)Then you can get the actions result fromresponse.meta['actions_result'], result is1.ExampleFor more detail, please seeexample.Also you can directly run with Docker:docker run germey/gerapy-pyppeteer-exampleOutputs:2020-07-1301:49:13[scrapy.utils.log]INFO:Scrapy2.2.0started(bot:example)2020-07-1301:49:13[scrapy.utils.log]INFO:Versions:lxml4.3.3.0,libxml22.9.9,cssselect1.1.0,parsel1.6.0,w3lib1.22.0,Twisted20.3.0,Python3.7.7(default,May62020,04:59:01)-[Clang4.0.1(tags/RELEASE_401/final)],pyOpenSSL19.1.0(OpenSSL1.1.1d10Sep2019),cryptography2.8,PlatformDarwin-19.4.0-x86_64-i386-64bit2020-07-1301:49:13[scrapy.utils.log]DEBUG:Usingreactor:twisted.internet.asyncioreactor.AsyncioSelectorReactor2020-07-1301:49:13[scrapy.crawler]INFO:Overriddensettings:{'BOT_NAME':'example','CONCURRENT_REQUESTS':3,'NEWSPIDER_MODULE':'example.spiders','RETRY_HTTP_CODES':[403,500,502,503,504],'SPIDER_MODULES':['example.spiders']}2020-07-1301:49:13[scrapy.extensions.telnet]INFO:TelnetPassword:83c276fb41754bd02020-07-1301:49:13[scrapy.middleware]INFO:Enabledextensions:['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage','scrapy.extensions.logstats.LogStats']2020-07-1301:49:13[scrapy.middleware]INFO:Enableddownloadermiddlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','gerapy_pyppeteer.downloadermiddlewares.PyppeteerMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']2020-07-1301:49:13[scrapy.middleware]INFO:Enabledspidermiddlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']2020-07-1301:49:13[scrapy.middleware]INFO:Enableditempipelines:[]2020-07-1301:49:13[scrapy.core.engine]INFO:Spideropened2020-07-1301:49:13[scrapy.extensions.logstats]INFO:Crawled0pages(at0pages/min),scraped0items(at0items/min)2020-07-1301:49:13[scrapy.extensions.telnet]INFO:Telnetconsolelisteningon127.0.0.1:60232020-07-1301:49:13[example.spiders.book]INFO:crawlinghttps://dynamic5.scrape.center/page/12020-07-1301:49:13[gerapy.pyppeteer]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/page/1>2020-07-1301:49:13[gerapy.pyppeteer]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:14[gerapy.pyppeteer]DEBUG:crawlinghttps://dynamic5.scrape.center/page/12020-07-1301:49:19[gerapy.pyppeteer]DEBUG:waitingfor.item.namefinished2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:waitfor.item.namefinished2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:closepyppeteer2020-07-1301:49:20[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://dynamic5.scrape.center/page/1>(referer:None)2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/26898909>2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/26861389>2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/26855315>2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:20[gerapy.pyppeteer]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:21[gerapy.pyppeteer]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:21[gerapy.pyppeteer]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/268553152020-07-1301:49:21[gerapy.pyppeteer]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/268613892020-07-1301:49:21[gerapy.pyppeteer]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/268989092020-07-1301:49:24[gerapy.pyppeteer]DEBUG:waitingfor.item.namefinished2020-07-1301:49:24[gerapy.pyppeteer]DEBUG:waitfor.item.namefinished2020-07-1301:49:24[gerapy.pyppeteer]DEBUG:closepyppeteer2020-07-1301:49:24[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://dynamic5.scrape.center/detail/26861389>(referer:https://dynamic5.scrape.center/page/1)2020-07-1301:49:24[gerapy.pyppeteer]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/page/2>2020-07-1301:49:24[gerapy.pyppeteer]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:25[scrapy.core.scraper]DEBUG:Scrapedfrom<200https://dynamic5.scrape.center/detail/26861389>{'name':'壁穴ヘブンホール','score':'5.6','tags':['BL漫画','小基漫','BL','『又腐又基』','BLコミック']}2020-07-1301:49:25[gerapy.pyppeteer]DEBUG:waitingfor.item.namefinished2020-07-1301:49:25[gerapy.pyppeteer]DEBUG:crawlinghttps://dynamic5.scrape.center/page/22020-07-1301:49:26[gerapy.pyppeteer]DEBUG:waitfor.item.namefinished2020-07-1301:49:26[gerapy.pyppeteer]DEBUG:closepyppeteer2020-07-1301:49:26[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://dynamic5.scrape.center/detail/26855315>(referer:https://dynamic5.scrape.center/page/1)2020-07-1301:49:26[gerapy.pyppeteer]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/27047626>2020-07-1301:49:26[gerapy.pyppeteer]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:26[scrapy.core.scraper]DEBUG:Scrapedfrom<200https://dynamic5.scrape.center/detail/26855315>{'name':'冒险小虎队','score':'9.4','tags':['冒险小虎队','童年','冒险','推理','小时候读的']}2020-07-1301:49:26[gerapy.pyppeteer]DEBUG:waitingfor.item.namefinished2020-07-1301:49:26[gerapy.pyppeteer]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/270476262020-07-1301:49:27[gerapy.pyppeteer]DEBUG:waitfor.item.namefinished2020-07-1301:49:27[gerapy.pyppeteer]DEBUG:closepyppeteer
...Trouble ShootingPyppeteer does not start properlyChromium download speed is too slow, it can not be used normally.Here are two solutions:Solution 1 (Recommended)Modify drive download source atpyppeteer/chromium_downloader.pyline 22:# Default：DEFAULT_DOWNLOAD_HOST='https://storage.googleapis.com'# modifyDEFAULT_DOWNLOAD_HOST=http://npm.taobao.org/mirrorSolution 2Modify drive execution path atpyppeteer/chromium_downloader.pyline 45:# Default：chromiumExecutable={'linux':DOWNLOADS_FOLDER/REVISION/'chrome-linux'/'chrome','mac':(DOWNLOADS_FOLDER/REVISION/'chrome-mac'/'Chromium.app'/'Contents'/'MacOS'/'Chromium'),'win32':DOWNLOADS_FOLDER/REVISION/windowsArchive/'chrome.exe','win64':DOWNLOADS_FOLDER/REVISION/windowsArchive/'chrome.exe',}You can find your own operating system, modify your chrome or chrome executable path.