Package: gerapy-selenium
Gerapy SeleniumThis is a package for supporting selenium in Scrapy, also this
package is a module inGerapy.Installationpip3installgerapy-seleniumUsageYou can useSeleniumRequestto specify a request which uses selenium to render.For example:yieldSeleniumRequest(detail_url,callback=self.parse_detail)And you also need to enableSeleniumMiddlewareinDOWNLOADER_MIDDLEWARES:DOWNLOADER_MIDDLEWARES={'gerapy_selenium.downloadermiddlewares.SeleniumMiddleware':543,}Congratulate, you've finished the all of the required configuration.If you run the Spider again, Selenium will be started to render every
web page which you configured the request as SeleniumRequest.SettingsGerapySelenium provides some optional settings.ConcurrencyYou can directly use Scrapy's setting to set Concurrency of Selenium,
for example:CONCURRENT_REQUESTS=3Pretend as Real BrowserSome website will detect WebDriver or Headless, GerapySelenium can
pretend Chromium by inject scripts. This is enabled by default.You can close it if website does not detect WebDriver to speed up:GERAPY_SELENIUM_PRETEND=FalseAlso you can usepretendattribute inSeleniumRequestto overwrite this
configuration.Logging LevelBy default, Selenium will log all the debug messages, so GerapySelenium
configured the logging level of Selenium to WARNING.If you want to see more logs from Selenium, you can change the this setting:importloggingGERAPY_SELENIUM_LOGGING_LEVEL=logging.DEBUGDownload TimeoutSelenium may take some time to render the required web page, you can also change this setting, default is30s:# selenium timeoutGERAPY_SELENIUM_DOWNLOAD_TIMEOUT=30HeadlessBy default, Selenium is running inHeadlessmode, you can also
change it toFalseas you need, default isTrue:GERAPY_SELENIUM_HEADLESS=FalseWindow SizeYou can also set the width and height of Selenium window:GERAPY_SELENIUM_WINDOW_WIDTH=1400GERAPY_SELENIUM_WINDOW_HEIGHT=700Default is 1400, 700.SeleniumRequestSeleniumRequestprovide args which can override global settings above.url: request urlcallback: callbackwait_for: wait for some element to load, also supports dictscript: script to executeproxy: use proxy for this time, likehttp://x.x.x.x:xsleep: time to sleep after loaded, overrideGERAPY_SELENIUM_SLEEPtimeout: load timeout, overrideGERAPY_SELENIUM_DOWNLOAD_TIMEOUTpretend: pretend as normal browser, overrideGERAPY_SELENIUM_PRETENDscreenshot: ignored resource types, seehttps://miyakogi.github.io/selenium/_modules/selenium/page.html#Page.screenshot,
overrideGERAPY_SELENIUM_SCREENSHOTFor example, you can configure SeleniumRequest as:fromgerapy_seleniumimportSeleniumRequestdefparse(self,response):yieldSeleniumRequest(url,callback=self.parse_detail,wait_for='title',script='() => { console.log(document) }',sleep=2)Then Selenium will:wait for title to loadexecuteconsole.log(document)scriptsleep for 2sreturn the rendered web page contentExampleFor more detail, please seeexample.Also you can directly run with Docker:docker run germey/gerapy-selenium-exampleOutputs:2020-07-1301:49:13[scrapy.utils.log]INFO:Scrapy2.2.0started(bot:example)2020-07-1301:49:13[scrapy.utils.log]INFO:Versions:lxml4.3.3.0,libxml22.9.9,cssselect1.1.0,parsel1.6.0,w3lib1.22.0,Twisted20.3.0,Python3.7.7(default,May62020,04:59:01)-[Clang4.0.1(tags/RELEASE_401/final)],pyOpenSSL19.1.0(OpenSSL1.1.1d10Sep2019),cryptography2.8,PlatformDarwin-19.4.0-x86_64-i386-64bit2020-07-1301:49:13[scrapy.utils.log]DEBUG:Usingreactor:twisted.internet.asyncioreactor.AsyncioSelectorReactor2020-07-1301:49:13[scrapy.crawler]INFO:Overriddensettings:{'BOT_NAME':'example','CONCURRENT_REQUESTS':3,'NEWSPIDER_MODULE':'example.spiders','RETRY_HTTP_CODES':[403,500,502,503,504],'SPIDER_MODULES':['example.spiders']}2020-07-1301:49:13[scrapy.extensions.telnet]INFO:TelnetPassword:83c276fb41754bd02020-07-1301:49:13[scrapy.middleware]INFO:Enabledextensions:['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage','scrapy.extensions.logstats.LogStats']2020-07-1301:49:13[scrapy.middleware]INFO:Enableddownloadermiddlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','gerapy_selenium.downloadermiddlewares.SeleniumMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']2020-07-1301:49:13[scrapy.middleware]INFO:Enabledspidermiddlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']2020-07-1301:49:13[scrapy.middleware]INFO:Enableditempipelines:[]2020-07-1301:49:13[scrapy.core.engine]INFO:Spideropened2020-07-1301:49:13[scrapy.extensions.logstats]INFO:Crawled0pages(at0pages/min),scraped0items(at0items/min)2020-07-1301:49:13[scrapy.extensions.telnet]INFO:Telnetconsolelisteningon127.0.0.1:60232020-07-1301:49:13[example.spiders.book]INFO:crawlinghttps://dynamic5.scrape.center/page/12020-07-1301:49:13[gerapy.selenium]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/page/1>2020-07-1301:49:13[gerapy.selenium]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:14[gerapy.selenium]DEBUG:crawlinghttps://dynamic5.scrape.center/page/12020-07-1301:49:19[gerapy.selenium]DEBUG:waitingfor.item.namefinished2020-07-1301:49:20[gerapy.selenium]DEBUG:waitfor.item.namefinished2020-07-1301:49:20[gerapy.selenium]DEBUG:closeselenium2020-07-1301:49:20[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://dynamic5.scrape.center/page/1>(referer:None)2020-07-1301:49:20[gerapy.selenium]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/26898909>2020-07-1301:49:20[gerapy.selenium]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/26861389>2020-07-1301:49:20[gerapy.selenium]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/26855315>2020-07-1301:49:20[gerapy.selenium]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:20[gerapy.selenium]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:21[gerapy.selenium]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:21[gerapy.selenium]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/268553152020-07-1301:49:21[gerapy.selenium]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/268613892020-07-1301:49:21[gerapy.selenium]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/268989092020-07-1301:49:24[gerapy.selenium]DEBUG:waitingfor.item.namefinished2020-07-1301:49:24[gerapy.selenium]DEBUG:waitfor.item.namefinished2020-07-1301:49:24[gerapy.selenium]DEBUG:closeselenium2020-07-1301:49:24[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://dynamic5.scrape.center/detail/26861389>(referer:https://dynamic5.scrape.center/page/1)2020-07-1301:49:24[gerapy.selenium]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/page/2>2020-07-1301:49:24[gerapy.selenium]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:25[scrapy.core.scraper]DEBUG:Scrapedfrom<200https://dynamic5.scrape.center/detail/26861389>{'name':'壁穴ヘブンホール','score':'5.6','tags':['BL漫画','小基漫','BL','『又腐又基』','BLコミック']}2020-07-1301:49:25[gerapy.selenium]DEBUG:waitingfor.item.namefinished2020-07-1301:49:25[gerapy.selenium]DEBUG:crawlinghttps://dynamic5.scrape.center/page/22020-07-1301:49:26[gerapy.selenium]DEBUG:waitfor.item.namefinished2020-07-1301:49:26[gerapy.selenium]DEBUG:closeselenium2020-07-1301:49:26[scrapy.core.engine]DEBUG:Crawled(200)<GEThttps://dynamic5.scrape.center/detail/26855315>(referer:https://dynamic5.scrape.center/page/1)2020-07-1301:49:26[gerapy.selenium]DEBUG:processingrequest<GEThttps://dynamic5.scrape.center/detail/27047626>2020-07-1301:49:26[gerapy.selenium]DEBUG:setoptions{'headless':True,'dumpio':False,'devtools':False,'args':['--window-size=1400,700','--disable-extensions','--hide-scrollbars','--mute-audio','--no-sandbox','--disable-setuid-sandbox','--disable-gpu']}2020-07-1301:49:26[scrapy.core.scraper]DEBUG:Scrapedfrom<200https://dynamic5.scrape.center/detail/26855315>{'name':'冒险小虎队','score':'9.4','tags':['冒险小虎队','童年','冒险','推理','小时候读的']}2020-07-1301:49:26[gerapy.selenium]DEBUG:waitingfor.item.namefinished2020-07-1301:49:26[gerapy.selenium]DEBUG:crawlinghttps://dynamic5.scrape.center/detail/270476262020-07-1301:49:27[gerapy.selenium]DEBUG:waitfor.item.namefinished2020-07-1301:49:27[gerapy.selenium]DEBUG:closeselenium
...