Package: scrapy-pyppeteer
Pyppeteer integration for ScrapyThis project provides a Scrapy Download Handler which performs requests usingPyppeteer. It can be used to handle
pages that require JavaScript. This package does not interfere with regular
Scrapy workflows such as request scheduling or item processing.MotivationAfter the release ofversion 2.0,
which includes partialcoroutine syntax supportand experimentalasyncio support, Scrapy allows
to integrateasyncio-based projects such asPyppeteer.RequirementsPython 3.6+Scrapy 2.0+Pyppeteer 0.0.23+Installation$ pip install scrapy-pyppeteerConfigurationReplace the defaulthttpandhttpsDownload Handlers throughDOWNLOAD_HANDLERS:DOWNLOAD_HANDLERS={"http":"scrapy_pyppeteer.handler.ScrapyPyppeteerDownloadHandler","https":"scrapy_pyppeteer.handler.ScrapyPyppeteerDownloadHandler",}Note that theScrapyPyppeteerDownloadHandlerclass inherits from the defaulthttp/httpshandler, and it will only use Pyppeteer for requests that are
explicitly marked (see the "Basic usage" section for details).Also, be sure toinstall theasyncio-based Twisted reactor:TWISTED_REACTOR="twisted.internet.asyncioreactor.AsyncioSelectorReactor"scrapy-pyppeteeraccepts the following settings:PYPPETEER_LAUNCH_OPTIONS(typedict, default{})A dictionary with options to be passed when launching the Browser.
See the docs forpyppeteer.launcher.launchPYPPETEER_NAVIGATION_TIMEOUT(typeOptional[int], defaultNone)Default timeout (in milliseconds) to be used when requesting pages by Pyppeteer. IfNoneor unset,
the default value will be used (30000 ms at the time of writing this).
See the docs forpyppeteer.page.Page.setDefaultNavigationTimeoutPYPPETEER_PAGE_COROUTINE_TIMEOUT(typeOptional[Union[int, float]], defaultNone)Default timeout (in milliseconds) to be passed when usingpage coroutines,
such aswaitForSelectororwaitForXPath.
IfNoneor unset, the default value will be used (30000 ms at the time of writing this).Basic usageSet thepyppeteerRequest.metakey to download a request using Pyppeteer:importscrapyclassAwesomeSpider(scrapy.Spider):name="awesome"defstart_requests(self):# GET requestyieldscrapy.Request("https://httpbin.org/get",meta={"pyppeteer":True})# POST requestyieldscrapy.FormRequest(url="https://httpbin.org/post",formdata={"foo":"bar"},meta={"pyppeteer":True},)defparse(self,response):# 'response' contains the page as seen by the browseryield{"url":response.url}Page coroutinesA sorted iterable (list,tupleordict, for instance) could be passed
in thepyppeteer_page_coroutinesRequest.metakey to request coroutines to be awaited on thePagebefore returning the finalResponseto the callback.This is useful when you need to perform certain actions on a page, like scrolling
down or clicking links, and you want everything to count as a single Scrapy
Response, containing the final result.Supported actionsscrapy_pyppeteer.page.PageCoroutine(method: str, *args, **kwargs):Represents a coroutine to be awaited on apyppeteer.page.Pageobject,
such as "click", "screenshot", "evaluate", etc.methodshould be the name of the coroutine,*argsand**kwargsare passed to the function call.The coroutine result will be stored in thePageCoroutine.resultattributeFor instance,PageCoroutine("screenshot",options={"path":"quotes.png","fullPage":True})produces the same effect as:# 'page' is a pyppeteer.page.Page objectawaitpage.screenshot(options={"path":"quotes.png","fullPage":True})scrapy_pyppeteer.page.NavigationPageCoroutine(method: str, *args, **kwargs):Subclass ofPageCoroutine. It waits for a navigation event: use this when you know
a coroutine will trigger a navigation event, for instance when clicking on a link.
This forces aPage.waitForNavigation()call wrapped inasyncio.gather, as recommended inthe Pyppeteer docs.For instance,NavigationPageCoroutine("click",selector="a")produces the same effect as:# 'page' is a pyppeteer.page.Page objectawaitasyncio.gather(page.waitForNavigation(),page.click(selector="a"),)Receiving the Page object in the callbackSpecifyingpyppeteer.page.Pageas the type for a callback argument will result
in the correspondingPageobject being injected in the callback. In order to
able toawaitcoroutines on the providedPageobject, the callback needs to
be defined as a coroutine function (async def).importscrapyimportpyppeteerclassAwesomeSpiderWithPage(scrapy.Spider):name="page"defstart_requests(self):yieldscrapy.Request("https://example.org",meta={"pyppeteer":True})asyncdefparse(self,response,page:pyppeteer.page.Page):title=awaitpage.title()# "Example Domain"yield{"title":title}awaitpage.close()Notes:In order to avoid memory issues, it is recommended to manually close the page
by awaiting thePage.closecoroutine.Any network operations resulting from awaiting a coroutine on aPageobject
(goto,goBack, etc) will be executed directly by Pyppeteer, bypassing the
Scrapy request workflow (Scheduler, Middlewares, etc).ExamplesClick on a link, save the resulting page as PDFimportscrapyfromscrapy_pyppeteer.pageimportPageCoroutine,NavigationPageCoroutineclassClickAndSavePdfSpider(scrapy.Spider):name="pdf"defstart_requests(self):yieldscrapy.Request(url="https://example.org",meta=dict(pyppeteer=True,pyppeteer_page_coroutines={"click":NavigationPageCoroutine("click",selector="a"),"pdf":PageCoroutine("pdf",options={"path":"/tmp/file.pdf"}),},),)defparse(self,response):pdf_bytes=response.meta["pyppeteer_page_coroutines"]["pdf"].resultwithopen("iana.pdf","wb")asfp:fp.write(pdf_bytes)yield{"url":response.url}# response.url is "https://www.iana.org/domains/reserved"Scroll down on an infinite scroll page, take a screenshot of the full pageimportscrapyimportpyppeteerfromscrapy_pyppeteer.pageimportPageCoroutineclassScrollSpider(scrapy.Spider):name="scroll"defstart_requests(self):yieldscrapy.Request(url="http://quotes.toscrape.com/scroll",meta=dict(pyppeteer=True,pyppeteer_page_coroutines=[PageCoroutine("waitForSelector","div.quote"),PageCoroutine("evaluate","window.scrollBy(0, document.body.scrollHeight)"),PageCoroutine("waitForSelector","div.quote:nth-child(11)"),# 10 per pagePageCoroutine("screenshot",options={"path":"quotes.png","fullPage":True}),],),)defparse(self,response):return{"quote_count":len(response.css("div.quote"))}AcknowledgementsThis project was inspired by:https://github.com/scrapy/scrapy/pull/1455https://github.com/michalmo/scrapy-browserhttps://github.com/lopuhin/scrapy-pyppeteerhttps://github.com/clemfromspace/scrapy-puppeteer