Package: scrapy-playwright
scrapy-playwright: Playwright integration for ScrapyAScrapyDownload Handler which performs requests usingPlaywright for Python.
It can be used to handle pages that require JavaScript (among other things),
while adhering to the regular Scrapy workflow (i.e. without interfering
with request scheduling, item processing, etc).RequirementsAfter the release ofversion 2.0,
which includescoroutine syntax supportandasyncio support, Scrapy allows
to integrateasyncio-based projects such asPlaywright.Minimum required versionsPython >= 3.8Scrapy >= 2.0 (!= 2.4.0)Playwright >= 1.15Installationscrapy-playwrightis available on PyPI and can be installed withpip:pip install scrapy-playwrightplaywrightis defined as a dependency so it gets installed automatically,
however it might be necessary to install the specific browser(s) that will be
used:playwright installIt's also possible to install only a subset of the available browsers:playwright install firefox chromiumChangelogSee thechangelogdocument.ActivationReplace the defaulthttpand/orhttpsDownload Handlers throughDOWNLOAD_HANDLERS:DOWNLOAD_HANDLERS={"http":"scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler","https":"scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",}Note that theScrapyPlaywrightDownloadHandlerclass inherits from the defaulthttp/httpshandler. Unless explicitly marked (seeBasic usage),
requests will be processed by the regular Scrapy download handler.Also, be sure toinstall theasyncio-based Twisted reactor:TWISTED_REACTOR="twisted.internet.asyncioreactor.AsyncioSelectorReactor"Basic usageSet theplaywrightRequest.metakey to download a request using Playwright:importscrapyclassAwesomeSpider(scrapy.Spider):name="awesome"defstart_requests(self):# GET requestyieldscrapy.Request("https://httpbin.org/get",meta={"playwright":True})# POST requestyieldscrapy.FormRequest(url="https://httpbin.org/post",formdata={"foo":"bar"},meta={"playwright":True},)defparse(self,response,**kwargs):# 'response' contains the page as seen by the browserreturn{"url":response.url}Notes about the User-Agent headerBy default, outgoing requests include theUser-Agentset by Scrapy (either with theUSER_AGENTorDEFAULT_REQUEST_HEADERSsettings or via theRequest.headersattribute).
This could cause some sites to react in unexpected ways, for instance if the user agent
does not match the running Browser. If you prefer theUser-Agentsent by
default by the specific browser you're using, set the Scrapy user agent toNone.SupportedsettingsPLAYWRIGHT_BROWSER_TYPETypestr, default"chromium".The browser type to be launched, e.g.chromium,firefox,webkit.PLAYWRIGHT_BROWSER_TYPE="firefox"PLAYWRIGHT_LAUNCH_OPTIONSTypedict, default{}A dictionary with options to be passed as keyword arguments when launching the
Browser. See the docs forBrowserType.launchfor a list of supported keyword arguments.PLAYWRIGHT_LAUNCH_OPTIONS={"headless":False,"timeout":20*1000,# 20 seconds}PLAYWRIGHT_CDP_URLTypeOptional[str], defaultNoneThe endpoint of a remote Chromium browser to connect using theChrome DevTools Protocol,
viaBrowserType.connect_over_cdp.
If this setting is used:all non-persistent contexts will be created on the connected remote browserthePLAYWRIGHT_LAUNCH_OPTIONSsetting is ignoredthePLAYWRIGHT_BROWSER_TYPEsetting must not be set to a value different than "chromium"PLAYWRIGHT_CDP_URL="http://localhost:9222"PLAYWRIGHT_CDP_KWARGSTypedict[str, Any], default{}Additional keyword arguments to be passed toBrowserType.connect_over_cdpwhen usingPLAYWRIGHT_CDP_URL. Theendpoint_urlkey is always ignored,PLAYWRIGHT_CDP_URLis used instead.PLAYWRIGHT_CDP_KWARGS={"slow_mo":1000,"timeout":10*1000}PLAYWRIGHT_CONTEXTSTypedict[str, dict], default{}A dictionary which defines Browser contexts to be created on startup.
It should be a mapping of (name, keyword arguments).PLAYWRIGHT_CONTEXTS={"foobar":{"context_arg1":"value","context_arg2":"value",},"default":{"context_arg1":"value","context_arg2":"value",},"persistent":{"user_data_dir":"/path/to/dir",# will be a persistent context"context_arg1":"value",},}See the section onbrowser contextsfor more information.
See also the docs forBrowser.new_context.PLAYWRIGHT_MAX_CONTEXTSTypeOptional[int], defaultNoneMaximum amount of allowed concurrent Playwright contexts. If unset orNone,
no limit is enforced. See theMaximum concurrent context countsection for more information.PLAYWRIGHT_MAX_CONTEXTS=8PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUTTypeOptional[float], defaultNoneTimeout to be used when requesting pages by Playwright, in milliseconds. IfNoneor unset, the default value will be used (30000 ms at the time of writing).
See the docs forBrowserContext.set_default_navigation_timeout.PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT=10*1000# 10 secondsPLAYWRIGHT_PROCESS_REQUEST_HEADERSTypeOptional[Union[Callable, str]], defaultscrapy_playwright.headers.use_scrapy_headersA function (or the path to a function) that processes headers for a given request
and returns a dictionary with the headers to be used (note that, depending on the browser,
additional default headers could be sent as well). Coroutine functions (async def) are
supported.This will be called at least once for each Scrapy request (receiving said request and the
corresponding Playwright request), but it could be called additional times if the given
resource generates more requests (e.g. to retrieve assets like images or scripts).The function must return adictobject, and receives the following positional arguments:-browser_type:str-playwright_request:playwright.async_api.Request-scrapy_headers:scrapy.http.headers.HeadersThe default function (scrapy_playwright.headers.use_scrapy_headers) tries to
emulate Scrapy's behaviour for navigation requests, i.e. overriding headers
with their values from the Scrapy request. For non-navigation requests (e.g.
images, stylesheets, scripts, etc), only theUser-Agentheader is overriden,
for consistency.SettingPLAYWRIGHT_PROCESS_REQUEST_HEADERS=Nonewill give complete control to
Playwright, i.e. headers from Scrapy requests will be ignored and only headers
set by Playwright will be sent. Keep in mind that in this case, headers passed
via theRequest.headersattribute or set by Scrapy components are ignored
(including cookies set via theRequest.cookiesattribute).defcustom_headers(browser_type:str,playwright_request:playwright.async_api.Request,scrapy_headers:scrapy.http.headers.Headers,)->dict:ifbrowser_type=="firefox":return{"User-Agent":"foo"}return{"User-Agent":"bar"}PLAYWRIGHT_PROCESS_REQUEST_HEADERS=custom_headersPLAYWRIGHT_MAX_PAGES_PER_CONTEXTTypeint, defaults to the value of Scrapy'sCONCURRENT_REQUESTSsettingMaximum amount of allowed concurrent Playwright pages for each context.
See thenotes about leaving unclosed pages.PLAYWRIGHT_MAX_PAGES_PER_CONTEXT=4PLAYWRIGHT_ABORT_REQUESTTypeOptional[Union[Callable, str]], defaultNoneA predicate function (or the path to a function) that receives aplaywright.async_api.Requestobject and must returnTrueif the request should be aborted,Falseotherwise.
Coroutine functions (async def) are supported.Note that all requests will appear in the DEBUG level logs, however there will
be no corresponding response log lines for aborted requests. Aborted requests
are counted in theplaywright/request_count/abortedjob stats item.defshould_abort_request(request):return(request.resource_type=="image"or".jpg"inrequest.url)PLAYWRIGHT_ABORT_REQUEST=should_abort_requestGeneral note about settingsFor settings that accept object paths as strings, passing callable objects is
only supported when using Scrapy>=2.4. With prior versions, only strings are
supported.SupportedRequest.metakeysplaywrightTypebool, defaultFalseIf set to a value that evaluates toTruethe request will be processed by Playwright.returnscrapy.Request("https://example.org",meta={"playwright":True})playwright_contextTypestr, default"default"Name of the context to be used to download the request.
See the section onbrowser contextsfor more information.returnscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_context":"awesome_context",},)playwright_context_kwargsTypedict, default{}A dictionary with keyword arguments to be used when creating a new context, if a context
with the name specified in theplaywright_contextmeta key does not exist already.
See the section onbrowser contextsfor more information.returnscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_context":"awesome_context","playwright_context_kwargs":{"ignore_https_errors":True,},},)playwright_include_pageTypebool, defaultFalseIfTrue, thePlaywright pagethat was used to download the request will be available in the callback atresponse.meta['playwright_page']. IfFalse(or unset) the page will be
closed immediately after processing the request.Important!This meta key is entirely optional, it's NOT necessary for the page to load or for any
asynchronous operation to be performed (specifically, it's NOT necessary forPageMethodobjects to be applied). Use it only if you need access to the Page object in the callback
that handles the response.For more information and important notes seeReceiving Page objects in callbacks.returnscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_include_page":True},)playwright_page_event_handlersTypeDict[Str, Callable], default{}A dictionary of handlers to be attached to page events.
SeeHandling page events.playwright_page_init_callbackTypeOptional[Union[Callable, str]], defaultNoneA coroutine function (async def) to be invoked for newly created pages.
Called after attaching page event handlers & setting up internal route
handling, before making any request. It receives the Playwright page and the
Scrapy request as positional arguments. Useful for initialization code.
Ignored if the page for the request already exists (e.g. by passingplaywright_page).asyncdefinit_page(page,request):awaitpage.add_init_script(path="./custom_script.js")classAwesomeSpider(scrapy.Spider):defstart_requests(self):yieldscrapy.Request(url="https://httpbin.org/headers",meta={"playwright":True,"playwright_page_init_callback":init_page,},)Important!scrapy-playwrightusesPage.route&Page.unrouteinternally, avoid using
these methods unless you know exactly what you're doing.playwright_page_methodsTypeIterable[PageMethod], default()An iterable ofscrapy_playwright.page.PageMethodobjects to indicate actions to be performed on the page before returning the
final response. SeeExecuting actions on pages.playwright_pageTypeOptional[playwright.async_api.Page], defaultNoneAPlaywright pageto be used to
download the request. If unspecified, a new page is created for each request.
This key could be used in conjunction withplaywright_include_pageto make a chain of
requests using the same page. For instance:defstart_requests(self):yieldscrapy.Request(url="https://httpbin.org/get",meta={"playwright":True,"playwright_include_page":True},)defparse(self,response,**kwargs):page=response.meta["playwright_page"]yieldscrapy.Request(url="https://httpbin.org/headers",callback=self.parse_headers,meta={"playwright":True,"playwright_page":page},)playwright_page_goto_kwargsTypedict, default{}A dictionary with keyword arguments to be passed to the page'sgotomethodwhen navigating to an URL. Theurlkey is ignored if present, the request
URL is used instead.returnscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_page_goto_kwargs":{"wait_until":"networkidle",},},)playwright_security_detailsTypeOptional[dict], read onlyA dictionary withsecurity informationabout the give response. Only available for HTTPS requests. Could be accessed
in the callback viaresponse.meta['playwright_security_details']defparse(self,response,**kwargs):print(response.meta["playwright_security_details"])# {'issuer': 'DigiCert TLS RSA SHA256 2020 CA1', 'protocol': 'TLS 1.3', 'subjectName': 'www.example.org', 'validFrom': 1647216000, 'validTo': 1678838399}Receiving Page objects in callbacksSpecifying a value that evaluates toTruein theplaywright_include_pagemeta key for a
request will result in the correspondingplaywright.async_api.Pageobject
being available in theplaywright_pagemeta key in the request callback.
In order to be able toawaitcoroutines on the providedPageobject,
the callback needs to be defined as a coroutine function (async def).CautionUse this carefully, and only if you really need to do things with the Page
object in the callback. If pages are not properly closed after they are no longer
necessary the spider job could get stuck because of the limit set by thePLAYWRIGHT_MAX_PAGES_PER_CONTEXTsetting.importscrapyclassAwesomeSpiderWithPage(scrapy.Spider):name="page_spider"defstart_requests(self):yieldscrapy.Request(url="https://example.org",callback=self.parse_first,meta={"playwright":True,"playwright_include_page":True},errback=self.errback_close_page,)defparse_first(self,response):page=response.meta["playwright_page"]returnscrapy.Request(url="https://example.com",callback=self.parse_second,meta={"playwright":True,"playwright_include_page":True,"playwright_page":page},errback=self.errback_close_page,)asyncdefparse_second(self,response):page=response.meta["playwright_page"]title=awaitpage.title()# "Example Domain"awaitpage.close()return{"title":title}asyncdeferrback_close_page(self,failure):page=failure.request.meta["playwright_page"]awaitpage.close()Notes:When passingplaywright_include_page=True, make sure pages are always closed
when they are no longer used. It's recommended to set a Request errback to make
sure pages are closed even if a request fails (ifplaywright_include_page=Falsepages are automatically closed upon encountering an exception).
This is important, as open pages count towards the limit set byPLAYWRIGHT_MAX_PAGES_PER_CONTEXTand crawls could freeze if the limit is reached
and pages remain open indefinitely.Defining callbacks asasync defis only necessary if you need toawaitthings,
it's NOT necessary if you just need to pass over the Page object from one callback
to another (see the example above).Any network operations resulting from awaiting a coroutine on a Page object
(goto,go_back, etc) will be executed directly by Playwright, bypassing the
Scrapy request workflow (Scheduler, Middlewares, etc).Browser contextsMultiplebrowser contextsto be launched at startup can be defined via thePLAYWRIGHT_CONTEXTSsetting.Choosing a specific context for a requestPass the name of the desired context in theplaywright_contextmeta key:yieldscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_context":"first"},)Default contextIf a request does not explicitly indicate a context via theplaywright_contextmeta key, it falls back to using a general context calleddefault. Thisdefaultcontext can also be customized on startup via thePLAYWRIGHT_CONTEXTSsetting.Persistent contextsPass a value for theuser_data_dirkeyword argument to launch a context as
persistent. See alsoBrowserType.launch_persistent_context.Note that persistent contexts are launched independently from the main browser
instance, hence keyword arguments passed in thePLAYWRIGHT_LAUNCH_OPTIONSsetting do not apply.Creating contexts while crawlingIf the context specified in theplaywright_contextmeta key does not exist, it will be created.
You can specify keyword arguments to be passed toBrowser.new_contextin theplaywright_context_kwargsmeta key:yieldscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_context":"new","playwright_context_kwargs":{"java_script_enabled":False,"ignore_https_errors":True,"proxy":{"server":"http://myproxy.com:3128","username":"user","password":"pass",},},},)Please note that if a context with the specified name already exists,
that context is used andplaywright_context_kwargsare ignored.Closing contexts while crawlingAfterreceiving the Page object in your callback,
you can access a context though the correspondingPage.contextattribute, and awaitcloseon it.defparse(self,response,**kwargs):yieldscrapy.Request(url="https://example.org",callback=self.parse_in_new_context,errback=self.close_context_on_error,meta={"playwright":True,"playwright_context":"awesome_context","playwright_include_page":True,},)asyncdefparse_in_new_context(self,response):page=response.meta["playwright_page"]title=awaitpage.title()awaitpage.close()awaitpage.context.close()return{"title":title}asyncdefclose_context_on_error(self,failure):page=failure.request.meta["playwright_page"]awaitpage.close()awaitpage.context.close()Avoid race conditions & memory leaks when closing contextsMake sure to close the page before closing the context. Seethis commentin#191for more information.Maximum concurrent context countSpecify a value for thePLAYWRIGHT_MAX_CONTEXTSsetting to limit the amount
of concurent contexts. Use with caution: it's possible to block the whole crawl
if contexts are not closed after they are no longer used (refer tothis sectionto dinamically close contexts).
Make sure to define an errback to still close contexts even if there are errors.Proxy supportProxies are supported at the Browser level by specifying theproxykey in
thePLAYWRIGHT_LAUNCH_OPTIONSsetting:fromscrapyimportSpider,RequestclassProxySpider(Spider):name="proxy"custom_settings={"PLAYWRIGHT_LAUNCH_OPTIONS":{"proxy":{"server":"http://myproxy.com:3128","username":"user","password":"pass",},}}defstart_requests(self):yieldRequest("http://httpbin.org/get",meta={"playwright":True})defparse(self,response,**kwargs):print(response.text)Proxies can also be set at the context level with thePLAYWRIGHT_CONTEXTSsetting:PLAYWRIGHT_CONTEXTS={"default":{"proxy":{"server":"http://default-proxy.com:3128","username":"user1","password":"pass1",},},"alternative":{"proxy":{"server":"http://alternative-proxy.com:3128","username":"user2","password":"pass2",},},}Or passing aproxykey whencreating contexts while crawling.See also:zyte-smartproxy-playwright:
seamless support forZyte Smart Proxy Managerin the Node.js version of Playwright.theupstream Playwright for Python sectionon HTTP Proxies.Executing actions on pagesA sorted iterable (e.g.list,tuple,dict) ofPageMethodobjects
could be passed in theplaywright_page_methodsRequest.metakey to request methods to be invoked on thePageobject before returning the finalResponseto the callback.This is useful when you need to perform certain actions on a page (like scrolling
down or clicking links) and you want to handle only the final result in your callback.PageMethodclassscrapy_playwright.page.PageMethod(method: str, *args, **kwargs):Represents a method to be called (and awaited if necessary) on aplaywright.page.Pageobject (e.g. "click", "screenshot", "evaluate", etc).methodis the name of the method,*argsand**kwargsare passed when calling such method. The return value
will be stored in thePageMethod.resultattribute.For instance:defstart_requests(self):yieldRequest(url="https://example.org",meta={"playwright":True,"playwright_page_methods":[PageMethod("screenshot",path="example.png",full_page=True),],},)defparse(self,response,**kwargs):screenshot=response.meta["playwright_page_methods"][0]# screenshot.result contains the image's bytesproduces the same effect as:defstart_requests(self):yieldRequest(url="https://example.org",meta={"playwright":True,"playwright_include_page":True},)asyncdefparse(self,response,**kwargs):page=response.meta["playwright_page"]screenshot=awaitpage.screenshot(path="example.png",full_page=True)# screenshot contains the image's bytesawaitpage.close()Supported methodsRefer to theupstream docs for thePageclassto see available methods.Impact on Response objectsCertainResponseattributes (e.g.url,ip_address) reflect the state after the last
action performed on a page. If you issue aPageMethodwith an action that results in
a navigation (e.g. aclickon a link), theResponse.urlattribute will point to the
new URL, which might be different from the request's URL.Handling page eventsA dictionary of Page event handlers can be specified in theplaywright_page_event_handlersRequest.metakey.
Keys are the name of the event to be handled (e.g.dialog,download, etc).
Values can be either callables or strings (in which case a spider method with the name will be looked up).Example:fromplaywright.async_apiimportDialogasyncdefhandle_dialog(dialog:Dialog)->None:logging.info(f"Handled dialog with message:{dialog.message}")awaitdialog.dismiss()classEventSpider(scrapy.Spider):name="event"defstart_requests(self):yieldscrapy.Request(url="https://example.org",meta={"playwright":True,"playwright_page_event_handlers":{"dialog":handle_dialog,"response":"handle_response",},},)asyncdefhandle_response(self,response:PlaywrightResponse)->None:logging.info(f"Received response with URL{response.url}")See theupstreamPagedocsfor a list of the accepted events and the arguments passed to their handlers.Notes about page event handlersEvent handlers will remain attached to the page and will be called for
subsequent downloads using the same page unless they areremoved later.
This is usually not a problem, since by default requests are performed in
single-use pages.Event handlers will process Playwright objects, not Scrapy ones. For example,
for each Scrapy request/response there will be a matching Playwright
request/response, but not the other way: background requests/responses to get
images, scripts, stylesheets, etc are not seen by Scrapy.Memory usage extensionThe default Scrapy memory usage extension
(scrapy.extensions.memusage.MemoryUsage) does not include the memory used by
Playwright because the browser is launched as a separate process. The
scrapy-playwright package provides a replacement extension which also considers
the memory used by Playwright. This extension needs thepsutilpackage to work.Update theEXTENSIONSsetting to disable the built-in Scrapy extension and replace it with the one
from the scrapy-playwright package:# settings.pyEXTENSIONS={"scrapy.extensions.memusage.MemoryUsage":None,"scrapy_playwright.memusage.ScrapyPlaywrightMemoryUsageExtension":0,}Refer to theupstream docsfor more information about supported settings.ExamplesClick on a link, save the resulting page as PDFclassClickAndSavePdfSpider(scrapy.Spider):name="pdf"defstart_requests(self):yieldscrapy.Request(url="https://example.org",meta=dict(playwright=True,playwright_page_methods={"click":PageMethod("click",selector="a"),"pdf":PageMethod("pdf",path="/tmp/file.pdf"),},),)defparse(self,response,**kwargs):pdf_bytes=response.meta["playwright_page_methods"]["pdf"].resultwithopen("iana.pdf","wb")asfp:fp.write(pdf_bytes)yield{"url":response.url}# response.url is "https://www.iana.org/domains/reserved"Scroll down on an infinite scroll page, take a screenshot of the full pageclassScrollSpider(scrapy.Spider):name="scroll"defstart_requests(self):yieldscrapy.Request(url="http://quotes.toscrape.com/scroll",meta=dict(playwright=True,playwright_include_page=True,playwright_page_methods=[PageMethod("wait_for_selector","div.quote"),PageMethod("evaluate","window.scrollBy(0, document.body.scrollHeight)"),PageMethod("wait_for_selector","div.quote:nth-child(11)"),# 10 per page],),)asyncdefparse(self,response,**kwargs):page=response.meta["playwright_page"]awaitpage.screenshot(path="quotes.png",full_page=True)awaitpage.close()return{"quote_count":len(response.css("div.quote"))}# quotes from several pagesSee theexamplesdirectory for more.Known issuesLack of native support for WindowsThis package does not work natively on Windows. This is because:Playwright runs the driver in a subprocess. Source:Playwright repository."On Windows, the default event loopProactorEventLoopsupports subprocesses,
whereasSelectorEventLoopdoes not". Source:Python docs.Twisted'sasyncioreactor requires theSelectorEventLoop. Source:Twisted repository.Some users have reported having successrunning under WSL.
See also#78for information about working in headful mode under WSL.No per-request proxy supportSpecifying a proxy via theproxyRequest meta key is not supported.
Refer to theProxy supportsection for more information.Unsopported signalsTheheaders_receivedandbytes_receivedsignals are not fired by the
scrapy-playwright download handler.Reporting issuesBefore opening an issue please make sure the unexpected behavior can only be
observed by using this package and not with standalone Playwright. To do this,
translate your spider code to a reasonably close Playwright script: if the
issue also occurs this way, you should instead report itupstream.
For instance:importscrapyclassExampleSpider(scrapy.Spider):name="example"defstart_requests(self):yieldscrapy.Request(url="https://example.org",meta=dict(playwright=True,playwright_page_methods=[PageMethod("screenshot",path="example.png",full_page=True),],),)translates roughly to:importasynciofromplaywright.async_apiimportasync_playwrightasyncdefmain():asyncwithasync_playwright()aspw:browser=awaitpw.chromium.launch()page=awaitbrowser.new_page()awaitpage.goto("https://example.org")awaitpage.screenshot(path="example.png",full_page=True)awaitbrowser.close()asyncio.run(main())Frequently Asked QuestionsSee theFAQdocument.Deprecation policyDeprecated features will be supported for at least six months
following the release that deprecated them. After that, they
may be removed at any time. See thechangelogfor more information about deprecations and removals.