Package: playwright-request
Playwright Request for PythonThis is a library aiming to help programmers to create requests by using playwright browser
and bypass sites likewww.amazon.com,www.airbnb.comorwww.tripadvisor.comin general, all sites that block regular requests or require a proxy to crawl pages in parallel.WithPlaywrightRequestyou can process many urls asynchronously (at high speed) and parse the htmls
or create a function to process every open page that requires extra work, useful to get information
hidden until a user interacts with the page, for example when you need scrape images but the site requires
you to click on a button to open a popup window and then get the images-src and then close the popup window.This library contains:
the code to perform requests with the ability to extend and manipulate pages.Installationpipinstallplaywright-request
playwrightinstall
playwrightinstall-depsInstallation on docker imagesIt's probable to have caveats when working with docker images, to avoid this issues,
you must include the following code in yourDockerfileRUNapt-getupdate&&\apt-getinstall-y\apt-transport-https\ca-certificates\curl\gnupg\build-essential\python3-dev\python3-setuptools\gcc\make\apt-utils\libxcb-shm0\libx11-xcb-dev\libxext-dev\libxrandr-dev\libxcomposite-dev\libxcursor-dev\libxdamage-dev\libxi-dev\libxtst-dev\libgtk-3-dev\libasound-dev\libdbus-glib-1-devpipinstallplaywright
playwrightinstall
playwrightinstall-depsUse in cloud environments likeGCPorAWSIn order to usePlaywrightRequestin AWS or GCP it's necessary to create a docker image with your code
(include the code shown above).
Running your docker image locally is straightforward but in the cloud there is a little issue because
the commandplaywright install-depsinstallplaywrightwith the userroot.
In the cloud, the docker image is execute by a random user and that user is not able to findplaywrightbrowsers...
the solution is to include the following command in the code you usePlaywrightRequestand install
the browsers you need locallyimportplaywrightimportos# use only one of the following commands depending on your needsos.system("playwright install")# use this to install all browsersos.system("playwright install firefox")# to install firefoxos.system("playwright install chromium")# to install chromiumos.system("playwright install webkit")# to install webkitUsageExample #1: simple usagefromplaywright_request.playwright_requestimportPlaywrightRequest#crawlrequester=PlaywrightRequest()responses=requester.get(urls=["SITE1","SITE2"])print(responses[0].status_code,responses[0].html)print(responses[1].status_code,responses[1].html)Example #2: simple usage with Chromiumfromplaywright_request.playwright_requestimportPlaywrightRequestfromplaywright_request.browser_typeimportBrowserType#crawlrequester=PlaywrightRequest(browser=BrowserType.CHROMIUM,headless=False)responses=requester.get(urls=["SITE1"])print(responses[0].status_code,responses[0].html)Example #3: define interceptor to avoid loading imagesfromplaywright_request.playwright_requestimportPlaywrightRequestfromplaywright_request.route_interceptorimportRouteInterceptor#crawlinterceptor=RouteInterceptor().set_default_exclusions()requester=PlaywrightRequest(route_interceptor=interceptor)responses=requester.get(urls=["SITE1"])print(responses[0].status_code,responses[0].html)Example #4: extra processing resultsfromplaywright.async_apiimportPagefromplaywright_request.playwright_requestimportPlaywrightRequestfromplaywright_request.route_interceptorimportRouteInterceptorasyncdefget_all_photos(page:Page)->list[str]:# 1. Click on show all button and popup photo windowpage.locator('button',has_text='Show all photos').click()# 2. Wait for state is loaded and then wait for the selectorpage.wait_for_load_state('networkidle',timeout=3000)page.wait_for_load_state(timeout=3000)page.wait_for_selector('div[data-testid=photo-viewer-section]',timeout=3000)# 3. get photo section selectorphotos_section=page.query_selector('div[data-testid=photo-viewer-section]')# 4. get all picture elements withinall_pictures=photos_section.query_selector_all('picture')# 5. get all selector images and extract the attribute we needimages=[a.query_selector("img").get_attribute("data-original-uri")forainall_pictures]# 6. close the popup window and return the imagespage.locator('//button[@aria-label="Close"]').click()returnimagesrequester=PlaywrightRequest(extra_async_function_ptr=get_all_photos)responses=requester.get(urls=[f"SITE-{k}"forkinrange(100)])images=[response.extra_resultforresponseinresponses]forresponseinresponses:images=response.extra_resultprint(response.status_code,len(images))Example 5: detect amazon error pagesfromplaywright_request.commom_error_page_detectors.amazon_error_page_detectorimportAmazonErrorPageDetectorfromplaywright_request.playwright_requestimportPlaywrightRequestamazon_detector=AmazonErrorPageDetector()requester=PlaywrightRequest(error_page_detectors=[amazon_detector])responses=requester.get(urls=[f"AMAZON-ASIN-{k}"forkinrange(100)])valid_htmls=[response.htmlforresponseinresponsesifresponse.status_code==200andnotresponse.error_list]AuthorPedro Mayorga.