Package: get-html
get-html: get raw or rendered HTML (for humans)Read all the details on how I started implementing this at ⇝Rendering-HTML_a-journey⇜This module is made for anyone needing to scrape HTML (i.e. scrape the web).It knows how to do only one thing, but does it well: getting HTML from a web page. The kind of HTML is up to you. Either:theraw HTML, directly from the server or,therendered HTML, after JS/AJAX calls.Moreover, it is made such that you can use a unique method throughout your project,
and switch between the two behaviors at launch by setting an environment variable.HtmlRenderer: a class to seamlessly render a pageHtmlRendererhandles all the specificpyppeteerstuff for you. It is also thread-safe."sync" usageHere is atypical usage:fromget_htmlimportHtmlRendererrenderer=HtmlRenderer()try:# use the renderer. The underlying browser will be instantiated on first call to render.response=renderer.render(url='https://xkcd.com')html=response.text# or resposne.content to get the raw bytes# ... etc.finally:# close the underlying browserrenderer.close()Or simply use acontext manager:fromget_htmlimportcreate_rendererwithcreate_renderer()asrenderer:# use the renderer. The underlying browser will be instanciated on first call to renderresponse=renderer.render(url='https://xkcd.com')html=response.text# or resposne.content to get the raw bytes# here, the underlying browser will be closedIf you need tomanipulate the pagebefore getting the content, pass anasyncfunction torender.
It will be called after the page loaded, but before the HTML content is fetched.
For example:fromget_htmlimportcreate_rendererasyncdefscroll_to_end(page):# https://github.com/miyakogi/pyppeteer/issues/205#issuecomment-470886682awaitpage.evaluate('{window.scrollBy(0, document.body.scrollHeight);}')withcreate_renderer()asrenderer:response=renderer.render('https://9gag.com',manipulate_page_func=scroll_to_end)"async" usageAll public methods have anasynccounterpart. When usingasync, however, you need to ensure thatthe browser is created once,the browser is closed once.By default, the browser is launched upon first use, usually on the first call ofrender.
When usingasync, this may be a problem, as multiple coroutine will try to create the browser multiple times.
To avoid this, ensure you trigger the browser creationbeforelaunching the other tasks (same for closing, wait for all tasks to complete).A concrete example is available inexamples/async_example.py. Here is the gist:importasynciofromget_htmlimportHtmlRendererloop=asyncio.get_event_loop()renderer=HtmlRenderer()# trigger the browser creation only once, before the tasksloop.run_until_complete(renderer.async_browser)# .. TASKS WITH RENDERING CALLS ...#    e.g. loop.run_until_complete(someRenderingTask())# finally close the browser once the tasks completedloop.run_until_complete(renderer.async_close())do_get: seemlessly switch between behaviorsfromget_html.env_defined_getimportdo_getresponse=do_get('https://xkcd.com')assertresponse.status_code==200html_bytes=response.contenthtml_string=response.textThe actual behavior ofdo_getwill depend on the environment variableRENDER_HTML:RENDER_HTML=[1|y|true|on]:do_getwill launch a chromium instance under the hood and render the page (rendered HTML)RENDER_HTML=<anything BUT 2>(default):do_getwill forward the call torequests.get(raw HTML).
DoNOTuse2before reading through the multi-threading section.If rendering support is on, a browser instance will be launchedon module load, and will be kept alive throughout the life of the application.
Keep that in mind if you have low-memory (chromium !!).Multi-threadingHtmlRendereris thread-safe.Fordo_getwith rendering support, there are two possibilities.Createonly one browser, shared by all threads. In this case, only one thread can executerenderat a time (locking mechanism);Createone browser per thread. In this case, threads can render in parallel. But be careful, each time a new thread callsdo_get,a new browser is launched, that will keep running until the end of the program (or until you callget_html.env_defined_get.close()).Enable mode (2) by settingRENDER_HTML=2. But again, ensure you don't have too many threads, since chromium needs a lot of memory.Running testsOn Windows/Linux:pipinstalltox
toxOn Mac (seehttps://github.com/tox-dev/tox/issues/1485):pipinstall"tox<3.7"toxCommon errorslibXX not found(Linux)Seeissue #290 of puppeteersudoapt-getinstallgconf-servicelibasound2libatk1.0-0libatk-bridge2.0-0libc6libcairo2libcups2libdbus-1-3libexpat1libfontconfig1libgcc1libgconf-2-4libgdk-pixbuf2.0-0libglib2.0-0libgtk-3-0libnspr4libpango-1.0-0libpangocairo-1.0-0libstdc++6libx11-6libx11-xcb1libxcb1libxcomposite1libxcursor1libxdamage1libxext6libxfixes3libxi6libxrandr2libxrender1libxss1libxtst6ca-certificatesfonts-liberationlibappindicator1libnss3lsb-releasexdg-utilswget