Package: scrapy-headless-selenium
Scrapy with Headless SeleniumScrapy middleware to handle dynamic web pages, using Selenium and running in headless mode by default:Running in headless mode by defaultRunning by default with ad blocking browser plugin, for faster scraping (only for FireFox, seethis issue)Dynamic responses, to allow interaction with the web page being scrapedInstallation$ pip install scrapy-headless-seleniumYou should usepython>=3.6.
You will also need one of the Selenium compatible browsers and drivers (FireFox & geckodriver or Chrome & chromium-driver).ConfigurationAdd the browser to use, the path to the driver executable, and the arguments to pass to the executable to the scrapy settings:fromshutilimportwhichSELENIUM_DRIVER_NAME='firefox'SELENIUM_DRIVER_EXECUTABLE_PATH=which('geckodriver')SELENIUM_DRIVER_ARGUMENTS=['-headless']# '--headless' if using chrome instead of firefoxOptionally, set the path to the browser executable:python SELENIUM_BROWSER_EXECUTABLE_PATH = which('firefox')Add theSeleniumMiddlewareto the downloader middlewares and to the spider middlewares:DOWNLOADER_MIDDLEWARES={'scrapy_headless.SeleniumMiddleware':800}SPIDER_MIDDLEWARES={'scrapy_headless.SeleniumMiddleware':800}UsageUse thescrapy_headless.SeleniumRequestinstead of the scrapy built-inRequestlike below:fromscrapy_headlessimportSeleniumRequestyieldSeleniumRequest(url,self.parse_result)The request will be handled by selenium, and the request will have an additionalmetakey, nameddrivercontaining the selenium driver with the request processed.defparse_result(self,response):print(response.request.meta['driver'].title)For more information about the available driver methods and attributes, refer to theselenium python documentationTheselectorresponse attribute work as usual (but contains the html processed by the selenium driver).defparse_result(self,response):print(response.selector.xpath('//title/@text'))The Selenium WebDriver is also exposed through theresponse.interactproperty, to allow interaction with the browser.
The response also implements aclickmethod which excepts a CSS / XPATH selector, to click on an element and return a new response with the new body:defparse_result(self,response):response=response.click('#id')# equivalent to response.click('//[@id="id"]')print(response.selector.xpath('//title/@text'))# searches the reloaded response bodyAdditional argumentsThescrapy_headless.SeleniumRequestaccept 4 additional arguments:wait_time/wait_untilWhen used, selenium will perform anExplicit waitbefore returning the response to the spider.fromselenium.webdriver.common.byimportByfromselenium.webdriver.supportimportexpected_conditionsasECyieldSeleniumRequest(url=url,callback=self.parse_result,wait_time=10,wait_until=EC.element_to_be_clickable((By.ID,'someid')))screenshotWhen used, selenium will take a screenshot of the page and the binary data of the .png captured will be added to the responsemeta:yieldSeleniumRequest(url=url,callback=self.parse_result,screenshot=True)defparse_result(self,response):withopen('image.png','wb')asimage_file:image_file.write(response.meta['screenshot'])scriptWhen used, selenium will execute custom JavaScript code.yieldSeleniumRequest(url,self.parse_result,script='window.scrollTo(0, document.body.scrollHeight);',)ThanksSpecial thanks to @clemfromspace which wrotescrapy-selenium, which is the original fork for this project.