Package: selenium-scrapy
Scrapy with seleniumScrapy middleware to handle javascript pages using selenium==4.3.0.This was originally a fork ofscrapy-seleniumbut I couldn't use the below command on it.Therefore this is actually a rip-off of said repo.Installation$ pip install git+https://github.com/mrafee113/selenium_scrapy.git@v0.1.0You should usepython>=3.9.
You will also need one of the Seleniumcompatible browsers.ConfigurationAdd the browser to use, the path to the driver executable, and the arguments to pass to the executable to the scrapy settings:fromshutilimportwhichSELENIUM_DRIVER_NAME='firefox'SELENIUM_DRIVER_EXECUTABLE_PATH=which('geckodriver')SELENIUM_DRIVER_ARGUMENTS=['-headless']# '--headless' if using chrome instead of firefoxSELENIUM_DRIVER_PREFERENCES={# (for firefox) to allow browser data to be cached among sessions."browser.cahce.disk.parent_directory":'/tmp/firefox-data-directory',# template firefox preferences to customize downloading files."browser.download.folderList":2,"browser.download.manager.showWhenStarting":False,"browser.download.dir":'/tmp/firefox-download-directory',"browser.helperApps.neverAsk.saveToDisk":"application/x-gzip"}# WARNING: To support localstorage the github project pyselenium_localstorage was used.#  This project has its obvious limitations (read its code).#  So be careful for type conversion when setting and getting values.SELENIUM_DRIVER_LOCALSTORAGE_DATA=[("https://python.org",{"key1":"value1"}),("https://google.com",{"data":"information"})]# or alternativelySELENIUM_DRIVER_LOCALSTORAGE_DATA=("https://github.ccm",{"username":"code"})Optionally, set the path to the browser executable:python SELENIUM_BROWSER_EXECUTABLE_PATH = which('firefox')In order to use a remote Selenium driver, specifySELENIUM_COMMAND_EXECUTORinstead ofSELENIUM_DRIVER_EXECUTABLE_PATH:python SELENIUM_COMMAND_EXECUTOR = 'http://localhost:4444/wd/hub'Add theSeleniumMiddlewareto the downloader middlewares:DOWNLOADER_MIDDLEWARES={'scrapy_selenium.SeleniumMiddleware':800}UsageUse thescrapy_selenium.SeleniumRequestinstead of the scrapy built-inRequestlike below:fromscrapy_seleniumimportSeleniumRequestyieldSeleniumRequest(url=url,callback=self.parse_result)The request will be handled by selenium, and the request will have an additionalmetakey, nameddrivercontaining the selenium driver with the request processed.defparse_result(self,response):print(response.request.meta['driver'].title)For more information about the available driver methods and attributes, refer to theselenium python documentationTheselectorresponse attribute work as usual (but contains the html processed by the selenium driver).defparse_result(self,response):print(response.selector.xpath('//title/@text'))Additional argumentsThescrapy_selenium.SeleniumRequestaccept 4 additional arguments:wait_time/wait_untilWhen used, selenium will perform anExplicit waitbefore returning the response to the spider.fromselenium.webdriver.common.byimportByfromselenium.webdriver.supportimportexpected_conditionsasECyieldSeleniumRequest(url=url,callback=self.parse_result,wait_time=10,wait_until=EC.element_to_be_clickable((By.ID,'someid')))screenshotWhen used, selenium will take a screenshot of the page and the binary data of the .png captured will be added to the responsemeta:yieldSeleniumRequest(url=url,callback=self.parse_result,screenshot=True)defparse_result(self,response):withopen('image.png','wb')asimage_file:image_file.write(response.meta['screenshot'])scriptWhen used, selenium will execute custom JavaScript code.yieldSeleniumRequest(url=url,callback=self.parse_result,script='window.scrollTo(0, document.body.scrollHeight);',)