Question: I'm trying to scrape data from the following website: https://nfeweb.sefaz.go.gov.br/nfeweb/sites/nfe/consulta-completa STEP 1  STEP 2 When the Access Key is inserted I need to press the "Pesquisar" button: In this case I've used the following access key: 52241061585865236600650040001896941930530252  and it returns the following page: https://nfeweb.sefaz.go.gov.br/nfeweb/sites/nfce/render/xml-consulta-completa?g-recaptcha-response=03AFcWeA7_oqqL4KubId8rW_TapI_NSJDOGBzrx_JB2XAtJitNaBl23zLKbjbj45m9eUZam3xp6R57BI47AI0lp_K3KS-CbtpPiTNAHqcxLV-Gnp2Vf778i3NeLMCKNoHpk7IitkwPHvHJjkg1sWRqdTZrHkhVHiMwFbTC4qFw6436ddwu9rRERxOiY532lIoijoHzDga85l7RvbHkyGUdWD7QVlTUNUU-2ztx21cQ_pDDQrxreDFEL8eCR0ijYAMrOtKEXMwqGSuHFTOSkZ83DCJ4S610YWujUukTXbOSdaAuGpeHljf4CsswFLWTKN8UoKTjlEia_I0cO17zgSnY9Z9rQDEZR1Xeq00CDmpbB73m95EOo0prSrL2RcsRnWkPytDIwJUIfsEAcEQ77vuacbNflj_yFpj2GSWVnGQnKXUrY4DsyRhNU6T6usZaYH5kTRb85qvrfm2FqOlgBfLDcvuwB_Q2JqRxyF6-oJlw64Sx2MZzUQC2gZjPtAIRwGCqOS80OkDkTmHZl9x3fM6tOr4fYM6BouHWrnjfyNz99O9bFcQv_bbdyREr1MVgJ6fujSZM6C7WoRJjwTv29kIuGc2l4nMkkilUU6rzK-apAYtgzSim_5T6N_zkvVQfOAo0mlKwjfVLVxCaWQYsGe5MfBe65ZmLVP_lIHnsJe_z0G9CMclmpaKTiynNEMtu_n8d6utw5ot6BHGp9OALHQq2_62hE_TTYMqVlrzugaPxMrTMKnGWd4W_kVPh-VqgqsKxdDW8xFXYtE8OM_WZNRg4m0ESnl4xW5NLZeZGu7onPt3jkw3vCt57YmdAgcHPpIhg0zPA7lNdBrY1zCeCM3edWoatnFng6irasc5R8fheSL2IS0lSUqCfN_cIuC6rYlPUGlU7pREqYe5ZTxHNkyI6GBvWM_pZSO4glw&chaveAcesso=52241061585865236600650040001896941930530252   STEP 3 In this stage I need to click in the "Visualizar NFC-e Detalhada" to finally get to the page with the data I want to scrape. The new path becomes: https://nfeweb.sefaz.go.gov.br/nfeweb/sites/nfce/render/NFCe?chNFe=52241061585865236600650040001896941930530252   The final step is to click in "Produtos e Serviços" Which will bring me to the following screen where is the data I want to scrape:  Errors If i try to access the webpage directly from the link: https://nfeweb.sefaz.go.gov.br/nfeweb/sites/nfce/render/NFCe?chNFe=52241061585865236600650040001896941930530252 it returns session expired:  If I try to scrape via python, it blocks me and i can't make any more search, even if it's via web browser. I need help on trying to scrape the data in that specific page, following all the steps and bypassing the recaptcha and bot block from the web site security.

Answer: The website may be blocking you because it's trying to stop scraping. However, the following code works for me even after multiple attempts: from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By from selenium.webdriver import ChromeOptions from selenium.webdriver.common.action_chains import ActionChains from functools import cache options = ChromeOptions() options.add_argument("--headless=true") url = "https://nfeweb.sefaz.go.gov.br/nfeweb/sites/nfe/consulta-completa" access_key = "52241061585865236600650040001896941930530252" @cache def wait(driver): return WebDriverWait(driver, 10) def click(driver, selector): button = wait(driver).until(EC.presence_of_element_located(selector)) ActionChains(driver).click(button).perform() def send(driver, selector, data): wait(driver).until(EC.presence_of_element_located(selector)).send_keys(data) def text(e): return e.text if e.text else e.get_attribute("textContent") with webdriver.Chrome(options) as driver: driver.get(url) send(driver, (By.ID, "chaveAcesso"), access_key) click(driver, (By.ID, "btnPesquisar")) click(driver, (By.CSS_SELECTOR, "button.btn-view-det")) click(driver, (By.ID, "tab_3")) # Now scrape the page. For example, print the text for a selection of labels selector = By.CSS_SELECTOR, "tr.col-4 td label" for label in wait(driver).until(EC.presence_of_all_elements_located(selector)): print(text(label))  Output: Base de Cálculo ICMS Valor do ICMS Valor do ICMS Desonerado Valor Total do FCP Código do Produto Código NCM Código CEST