Package: selenium2df-locate-element
Like WebDriverWait - visibility_of_element_located, but much easierpip install selenium2df-locate-elementTested against Windows 10 / Python 3.10 / Anaconda 3This module simplifies the process of locating and extracting elements from web pages using Selenium,
while leveraging the power and flexibility of Pandas DataFrames.
It saves time and effort by providing a streamlined approach to web data
extraction and facilitates further data processing and analysis tasks.Advantages of using the locate_element functionSimplified element location:The function abstracts away the complexities of locating elements using CSS selectors in Selenium. You can specify a CSS selector query to target the desired elements without dealing with low-level Selenium operations.Conversion to Pandas DataFrame:The located elements are converted into a Pandas DataFrame, providing a convenient data structure for data manipulation and analysis.
This allows you to leverage Pandas' extensive functionality for working with tabular data.Support for conditional modification:The function accepts a checkdf parameter, which allows you to provide a callable function for modifying the DataFrame
based on specific conditions. This can be useful when you want
to filter or transform the located elements based on custom criteria.Timeout handling:The function includes a timeout mechanism to wait for the desired
elements to appear on the page. You can specify the maximum time to
wait, and the function will
continue to retry until the timeout is reached or the elements are located.
This helps handle scenarios where elements may take some
time to load or appear dynamically on the page.LocateselementsonawebpageusingSeleniumandconvertsthemintoaPandasDataFrame.Args:checkdf:AcallablefunctionthattakesaPandasDataFrameasinputandreturnsamodifiedDataFramebasedoncertainconditions.DefaultstoNone.query:AstringrepresentingtheCSSselectorqueryforthedesiredelements.Defaultsto"*".timeout:Themaximumtimeinsecondstowaitforthedesiredelements.Defaultsto30.withmethods:AbooleanindicatingwhethertoincludemethodsforinteractingwiththeelementsintheresultingDataFrame.DefaultstoTrue.Returns:IfcheckdfisNone,returnsasinglePandasDataFramecontainingthelocatedelements.Ifcheckdfisprovided,returnsatupleoftwoPandasDataFrames:thefirstonecontainsalllocatedelements,andthesecondonecontainsthemodifiedDataFramereturnedbycheckdf.importundetected_chromedriverasucfromselenium2df_locate_elementimportselenium2dfwait,locate_elementif__name__=="__main__":driver=uc.Chrome()selenium2dfwait.driver=driverdriver.get(r"https://testpages.herokuapp.com/styled/index.html")df3=locate_element()df_all,df=locate_element(checkdf=lambdadf:df.loc[df.aa_localName=='a'],query='*',timeout=30,withmethods=True)print(df_all,df)print(df3)element...aa_window_switch0<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()1<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()2<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()3<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()4<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()...........181<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()182<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()183<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()184<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()185<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()[186rowsx136columns]print(df)element...aa_window_switch15<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()17<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()19<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()23<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()25<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()...........171<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()173<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()179<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()180<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()185<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()[66rowsx136columns]print(df_all)element...aa_window_switch0<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()1<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()2<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()3<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()4<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()...........181<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()182<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()183<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()184<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()185<undetected_chromedriver.webelement.WebElement......0A3ADC1FAEED9B036EB136728D29F595()[186rowsx136columns]