Package: br-helper
Automate browser the easy way.Python library to automate webbrowser.Most functionality is tested on Chrome and Firefox, but may work on others as well.This library made my working process of browser automation much easier,
hope it will help your team too.Here you can see short information about the module, but to see what is available,make sure to check documentation of available functions.Quick examples of usage1. Initialize the browser# at first, initialize BrowserHelper objectfrombr_helper.br_helperimportBrowserHelperbr=BrowserHelper(browser="chrome",# you can also use firefox heredriver_path='/home/tornike/Desktop/bin/chromedriver')# useful notes:# it you want to use same browser and driver_path most of the time,# after the installation you can run second line without# driver_path argument and follow the instructions in terminal/interpreter.# In this case, module will try to scan your system and find# any file with "chromedriver" or "geckodriver"(in case of firefox)# in name, then it lists all matches, and you can choose which one to use# by default if no driver_path argument is passed.# in previous step you can also pass full file path of driver# to use, if it is not listed,# ! make sure you are using appropriate version of# chromedriver/geckodriver for your chrome/firefox browser !# if you want to use proxy, with hidden browser(maybe to run on server without GUI),# and also hide images on pages, and use custom download location, you can supply options# argument when creating an object, with similar form:br=BrowserHelper("chrome",driver_path='your_driver_file_path',options={'proxy':'ip_address:port_number','visibility':False,'hide_images':True,'download_location':'your_folder_path'})All keys that you can use in previous stepKey name                 Value                   Value Type       
--------------------------------------------------------------
* visibility            - True/False                - boolean
* download_location     - /path/to/folder           - string
* window_size           - (width, height)           - tuple
* hide_images           - True/False                - boolean
* disable_javascript    - True/False                - boolean
* proxy                 - ip:port                   - string
* user_data_dir         - path/to/chrome/profile    - string
* disable_infobars      - True/False                - boolean2. Basic examples# get a webpagebr.get("example.com")# click on an element with partial text matchbr.find("More").click()# find is a powerfull method, check its documentation ( type: >>>help(br.find) )# to see how you can make case insensitive, exact or specific# tag-based element searches with just text# in some cases, it is  better to use the method click,# and pass matched element, which will try to click element using# javascript, and by default try to click its parent in case of error.# see click method for more information (type >>>help(br.click) )# go back in historybr._b()# and forwardbr._f()# scroll to the bottombr.bottom()# go backbr.top()# refreshbr.r()# log some informationbr.log_info("successfully refreshed the page")# after this step, check new log.txt file in your current working directory.# To use different log file, just pass log_file argument when initializing br object,# with preferred full path of log file to use later.# get new pagebr.get("uct.ge")# locating elements to interact with them# click on first matching element# by css selectorbr.css1("a#programming").click()# or use xpathbr.xpath1("//a[@id='programming']").click()# you will get an IndexError if element was not found,# so you can consider using css, or xpath, instead of css1 and xpath1 methods# and check if returned list is not empty to continue working.# locating elements to parse a webpage dataelements=br.bcss(".class_of_your_element")# here first 'b' stands for bs4 library, which is used with this method# we also have bcss1 to find first match.# wait until specific loader, or other element disappearsbr.wait_until_disappears("type_your_css_or_xpath_selector_of_loader_here")# take a screenshotbr.google("why to visit Georgia")br.screenshot(image_name='screenshot.png')# run javascript codereally=br.js('console.log("Python is cool"); return true')# login to a websitebr.login("facebook.com",("username","password"))br.login("twitter.com/login",("username","password"))br.login("linkedin.com",("username","password"))# if you want to check if login was successfull# in most basic cases you can rely on returned value of login method# otherwise, define your own logic, for exampleifbr.login("twitter.com/login",("your_username","your_password")):print("Hello, twitter")else:print("Unsuccessfull login")# if you are tired, trybr.mario()# we do not have reliable methods to play it yet...# or have some fun with this few lines of codebr.dino()# works for Google Chromeimporttime,randomwhileTrue:time.sleep(random.randint(1,5)/10)br.press("up")# ! this method is still in development !# Please do not try this!br.get("finder.ge")br._dance()3. Run multiple browser instances in parallel(almost)Module has helper class, for cases when we want to run more than one instance of browser at the same time. It allows you to automatically split given urls into number of instances that you want to get using browser, get them and call callback function after each request.
That specific class uses information about DRIVER_PATH, that was defined earlier, so make sure before using it to define path in modules file itself, or just initialize BrowserHelper class instance without giving driver_path and when it finds correct driver path, type the number of it in terminal to save it in file automatically(this method of getting information will probably change in next releases).Here is an examples of using it, with explanations:Example 1'''Code to download multiple files using given urls.Each given url should return file directly, when opened.Please take into account the fact that in this examplewe use very simple lambda expression as our callback argumentto just print urls, not waiting until downloads finish,so in real cases, you may want to use a bit more complex function,which also waits before downloads are completed.'''frombr_helper.br_helperimportMultiBrimportos# url pattern to use (I found that site recently to test the functionality)urls=['http://ipv4.download.thinkbroadband.com/1MB.zip'foriinrange(100)]# initializembr=MultiBr()# start processesmbr.get_with_multi(# use threadmulti_type="thread",# number of threadsmulti_num=5,# options for BrowserHelper instanceoptions={"download_location":os.path.abspath("downloaded_files")},# direct download urlsurls=urls,# just print urlcallback=lambdabr:print(br.meta['url']))############ Enjoy ! ############Example 2'''what test does:Opens few browser instancesusing separate threads/processes,loads finder.ge-website, searches for specificword, parses returned page data and savesresults in json lines or csv file.# make sure that in br_helper you have already saved driver location #'''frombr_helper.br_helperimportMultiBrimporttime# for now, module is mainly designed for get requests# but we can do also posts this way# not most efficient, but should work########################################### define file format to save results in##########################################CASE="csv"# comma separated values# CASE = "jl"   # json lines########################################### define what words we want search forsearch_words=["Javascript","C#","PHP","Python","Golang",""]*3# say that we want to get same urls multiple timesmain_url="finder.ge"urls=[main_urlforiinsearch_words]# define callback function# it will be called every time browser loads our urldefcallback(br):'''when called, br_helper object (browser) will be passed here.Also in each case we will have access to requested page-s urlwith br-s meta dictionary(using url key).If we supplied other meta data, in get_with_multi methodwe will have access to this data here.'''# at first, make sure that there are no clicks needed# (site specific - go to finder.ge first time to see it)ifbr.find("თანხმობა",all_=True):br.find("თანხმობა").click()br.find("გაგრძელება").click()br.find("აღწერა").click()# search# with urls we also will pass word as meta data, so get it heretime.sleep(2)# to avoid dos restriction(503)search_word=br.meta["search_word"]# Just in case print checkprint(br.meta["url"])# find input(for different selectors see br_helper's methods)br.css1("input[placeholder]").send_keys(search_word)br.css1("input[value='ძიება']").click()# breakpoint()# get data (not the most reliable way)#######################################ifCASE=="jl":####################################### jl case (generate dictionary)######################################data=[{k:j.strip()forj,kinzip(i.text.split("\n"),["დასახელება","კომპანია","ბოლო ვადა"])}foriinbr.css("div.content")]# we can save data here, by hand,# but this time we are going# to set save_results argument to True, so# data returned from this function# will be saved in# jl file named data_CURRENT_TIME.jlreturn{"search_word":search_word,"vacancies_data":data}######################################elifCASE=="csv":######################################## csv case (generate list)######################################data=[[search_word]+i.text.split("\n")foriinbr.css("div.content")]headers=["სიტყვა","დასახელება","კომპანია","ბოლო ვადა"]return(headers,data)######################################## supported save methods for nowassertCASEin['jl','csv']# initializembr=MultiBr(save_format=CASE)# save jl or csv# start processesmbr.get_with_multi(multi_type="thread",# we can use process or thread heremulti_num=5,# how many of themoptions={},# options to use in browser in br_helper class# useful if we want to pass different proxies,# in which case options will be list of# options dictsurls=urls,# all urls we want to loadcallback=callback,# our function abovesave_results=True,# do we want to save callback answers in jl# or csv file?# if we are adding meta data, it should be# list of dictionaries with same length# and sequence as urls and all data in# these dictionaries will be available# in callback in br.meta dictionary.# do not use url as keys, it will be added# with information about requested/# not redirected(if that is the case)# url, every time automaticallymeta=[{"search_word":i}foriinsearch_words])Installation methodspip install br-helpergit clonehttps://github.com/Tornike-Skhulukhia/browser_automation_helperDependenciesseleniumbeautifulsoup4And downloadedchromedriver - if you want to use Chromegeckodriver - if you want to use FirefoxPython versionCode is tested onPython3.6and above