Package: selenium-respectful
Seleniumis already a well-known player
when it comes to browser automation and is usually a part of any serious
integration testing stack. That being said, it is also an increasingly
popular choice for web scraping. Now APIs are usually the ones with
detailed rate-limiting systems but if you are the type of person that
understands that web scraping is sometimes necessary and you intend to
be relatively gentle about it,selenium-respectfulmight be for you!*selenium-respectful*:Is a minimalist wrapper for anySelenium Webdriverto work within
rate limits of any amount of services simultaneouslyCan scale out of a single thread, single process or even a single
machineEnables maximizing your allowed requests without ever going over set
limits and having to handle the falloutOverloads the Webdriver’sgetmethod and relays any other valid
callsWorks with both Python 2 and 3 and is thoroughly testedIs a sister library to the already establishedrequests-respectfulTypical *Selenium* get callfromselenium.webdriver.chrome.webdriverimportWebDriverdriver=WebDriver()driver.get("http://github.com")element=driver.find_element_by_tag_name("body")Get call with *selenium-respectful*fromselenium.webdriver.chrome.webdriverimportWebDriverfromselenium_respectfulimportRespectfulWebdriverdriver=RespectfulWebdriver(webdriver=WebDriver())# This can be done elsewhere but the realm needs to be registered!driver.register_realm("Github",max_requests=100,timespan=60)driver.get("http://github.com",realms=["Github"],wait=True)element=driver.find_element_by_tag_name("body")# Works as usual!RequirementsRedis> 2.8.0 (See FAQ if you are rolling your
eyes)Installationpipinstallselenium-respectfulConfigurationDefault Configuration Values{"redis":{"host":"localhost","port":6379,"database":0},"safety_threshold":10}Configuration Keysredis: Provides thehost,portanddatabaseof the
Redis instancesafety_threshold: A rate-limited exception will be raised at(realm_max_requests - safety_threshold). Prevents going over the
limit of services in scenarios where a large amount of requests are
issued in parallelOverriding Configuration ValuesWithselenium-respectful.config.ymlThe library auto-detects the presence of a YAML file namedselenium-respectful.config.ymlat the root of your project and will
attempt to load configuration values from it.Example:selenium-respectful.config.ymlredis:host:0.0.0.0port:6379database:5safety_threshold:25The resulting active configuration would be:RespectfulWebdriver(webdriver=WebDriver()).configOut[1]:{"redis":{"host":"0.0.0.0","port":6379,"database":5},"safety_threshold":25}UsageIn your quest to useselenium-respectful, you should only ever have to
bother with one class:RespectfulWebdriver. Instance this class and
you can perform all important operations.Before each example, it is assumed that the following code has already
been executed.fromselenium.webdriver.chrome.webdriverimportWebDriverfromselenium_respectfulimportRespectfulWebdriverdriver=RespectfulWebdriver(webdriver=WebDriver())RealmsRealms are simply named containers that are provided with a maximum
requesting rate. You are responsible of the management (i.e. CRUD) of
your realms.Realms track the HTTP requests that are performed under them and will
raise a catchable rate limit exception if you are over their allowed
requesting rate.Fetching the list of Realmsdriver.fetch_registered_realms()This returns a list of currently registered realm names.Registering a Realmdriver.register_realm("Google",max_requests=10,timespan=1)driver.register_realm("Github",max_requests=100,timespan=60)driver.register_realm("Twitter",max_requests=150,timespan=300)# ORrealm_tuples=[["Google",10,1],["Github",100,60],["Twitter",150,300]]driver.register_realms(realm_tuples)Either of these registers 3 realms: *Googleat a maximum requesting
rate of 10 requests per second *Githubat a maximum requesting rate
of 100 requests per minute *Twitterat a maximum requesting rate of
150 requests per 5 minutesUpdating a Realmdriver.update_realm("Google",max_requests=25,timespan=5)This updates the maximum requesting rate ofGoogleto 25 requests per
5 seconds.Getting the maximum requests value of a Realmdriver.realm_max_requests("Google")This would return 25.Getting the timespan value of a Realmdriver.realm_timespan("Google")This would return 5.Unregistering a Realmdriver.unregister_realm("Google")This would unregister theGooglerealm, preventing further queries
from executing on it.Unregistering multiple Realmsdriver.unregister_realms(["Google","Github","Twitter"])This would unregister all 3 realms in one operation, preventing further
queries from executing on them.RequestingUsing theSelenium Webdriverget methodTo pilot your web browser to a given URL, just use thegetmethod as
you would normally do with your WebDriver instance. The only major
difference is that arealmskwarg is expected. Awaitboolean kwargs
can also be provided (the behavior is explained later).Example of a valid call:driver.get("http://github.com",realms=["GitHub"])If not rate-limited, it would direct the browser to the provided URL.Multiple realms per requestYou can have a single request count against multiple realms if it makes
sense in your use case.driver.get("http://github.com",realms=["GitHub","GitHubUser123","GitHubServer3"])Handling exceptionsExecuting thesegetcalls will either perform the action in the
browser or raise a SeleniumRespectfulRateLimitedError exception. This
means that you’ll likely want to catch and handle that exception.fromselenium_respectfulimportSeleniumRespectfulRateLimitedErrortry:driver.get("http://github.com",realm="GitHub")exceptSeleniumRespectfulRateLimitedError:pass# Possibly requeue that call or wait.ThewaitkwargRequesting with agetcall accepts awaitkwarg that defaults to
False. If switched on and the realm is currently rate-limited, the
process will block, wait until it is safe to send requests again and
perform the requests then. Waiting is perfectly fine for scripts or
smaller operations but is discouraged for large, multi-realm, parallel
tasks (i.e. Background Tasks like Celery workers).TestsExist?YesExhaustive?YesFacepalm tactics?Yes -  Redis calls aren't mocked and google.com gets a few friendly callsRun them withpython-mpytest tests--specFAQWhoa, whoa, whoa! Redis?!Yes. The use of Redis allows forselenium-respectfulto go
multi-thread, multi-process and even multi-machine while still
respecting the maximum requesting rates of registered realms. Operations
like Redis’ SETEX are key in designing and working with rate-limiting
systems. If you are doing Python development, there is a decent chance
you already work with Redis as it is one of the two options to use as
Celery’s backend and one of the 2 major caching options in Web
development. If not, you can always keep things clean and use aDocker
Containeror evenbuild it from
source. Redis has kept a
consistent record over the years of being lightweight, solid software.