Package: requests-session-plus
requests-session-plusDrop in replacement forrequests.Session()with some quality of life enhancements.>>>fromrequests_session_plusimportSessionPlus# equivalent to "from requests import Session">>>s=SessionPlus()>>>r=s.get("https://httpbin.org/basic-auth/user/pass",auth=("user","pass"))>>>r.status_code200>>>r.headers["content-type"]'application/json'>>>r.encoding'utf-8'>>>r.text'{\n"authenticated": true,\n"user": "user"\n}\n'Installing requests_session_plusrequests_session_plus is available on PyPI:$python-mpipinstallrequests-session-plusComparison to Requests Session ClassAll of these features are currently available in the standard requests Session class with some configuration/modification.  The goal of SessionPlus is to make them more easily accessible.FeatureSession()SessionPlus()Default HTTP(S) Call Timeout010HTTP(S) Timeout Setper callglobally and per callDisable Certificate Verificationper callglobally and per callDisable Certificate Verification WarningsnoyesRaise Exceptions For Client/Server IssuesnoyesRetry Count05Retry Backoff Factor02Retry For Status Codes413, 429, 503413, 429, 500, 502-504Timeouts and certificate verification are enabled by default in SessionPlus, the others disabled.  All features can be enabled/disabled ad hoc as needed.UsageSessionPlus can be used in the exact same way as arequests Session objectso I'm going to rely on their documentation for most use cases.  In the following sections I'll just go over the benefits of each feature this package utilizes and how to enable/disable/modify them.To make the most out of this package and its features you should have a strong understanding of your HTTP endpoints and how each feature could help.  Some suspiciously specific examples that I may or may not have run into:Making API calls to an internal network appliance where tacacs occasionally fails?Disable certificate verificationEnable retriesAdd 401 to "retry_status_forcelist" parameterIs there an API endpoint hosted in Kubernetes where aggressive health checks recycle the containers leading to occasional 502 Bad Gateway responses?Enable retriesIs there an API endpoint that has a habit of either responding quickly or getting hung and not responding at all?Enable retriesSet the timeout to be a little higher than its average response time (if 10 second default isn't sufficient)Is there a cheeky developer who likes to use quirky or non-standard HTTP status codes (such as 418 I'm a Teapot) for client/server issues?Enable status exceptions which raises an exception for any status code >=400You can also enable retries and expand the "retry_status_forcelist" parameterAny status code in "retry_status_forcelist" will issue retriesAll other status codes >=400 will raise an exceptionExample: lets retry 418, but don't bother retrying for 411Certificate VerificationThis is enabled by default in both the default Session class and SessionPlus.  SessionPlus just provides an easy way to toggle it on and off globally.It is not recommended to disable certificate verification but useful when working with HTTP endpoints which use a self-signed certificate or have some other certificate issue.  It both disables thecertificate checkbut also disables the warnings that bark at you when you disable certificate checks.NOTE:Ifretriesare also enabled, retries will be issued for HTTP calls to servers that have bad certificates.Configuring Certificate VerificationParameter:verify : boolean : verify certificate or not. Defaults to True>>>fromrequests_session_plusimportSessionPlus>>>s=SessionPlus()# enabled by default>>>s=SessionPlus(verify=False)# disable certificate verificationCertificate verification can be toggled on/off>>>s=SessionPlus()# enabled by default>>>s.verify=False# temporarily disable it>>># ... make 1 or more HTTP call to server with a bad certificate ...>>>s.verify=True# re-enable and continueMaking an HTTP call to server with a bad cert>>>s=SessionPlus()>>>s.get("https://self-signed.badssl.com/")# ... output compressed ...# SSLError exception thrownrequests.exceptions.SSLError:HTTPSConnectionPool(host='self-signed.badssl.com',port=443):Maxretriesexceededwithurl:/(CausedbySSLError(SSLcertificateVerificationError(1,'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:997)')))>>>Same results with retries enabled, it just takes longer as retries are performed with increasing backoff timer>>>s=SessionPlus(retry=True)>>>s.get("https://self-signed.badssl.com/")# ... output compressed ...# SSLError exception thrown after 5 retries (default retry total)requests.exceptions.SSLError:HTTPSConnectionPool(host='self-signed.badssl.com',port=443):Maxretriesexceededwithurl:/(CausedbySSLError(SSLCertVerificationError(1,'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:997)')))If we utilize the requests way of disabling certificates, we get warnings>>>s=SessionPlus()>>>s.get("https://self-signed.badssl.com/",verify=False)# disable certificate verification# warnings are thrown/home/chambersh1129/Documents/code/personal/requests-session-plus/venv/lib/python3.10/site-packages/urllib3/connectionpool.py:1045:InsecureRequestWarning:UnverifiedHTTPSrequestisbeingmadetohost'self-signed.badssl.com'.Addingcertificateverificationisstronglyadvised.See:https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warningswarnings.warn(# but no exception is thrown<Response[200]>Now we can try it the SessionPlus way.  No exceptions, no warnings>>>s=SessionPlus(verify=False)>>>s.get("https://self-signed.badssl.com/")<Response[200]>Client/Server Error ExceptionsThis is disabled by default in both the default Session class and SessionPlus and takes a few steps to enable with the default Session class.  SessionPlus just provides an easy way to toggle it on and off.Sometimes you just want to know if the HTTP call worked or not instead of having a lot of conditionals checking the status code(s).  This setting will raise an exception if the status code is >=400.  The status code will be provided in the error message if you still want to get the status code but you will not have access to the response object.NOTE:Ifretriesare also enabled, certain status codes will issue a retry with a backoff timer.  These status codes are configurable withretry_status_forcelist.Configuring Client/Server Error ExceptionsParameter:status_exceptions : boolean : whether exceptions should be raised or not. Defaults to False>>>fromrequests_session_plusimportSessionPlus>>>s=SessionPlus()# disabled by default>>>s=SessionPlus(status_exceptions=True)# rase exception for status codes >= 400Status exceptions can be toggled on/off>>>s=SessionPlus(status_exceptions=True)# raise exceptions>>>s.status_exceptions=False# disable temporarily>>># ... make HTTP call we want response object regardless of status code ...>>>s.status_exceptions=True# back to raising exceptions for status codes >= 400An example, with status_exceptions and retries enabled.>>>s=SessionPlus(status_exceptions=True,retries=True)>>>s.get("https://httpstat.us/418/")# 418 I'm a teapot# ... output compressed ...# HTTPError exception thrown without retriesrequests.exceptions.HTTPError:418ClientError:Imateapotforurl:https://httpstat.us/418/Note:If both status_exceptions and retries are enabled and the status code is inretry_status_forcelist, retries will be issued.  If this is unwanted behavior, retry_status_forcelist could be modified to be an empty list or set.>>>s=SessionPlus(status_exceptions=True,retries=True)>>>s.get("https://httpstat.us/429/")# 429 Too Many Requests# ... output compressed ...# RetryError exception thrown after 5 retries (default)requests.exceptions.RetryError:HTTPSConnectionPool(host='httpstat.us',port=443):Maxretriesexceededwithurl:/429/(CausedbyResponseError('too many 429 error responses'))If we disable retries, we are back to the HTTPError we got with the 418.>>>s=SessionPlus(status_exceptions=True)>>>s.get("https://httpstat.us/429/")# 429 Too Many Requests# ... output compressed ...# HTTPError exception thrown without retries, same as the 418 aboverequests.exceptions.HTTPError:429ClientError:TooManyRequestsforurl:https://httpstat.us/429/RetriesThe default Session class does not perform retries, and when enabled a backoff of 0 is set meaning it does not wait between HTTP calls.Retries are helpful if the server uptime is spotty and the calls are idempotent.  Instead of setting a loop to try/fail/sleep/repeat (or worse, try/fail/break), SessionPlus will enable retries with somehelpful defaults.When are Retries Performed?If retries are enabled, they will be used for:TimeoutErrors when timeoutes are enabledSSLErrors when verify=TrueFor certain status codes set in retry_status_forcelist, even if status_exceptions=TrueThe default status codes configured for retries in SessionPlus are:413 Payload Too Large429 Too Many Requests500 Internal Server Error502 Bad Gateway503 Service Unavailable504 Gateway TimeoutHow long will Retries Take?There is a formula to determine how long to wait between retries (found in theRetry docs):{backoff factor} * (2 ** ({number of total retries} - 1))Example #1Parametersbackoff_factor = 2 (SessionPlus default)total = 5 (SessionPlus default)timeout = 10 (SessionPlus default)server responds immediately with a 429 Too Many Requests so timeout does not come into playRetries will be sent at2s after the first failure4s after the second failure8s after the third failure16s after the fourth failure32s after the fifth failureA total of 62 seconds is spent trying to get a response.  The Too Many Requests issue might be resolved by then.Example #2Parametersbackoff_factor = 2 (SessionPlus default)total = 5 (SessionPlus default)timeout = 10 (SessionPlus default)server takes >10 seconds to respondRetries will be sent at10s timeout + 2s after the first failure10s timeout + 4s after the second failure10s timeout + 8s after the third failure10s timeout + 16s after the fourth failure10s timeout + 32s after the fifth failurerequests.exceptions.ReadTimeout exception is raisedA total of 112 seconds is spent trying to get a response.  In this case,disabling or increasing the timeoutcould be useful.Example #3Parametersbackoff_factor = 10 (5x increase over SessionPlus default)total = 5 (SessionPlus default)timeout = 10 (SessionPlus default)server responds immediately with a 503 Service UnavailableRetries will be sent at10s after the first failure20s after the second failure40s after the third failure80s after the fourth failure160s after the fifth failureA total of 310 seconds is spent trying to get a response.  The Service Unavailable issue might be resolved by then.Configuring RetriesSessionPlus sets defaults for 3 specific parameters and there is a fourth parameter to enable/disable retries.  You can pass additional parameters for the retries by prepending "retry_" to them, a full list can be found in theurllib3 Retry docs.NOTE: Before altering the default values below be concious of how long the worst case HTTP call will take.  These defaults were chosen so the maximum time waiting for all retries is between 62 seconds (if server responds immediately) and 112 seconds (if timeouts are hit).  Increasing these values for performance or latency senstive applications could lead to issues.Parameter:retry : bool : enable or disable retry functionality. Defaults to Falseretry_backoff_factor : float : used in the formula to determine how long to wait before retrying.  Defaults to 2retry_status_forcelist : set : HTTP status codes to retry. Defaults to {413, 429, 500, 502, 503, 504}retry_total: int : total number of retries before failing. Defaults to 5>>>fromrequests_session_plusimportSessionPlus>>>s=SessionPlus()# retries disabled by default>>>s=SessionPlus(retry=True)# enable retriesRetries can be toggled on/off>>>s=SessionPlus(retry=True)>>>s.retry=False# retries are disabled>>># ... make HTTP call where retries aren't needed ...>>>s.retry=True# re-enable retriesViewing/Changing Retry Settings>>>s=SessionPlus(retry=True)>>># view the settings>>>s.retry_settings{'backoff_factor':2,'status_forcelist':{429,500,502,503,504,413},'total':5}>>># modify a default setting>>>s.retry_total=10>>>s.retry_backoff_factor=5>>>s.retry_settings{'backoff_factor':5.0,'status_forcelist':{429,500,502,503,504,413},'total':10}>>># passing in new Retry settings.  Note: no validation is done for those not listed above in Parameters>>>s.retry_raise_on_status=False>>>s.retry_settings{'backoff_factor':5.0,'status_forcelist':{429,500,502,503,504,413},'total':10,'raise_on_status':False}>>># initialize new session with all of these settings>>>new_session=SessionPlus(retry=True,retry_raise_on_status=False,retry_total=10,retry_backoff_factor=5)>>>new_session.retry_settings{'backoff_factor':5.0,'status_forcelist':{429,500,502,503,504,413},'total':10,'raise_on_status':False}>>># settings persist, even when retries are disabled>>>new_session.retry=False>>>new_session.retry_settings{'backoff_factor':5.0,'status_forcelist':{429,500,502,503,504,413},'total':10,'raise_on_status':False}retry_status_forcelist is a set, so you need to use add, remove, or = to update it>>>s=SessionPlus(retry=True)>>>s.retry_status_forcelist{429,500,502,503,504,413}>>># add a status code>>>s.retry_status_forcelist.add(307)>>>s.retry_status_forcelist{429,307,500,502,503,504,413}>>># remove a status code>>>s.retry_status_forcelist.remove(502)>>>s.retry_status_forcelist{429,307,500,503,504,413}>>># update entire set.  Can be set or list of integers>>>s.retry_status_forcelist={418}>>>s.retry_status_forcelist{418}For the retry settings to be completed updated, you need to run .update_retry().  This is done automatically when enabling/disabling retries, this is just a helper method to run if you need to change an existing enabled retry.>>>s=SessionPlus(retry=True)>>>s.retry_settings{'backoff_factor':2,'status_forcelist':{429,500,502,503,504,413},'total':5}>>># update the settings>>>s.retry_total=10>>>s.retry_settings{'backoff_factor':2,'status_forcelist':{429,500,502,503,504,413},'total':10}>>># apply the settings>>>s.update_retry()For the parameters SessionPlus has defaults for, there is also input validation.  Other parameters are passed straight to the Retry object.>>>s=SessionPlus(retry=True)>>>s.retry_status_forcelist=429Traceback(mostrecentcalllast):File"<stdin>",line1,in<module>File"/home/chambersh1129/Documents/code/personal/requests-session-plus/requests_session_plus/__init__.py",line68,in__init__self.retry_status_forcelist=retry_status_forcelistFile"/home/chambersh1129/Documents/code/personal/requests-session-plus/requests_session_plus/__init__.py",line123,inretry_status_forcelistraiseValueError("retry_status_forcelist must be a set of integers")ValueError:retry_status_forcelistmustbeasetofintegers>>>>>>s.retry_read="this should be an integer">>>s.update_retry()>>>s.get("https://httpstat.us/429/")# 429 Too Many Requests# ... output compressed ...# TypeError exception thrown when first retry is attemptedTypeError:'<'notsupportedbetweeninstancesof'str'and'int'TimeoutsSome HTTP calls should only take X amount of time, and if it takes longer the server is likely hung or some other issue.  Timeouts allow you to set a maximum time to wait before declaring the server unresponsive and moving on.The default Session class supports timeouts per HTTP call, SessionPlus just provides the ability to set it globally in addition to per call.Configuring TimeoutsParameter:timeout : [float,None] : how long to wait, in seconds, before raising an exception. Defaults to 10>>>fromrequests_session_plusimportSessionPlus>>>s=SessionPlus()# timeout set to 10 seconds>>>s=SessionPlus(timeout=None)# disable the timeoutGlobal Timeout can be toggled on/off>>>s=SessionPlus()# timeout set to 10 seconds>>>s.timeout=None# now no call has a timeout>>># ... make a really long HTTP call ...>>>s.timeout=30# set it back to whatever you wantYou can still overwite it per call and the default value is maintained for future calls>>>s=SessionPlus(timeout=1)>>>s.get("https://httpstat.us/200/?sleep=1250")# 200 OK that takes 1.25 seconds to respond# ... ouput compressed ...# TimeoutError triggers a ReadTimeoutError which triggers a MaxRetryError and eventually a ConnectionErrorrequests.exceptions.ConnectionError:HTTPSConnectionPool(host='httpstat.us',port=443):Maxretriesexceededwithurl:/200/?sleep=1250(CausedbyReadTimeoutError("HTTPSConnectionPool(host='httpstat.us', port=443):Readtimedout.(readtimeout=1.0)"))>>>>>># disable retries and try again>>>s.retry=False# ... output compressed ...# TimeoutError triggers a ReadTimeoutError which triggers a ReadTimeoutrequests.exceptions.ReadTimeout:HTTPSConnectionPool(host='httpstat.us',port=443):Readtimedout.(readtimeout=1.0)>>>>>># increase the timeout for this call only>>>s.timeout1.0>>># make the call again, overwriting global timeout>>>s.get("https://httpstat.us/200/?sleep=1250",timeout=2)# 200 OK that takes 1.25 seconds to respond<Response[200]>>>># global is unchanged>>>s.timeout1.0NOTE:Once again, disabling timeouts isn't a silver bullet.  Other timeouts come into play, both at the python level (urllib3 timeouts, socket timeouts) or at the server level (NGINX proxy_read_timeout for example).  Setting timeout=None for an HTTP call does not guarantee exceptions aren't raised.