Package: resilient-caller
Resilient CallerA Python package that provides a customizable wrapper to retry function calls with custom logic. This package was developed to address the need for executing numerous requests with similar, yet slightly different, exception handling. The wrapper reduces the need to write multiple while loops and try/except blocks for each request.The wrapper can be implemented for any function, not just requests-related functions. The module also includes a Python requests implementation with auto proxy formatting from a string, by simply passing the proxy as a string to the send_request function. Async functions support as well.The resilient caller supports the following keyword arguments (kwargs) for the wrapper. Note that these kwargs will not be passed to the wrapped function:conditions: A dictionary specifying the actions to take for a given outcome.conditions_criteria: The criteria to use when checking theconditions.exceptions: A dictionary specifying the actions to take for a given exception or 'all' for non-handled or all exceptions.retries: The maximum number of times to retry the function (disabled by default).delay: The number of seconds to sleep between retries.on_retry: A callback function to execute on retry, for example, a log function.
In the module, a Python requests implementation is provided with automatic proxy formatting from a string.Please refer to the usage examples below and the examples folder in the repository for more information on how to use the resilient caller.InstallationInstall the package using pip:pip install resilient_callerExamplesQuick simple and easy web scraping monitorIn this example, we will create aSimpleScraperclass that monitorsglizzykingdreko's medium blogfor new articles. By defining the start method with the@resilient_call()decorator and passingdelay=5andexceptions={"all": RETRY_EVENT}when calling it, we ensure that the scraper handles all exceptions with a retry and adds a delay of 5 seconds between each request.frombs4importBeautifulSoupfromrequestsimportSessionfromtypingimportListfromtimeimportsleepfromresilenter_callerimportresilient_call,RETRY_EVENTclassSimpleScraper:def__init__(self):self.session,self.articles=Session(),[]# This is making the function run through a while loop with a delay of 5 seconds# and handling any exception with a retry event.self.start(delay=5,exceptions={"all":RETRY_EVENT})@resilient_call()defstart(self)->None:data=self.load_api_data()self.load_response_details(data)defload_response_details(self,response:str)->List[str]:data=BeautifulSoup(response,"html.parser")new_articles=[[d.find("h2").text,d.find("a").get("href")]fordindata.find_all("article")ifd.find("h2").textnotinself.articles]forarticleinnew_articles:name,url=articleprint(f"New glizzykingdreko's article on medium\"{name}\"! Check it out at{url}.")self.articles.append(name)returnnew_articlesdefload_api_data(self)->str:url="https://medium.com/@glizzykingdreko"returnself.session.get(url).textif__name__=="__main__":SimpleScraper()Web scraping with retry, custom handling and handling for all exceptionsIn this example, we will use the send_request() function provided in the module to perform
web scraping. We will also use all the available options to customize the handling of
different HTTP response codes and handle all exceptions.frombs4importBeautifulSoupfromresilenter_callerimportsend_request,RETRY_EVENTdefhandle_success(response):print(f"Request successful, status code:{response.status_code}")soup=BeautifulSoup(response.text,"html.parser")title=soup.find("title")print(f"Page title:{title.string}")returnresponsedefhandle_not_found(response):print(f"Page not found, status code:{response.status_code}")returnRETRY_EVENTdefhandle_server_error(response):print(f"Server error, status code:{response.status_code}")returnRETRY_EVENTdefhandle_all_exceptions(exception):print(f"An exception occurred:{type(exception).__name__}-{exception}")returnRETRY_EVENTif__name__=="__main__":response=send_request("https://www.example.com",retries=3,delay=2,conditions={200:handle_success,404:handle_not_found,500:handle_server_error},exceptions={"all":handle_all_exceptions},on_retry=lambdatries:print(f"Retry{tries}"))File processing with retry and custom handlingIn this example, we will use the resilient_call() decorator to implement a function
that processes a file and retries the operation in case of failure. We will also
use all the available options to customize the handling of different file sizes.importosfromresilenter_callerimportresilient_call,RETRY_EVENTdefprocess_large_file(file_path):print(f"Processing large file:{file_path}")returnRETRY_EVENTdefprocess_small_file(file_path):print(f"Processing small file:{file_path}")returnRETRY_EVENTdefprocess_valid_file(file_path):print(f"Processing valid file:{file_path}")returnfile_path@resilient_call()defprocess_file(file_path):file_size=os.path.getsize(file_path)returnfile_sizeif__name__=="__main__":processed_file=process_file("example.txt",retries=5,delay=2,conditions={-1:process_large_file,1:process_small_file},conditions_criteria=lambdafile_size:-1iffile_size>1000000else1iffile_size<1000else0,on_retry=lambdatries:print(f"Retry{tries}"))Asynchronous API call with rate limiting retryIn this example, we will use the resilient_call() decorator to implement an
asynchronous function that makes an API call and retries the call in case of failure or rate limiting.importaiohttp,asynciofromresilenter_callerimportresilient_call,RETRY_EVENTasyncdefhandle_rate_limit(e):print(f"Rate limited:{e}")returnRETRY_EVENT@resilient_call()asyncdefasync_api_call(url):asyncwithaiohttp.ClientSession()assession:asyncwithsession.get(url)asresponse:ifresponse.status==429:raiseException("Rate limited")data=awaitresponse.json()returndataasyncdefmain():awaitasync_api_call("https://httpbin.org/status/429",retries=3,delay=5,exceptions={Exception:handle_rate_limit})if__name__=="__main__":asyncio.run(main())Custom backoff strategy (exponential backoff)In this example, we will create a function that will randomly fail with a 30% chance.
If it fails, we will retry the function with an exponential backoff strategy.importloggingimportrandomfromresilenter_callerimportresilient_call,RETRY_EVENT# Set level as debug to get full logslogging.basicConfig(level=logging.DEBUG)defexponential_backoff(tries):return2**(tries-1)+random.uniform(0,1)# If 2 arguments are passed to an exception function# the second argument will be the number of tries# (same thing for a condition)defprint_exception(exception,tries):print(f"Exception:{exception}(try{tries})")returnRETRY_EVENT@resilient_call()defexample_function():random_num=random.random()ifrandom_num<0.7:print("Failed, retrying...")raiseValueError("Random number too low")else:print("Success!")return"Successful response"if__name__=='__main__':result=example_function(retries=5,on_retry=exponential_backoff,exceptions={ValueError:print_exception})print("Result:",result)Pass the number of tries to the action functionIn this example, by using a function that takes 2 arguments, we
can pass the number of tries to the action function.importrandomfromresilenter_callerimportresilient_call,RETRY_EVENTdefsome_condition(response):returnresponse=="Retry"defhandle_response(response,tries):iftries<3andsome_condition(response):returnRETRY_EVENTelse:returnresponse# We set the max execution time to 10 seconds@resilient_call(max_elapsed_time=10)defexample_function():random_num=random.random()ifrandom_num<0.6:print("Returning 'Retry'")return"Retry"else:print("Success!")return"Successful response"if__name__=='__main__':result=example_function(# With 'all' we can handle all the# possible responses or exceptionsconditions={'all':handle_response},)print("Result:",result)These examples demonstrate the versatility and usefulness of the Resilient Caller module. This module can be applied to a wide range of use cases, from web scraping and file processing to API calls and custom backoff strategies. Make sure to explore the examples folder in the repository for even more examples and use cases with other parameters and configurations.Personal ThoughtsI hope this module will help many developers save time and make their code more efficient. Please feel free to contact me for any help or suggestions viaEmailorTwitter. Don't forget to follow me on GitHub and Medium for more exciting content and updates. I appreciate your feedback and contributions to the project.ContributingI welcome contributions to the Resilient Caller project! To contribute, please follow these steps:Fork the repository on GitHub.Create a new branch with a descriptive name.Make your changes, add new features, or fix bugs.Write tests to ensure that your changes work as expected.Update the documentation and examples to reflect your changes.Commit your changes and create a pull request.
Please make sure to follow the existing code style and provide clear, concise commit messages. If you have any questions, feel free to open an issue, and we'll be happy to help.LicenseThis project is licensed under the MITLicense. See the LICENSE file for more details.My linksProject repositoryGitHubTwitterMediumEmail