Package: aboto3
aboto3aboto3is an async boto3 client generator!There are other boto3-like libraries that offer asyncio but the interface can be quite different from normalboto3clients.
The goal ofaboto3is to closely replicate theboto3client interface with acceptable performance from the pythonThreadPoolExecutor!API NOTE-aboto3was created with boto3 API compatibility in mind.  Because of this it does not supportboto3"Resources", and there is no plan to support them. New "Resources" are no longer being added toboto3.Performance NOTE- Becauseaboto3provides an async wrapper aroundboto3, it is not truly async to it's core. It sends a boto3 call to a thread and the thread runs asynchronously.asynciotasks are much lighter weight than threads so if you want to run hundreds of concurrent calls with the best performance, pure async boto3 equivalents are a better fit.TutorialTo create an async client simply pass in the normalboto3client to create anAIOClient.Use the async client, in a coroutine, like you would if the boto3 client's API calls were all async!  Seeboto3docsfor details.importasyncioimportaboto3importboto3# create a normal boto3 clientec2_client=boto3.client("ec2")# create the asyncio version from the clientaio_ec2_client=aboto3.AIOClient(ec2_client)# you can still use the other client as usualinstances=ec2_client.describe_instances()# the async client must be used in a coroutine# but acts exactly the same as the boto3 client except method calls are asyncasyncdefaio_tester():aio_instances=awaitaio_ec2_client.describe_instances()returnaio_instancesaio_instances=asyncio.run(aio_tester())Pass in parameters to the coroutine like a normal client.importasyncioimportaboto3importboto3ec2_client=boto3.client("ec2")aio_ec2_client=aboto3.AIOClient(ec2_client)instances=ec2_client.describe_instances(InstanceIds=["i-123412341234"])asyncdefaio_tester():aio_instances=awaitaio_ec2_client.describe_instances(InstanceIds=["i-123412341234"])returnaio_instancesaio_instances=asyncio.run(aio_tester())Get an async paginator from theaboto3client.importasyncioimportaboto3importboto3ec2_client=boto3.client("ec2")aio_ec2_client=aboto3.AIOClient(ec2_client)filters=[{"Name":"instance-type","Values":["t2.micro"]}]pages=[]pager=ec2_client.get_paginator("describe_instances")forpageinpager.paginate(Filters=filters):pages.append(page)# note the use of an "async for" loop so calls for a page are non-blocking.asyncdefaio_tester():aio_pages=[]aio_pager=aio_ec2_client.get_paginator("describe_instances")asyncforpageinaio_pager.paginate(Filters=filters):aio_pages.append(page)returnaio_pagesaio_pages=asyncio.run(aio_tester())Client exceptions can be caught on theAIOClientjust like a normalboto3client.botocoreexceptions are caught as normal.importasyncioimportaboto3importboto3ssm_client=boto3.client("ssm")aio_ssm_client=aboto3.AIOClient(ssm_client)try:ssm_client.get_parameter(Name="/unknown/param")exceptssm_client.exceptions.ParameterNotFoundaserror:print("found an error here:{}".format(error))asyncdefaio_tester():try:aio_ssm_client.get_parameter(Name="/unknown/param")exceptaio_ssm_client.exceptions.ParameterNotFoundaserror:print("found an error here:{}".format(error))aio_pages=asyncio.run(aio_tester())You can also useboto3augmenting libraries sinceaboto3is only a wrapper.OptimizationWhen anAIOClientis created it will automatically create aThreadPoolExecutorto run the boto3 calls asynchronously.  The size of max workers of the pool is determined by the boto3 client's config formax_pool_connections. By default this is 10.
Seebotocore Config Referencefor more details.The thread pool adds a small amount of overhead for eachAIOClientthat is created (though this is far less than the overhead of creating a boto3 client).  To save some initialization time or have more control over total number of threads you can provide your ownThreadPoolExecutorand share this between clients.importasynciofromconcurrent.futuresimportThreadPoolExecutorimportaboto3importboto3boto3_thread_pool=ThreadPoolExecutor(max_workers=16)ec2_client=boto3.client("ec2")aio_ec2_client=AIOClient(boto3_client=ec2_client,thread_pool_executor=boto3_thread_pool)rds_client=boto3.client("rds")aio_rds_client=AIOClient(boto3_client=rds_client,thread_pool_executor=boto3_thread_pool)In general, for applications, you will want to cache the clients if possible. Try not to create a new one in every function. For applications, a shared thread pool can be useful in limiting the total number of threads, when necessary.If you are making large numbers of concurrent calls with the sameAIOClientyou may want to pass in a custombotocore.config.Configto the boto3 client with a highermax_pool_connections.  If you are using a shared thread pool you may also need to increase the max workers in that as well.The example below will allow up to 32 concurrent calls to be in flight for the EC2 and RDSAIOClient's.importasynciofromconcurrent.futuresimportThreadPoolExecutorimportaboto3importboto3frombotocore.configimportConfigboto_config=Config(max_pool_connections=32)boto3_thread_pool=ThreadPoolExecutor(max_workers=64)ec2_client=boto3.client("ec2",config=boto_config)aio_ec2_client=AIOClient(boto3_client=ec2_client,thread_pool_executor=boto3_thread_pool)rds_client=boto3.client("rds",config=boto_config)aio_rds_client=AIOClient(boto3_client=rds_client,thread_pool_executor=boto3_thread_pool)Or if you don't care about sharing the thread pool just pass in the config
and eachAIOClientwill have it's own pool of 32 threads.importasyncioimportaboto3importboto3frombotocore.configimportConfigboto_config=Config(max_pool_connections=32)ec2_client=boto3.client("ec2",config=boto_config)aio_ec2_client=AIOClient(boto3_client=ec2_client)rds_client=boto3.client("rds",config=boto_config)aio_rds_client=AIOClient(boto3_client=rds_client)DevelopmentInstall the package in editable mode with dev dependencies.(venv) $ pip install -e .[dev]noxis used to manage various dev functions.
Start with(venv) $ nox --helppyenvis used to manage python versions.
To run the nox tests for applicable python version you will first need to install them.
In the root project dir run:(venv) $ pyenv installChangelogChangelog foraboto3.
All notable changes to this project will be documented in this file.The format is based onKeep a Changelog,
and this project adheres toSemantic Versioning.[0.1.1] - 2024-01-28AddedTests for python 3.12ChangedUpdated README to reflect recommendations.RemovedSupport for python 3.7[0.1.0] - 2023-06-22Initial Release.