Package: bulkboto3
Bulk Boto3 (bulkboto3)Python package for fast and parallel transferring a bulk of files to S3 based on boto3!See on PyPI·View Examples·Report Bug/Request FeatureTable of ContentsAbout bulkboto3Getting StartedPrerequisitesInstallationUsageBlog PostsContributingContributorsContactLicenseAbout bulkboto3Boto3is the official Python SDK
for accessing and managing all AWS resources such as Amazon Simple Storage Service (S3).
Generally, it's pretty ok to transfer a small number of files using Boto3. However, transferring a large number of
small files impede performance. Although it only takes a few milliseconds per file to transfer,
it can take up to hours to transfer hundreds of thousands, or millions, of files if you do it sequentially.
Moreover, because Amazon S3 does not have folders/directories, managing the hierarchy of directories and files
manually can be a bit tedious especially if there are many files located in different folders.Thebulkboto3package solves these issues. It speeds up transferring of many small files to Amazon AWS S3 by
executing multiple download/upload operations in parallel by leveraging the Python multiprocessing module.
Depending on the number of cores of your machine, Bulk Boto3 can make S3 transfers even 100X faster than sequential
mode using traditional Boto3! Furthermore, Bulk Boto3 can keep the original folder structure of files and
directories when transferring them. There are also some other features as follows.Main FunctionalitiesMulti-thread uploading/downloading of a directory (keeping the directory structure) to/from S3 object storageDeleting all objects of an S3 bucketChecking the existence of an object on the S3 bucketListing all objects on an S3 bucketCreating a new bucket on the S3Getting StartedPrerequisitesPython 3.6+pipAPI credentials to access an S3Note:
You can deploy a free S3 server usingMinIOon your local machine by following the steps explained in:Deploy Standalone MinIO using Docker Compose on Linux.InstallationUse the package managerpipto installbulkboto3.pipinstallbulkboto3UsageYou can find the following scripts inexamples.pyandexamples.ipynb Notebook.Import and instantiate aBulkBoto3object with your credentialsfrombulkboto3importBulkBoto3TARGET_BUCKET="test-bucket"NUM_TRANSFER_THREADS=50TRANSFER_VERBOSITY=Truebulkboto_agent=BulkBoto3(resource_type="s3",endpoint_url="<Your storage endpoint>",aws_access_key_id="<Your access key>",aws_secret_access_key="<Your secret key>",max_pool_connections=300,verbose=TRANSFER_VERBOSITY,)Create a new bucketbulkboto_agent.create_new_bucket(bucket_name=TARGET_BUCKET)Upload a whole directory with its structure to an S3 bucket in multi-thread modeSuppose that there is a directory with the following structure on your local machine:test_dir├── first_subdir│   ├── f1│   ├── f2│   └── f3└── second_subdir└── f4To upload the directory (with its subdirectories) to the bucket
under a new directory name calledmy_storage_dir, use the following command:bulkboto_agent.upload_dir_to_storage(bucket_name=TARGET_BUCKET,local_dir="test_dir",storage_dir="my_storage_dir",n_threads=NUM_TRANSFER_THREADS,)# output:# 2022-03-26 18:12:40 — INFO — Start uploading from local 'test_dir' to 'my_storage_dir' on the object storage with 50 threads.# 100%|██████████| 4/4 [00:00<00:00,  4.00s/it]# 2022-03-26 18:12:41 — INFO — Successfully uploaded 4 files to bucket 'test-bucket' in 0.07 seconds.Download a whole directory with its structure to a local directory in multi-thread modebulkboto_agent.download_dir_from_storage(bucket_name=TARGET_BUCKET,storage_dir="my_storage_dir",local_dir="new_test_dir",n_threads=NUM_TRANSFER_THREADS,)# output:# 2022-03-26 18:14:08 — INFO — Start downloading from 'my_storage_dir' on storage to local 'new_test_dir' with 50 threads.# 100%|██████████| 4/4 [00:00<00:00,  4.00it/s]# 2022-03-26 18:14:09 — INFO — Successfully downloaded 4 files from bucket: 'test-bucket' in 0.04 seconds.The structure of the downloaded directory will be as follows on the local directory:new_test_dir└── my_storage_dir├── first_subdir│   ├── f1│   ├── f2│   └── f3└── second_subdir└── f4You can setlocal_dir=''(it is the default value) to avoid the creation of thenew_test_dirdirectory.Upload/Download arbitrary files to/from an S3 bucketTo transfer a list of arbitrary files to a bucket, you should instantiateStorageTransferPathclass
to determine the storage (s3) and local paths, and then use.upload()and.download()methods.
Here is an example:# upload arbitrary files from local to an S3 bucketupload_paths=[StorageTransferPath(local_path="test_dir/first_subdir/f2",storage_path="f2",),StorageTransferPath(local_path="test_dir/second_subdir/f4",storage_path="my_storage_dir/f4",),]bulkboto_agent.upload(bucket_name=TARGET_BUCKET,upload_paths=upload_paths)# output:# 100%|██████████| 2/2 [00:00<00:00,  2.44it/s]# 2022-04-05 13:40:10 — INFO — Successfully uploaded 2 files to bucket: 'test-bucket'.# download arbitrary files from an S3 bucket to localdownload_paths=[StorageTransferPath(storage_path="f2",local_path="f2",),StorageTransferPath(storage_path="my_storage_dir/f4",local_path="f5",),]bulkboto_agent.download(bucket_name=TARGET_BUCKET,download_paths=download_paths)# output:# 100%|██████████| 2/2 [00:00<00:00,  2.44it/s]# 2022-04-05 13:34:10 — INFO — Successfully downloaded 2 files from bucket: 'test-bucket'.Delete all objects on a bucketbulkboto_agent.empty_bucket(TARGET_BUCKET)# output:# 2022-03-26 22:23:23 — INFO — Successfully deleted objects on: 'test-bucket'.Check if a file exists in a bucketprint(bulkboto_agent.check_object_exists(bucket_name=TARGET_BUCKET,object_path="my_storage_dir/first_subdir/test_file.txt"))# output: Falseprint(bulkboto_agent.check_object_exists(bucket_name=TARGET_BUCKET,object_path="my_storage_dir/first_subdir/f1"))# output: TrueGet the list of objects in a bucket (with prefix)print(bulkboto_agent.list_objects(bucket_name=TARGET_BUCKET,storage_dir="my_storage_dir"))# output:# ['my_storage_dir/first_subdir/f1', 'my_storage_dir/first_subdir/f2', 'my_storage_dir/first_subdir/f3', 'my_storage_dir/second_subdir/f4']print(bulkboto_agent.list_objects(bucket_name=TARGET_BUCKET,storage_dir="my_storage_dir/second_subdir"))# output:# ['my_storage_dir/second_subdir/f4']BenchmarkUploaded 88800 small files (totally about 7GB) with 100 threads in 505 seconds that was about
72X faster than the non-parallel mode.Blog PostsBulkBoto3: Python package for fast and parallel transferring a bulk of files to S3 based on boto3!Deploy Standalone MinIO using Docker Compose on Linux.ContributingAny contributions you make aregreatly appreciated. If you have a suggestion that would make this better, please fork the repo and create a pull request.
You can also simply open an issue with the tag "enhancement". To contribute tobulkboto3, follow these steps:Fork this repositoryCreate a feature branch (git checkout -b feature/AmazingFeature)Make your changes and commit them (git commit -m 'Add some AmazingFeature')Push to the branch (git push origin feature/AmazingFeature)Open a pull requestAlternatively, see the GitHub documentation oncreating a pull request.ContributorsThanks to the following people who have contributed to this project:Amir Masoud Sefidian📖ContactIf you want to contact me you can reach me ata.m.sefidian@gmail.com.LicenseDistributed under theMITLicense. SeeLICENSEfor more information.