Package: boto-s3-router
Boto S3 RouterInteract with S3-compatible services side-by-side with S3, without making any changes to your code. This package provides a Boto3-like client that routes requests between S3 clients according to the bucket and the key in the request.Why use Boto S3 RouterConsider the following code, downloading an object from one bucket and uploading it to another after some manipulation:defcalculate(s3_client):obj=s3_client.get_object(Bucket="bucket1",Key="a/b/c/obj")obj2=do_something_to_obj(obj)s3_client.put_object(Body=obj2,Bucket="bucket2",Key="a/b/c/obj")Suppose you want to migrate onlybucket2to an S3-compatible service like lakeFS or MinIO.
Normally, you would have to refactor your code to allow using two S3 clients instead of just one.
Boto S3 Router allows you to do that without making any changes to thecalculatefunction!InstallationPrerequisitesBoto S3 Router requires Python >= 3.6 to run.Using pippipinstallboto-s3-routerSetuptoolsInstall viaSetuptools.pythonsetup.pyinstall--userBasic Usageimportboto3importboto_s3_routerass3r# Initialize two boto S3 clients:s3_east=boto3.client('s3',region_name='us-east-1',signature_version='v4',)s3_west=boto3.client('s3',region_name='us-west-1')# Define rules for routing between the clients:profiles={"s3_west":{"source_bucket_pattern":"example-bucket","mapped_bucket_name":"new-bucket","mapped_prefix":"test/"},"s3_east":{"source_bucket_pattern":"example-bucket-prefix*","source_key_pattern":"a/*","mapped_prefix":"test/"}}# Define the mapping between the profiles to the Boto clients:client_mapping={"s3_west":s3_west,"s3_east":s3_east,"default":s3_east}# Initialize the client:client=s3r.client(client_mapping,profiles)# Use Boto S3 Router as you would use any Boto client:client.put_object(Bucket="bucket",Prefix="a/b/obj")# routs to s3_west, the object will be "new-bucket/test/a/b/objUsage withlakeFSWhen you use Boto to access S3 and want to migrate only a subset of your data to lakeFS, Boto S3 Router allows you to use lakeFS and S3 side-by-side with minimum code changes.Consider the following code, accessing objects in two S3 buckets:importboto3s3=boto3.client('s3')s3.get_object(Bucket="bucket-a",Key="test/object.txt")s3.get_object(Bucket="bucket-b",Key="test/object.txt")Now suppose onlybucket-awas migrated to lakeFS, and that the new repository in lakeFS is calledexample-repo. You can route only requests to this specific bucket to lakeFS, with all other requests still being routed to S3:importboto3importboto_s3_routerass3rs3=boto3.client('s3')lakefs=boto3.client('s3',endpoint_url='https://lakefs.example.com')profiles={"lakefs":{"source_bucket_pattern":"bucket-a","mapped_bucket_name":"example-repo","mapped_prefix":"dev/"}}s3=s3r.client({"lakefs":lakefs,"default":s3},profiles)# All code accessing S3 stays the same as before:s3.get_object(Bucket="bucket-a",Key="test/object.txt")# routes to example-repo (dev branch) in lakeFSs3.get_object(Bucket="bucket-b",Key="test/object.txt")# routes to AWS S3Configurations3 = s3r.client(client_mapping, profiles)As can be seen in the examples above, Boto S3 Router is initialized using two configuration parameters:client_mapping: The mapping between profile names to S3 clients. Adefaultclient is required.For example, ifs3andminioare Boto S3 clients, theclient_mappingcan be:{"profile1":s3,"profile2":minio,"default":s3}profiles-  A mapping between profile name to profile:{"profile1":{"source_bucket_pattern":"example-bucket","source_key_pattern":"a/*","mapped_bucket_name":"new-bucket","mapped_prefix":"test/"},"profile2":{"source_bucket_pattern":"bucket","mapped_bucket_name":"bucket-a"}}A profile can have the following properties:PropertyDescriptionRequiredsource_bucket_patternRequests to buckets matching this pattern will use this profile.Yessource_key_patternRequests to keys matching this pattern will use this profile.Nomapped_bucket_nameThe bucket name to use when routing the request to the destination clientNomapped_prefixAn optional string to prepend to the key when routing the request to the destination clientNoLicenseApache-2.0 License