Package: b-cfn-s3-large-deployment
B.CfnS3LargeDeploymentb-cfn-s3-large-deployment- AWS CDK custom resource that handles large files deployment to S3 bucket.DescriptionThis custom resource deploys local files or S3 bucket objects to a destination bucket retaining their file-system
hierarchy.Two types of deployment sources are available:BucketDeploymentSource- uses another S3 bucket object(-s) as source for the deployment to a destination bucket.
Only files up to 5TB are supported due to S3 bucket limitations;AssetDeploymentSource- usesaws-cdk.aws-s3-assetslib to deploy local files as .zip files to assets bucket from
which extracted contents are moved to the destination bucket. Asset files more than 2GB in size are not supported.See "Known limits" sections below for more information on this resource limitations.This resource implementation is based on GitHub pull-requesthttps://github.com/aws/aws-cdk/pull/15220.RemarksBiomapasaims to modernise life-science
industry by sharing its IT knowledge with other companies and
the community.Related technologyPython >= 3.8Amazon Web Services (AWS)AssumptionsThe project assumes that the person working with it have basic knowledge in python
programming.Useful sourcesSee code documentation for any additional sources and references. Also seeaws-cdk.s3-deploymentlibrary for more
information as this implementation is based on work done there.InstallUse the package manager pip to install this package. This project is not in the PyPi
repository yet. Install directly from source or PyPI.pipinstall.Orpipinstallb-cfn-s3-large-deploymentUsage & ExamplesThis AWS CloudFormation custom resource is used pretty much the same way as any other resource. Simply initialize it
within any valid CDK scope giving it unique name/id, providing source(-s) and the destination for the deployment.The deployment of files depends on AWS Lambda's/tmpdirectory and its size limits. For large files/tmpdirectory
size can be configured using Ephemeral storage (DeploymentProps.ephemeral_storage_size) supported by AWS Lambda
functions.Optionally, if there's a need for even larger files deployment than what AWS Lambda's/tmpdirectory supports,
setting theDeploymentPops.use_efsandDeploymentPops.efs_propsfields, AWS Elastic File Storage (EFS) can be
enabled to allow such files handling.A simple example ofS3LargeDeploymentResourceusage is shown below:fromaws_cdk.coreimportApp,Stack,Constructfromaws_cdk.aws_s3importBucketfromb_cfn_s3_large_deployment.resourceimportS3LargeDeploymentResourcefromb_cfn_s3_large_deployment.deployment_propsimportDeploymentPropsfromb_cfn_s3_large_deployment.deployment_sourceimportAssetDeploymentSource,BucketDeploymentSourceclassExampleStack(Stack):def__init__(self,scope:Construct):super().__init__(...)S3LargeDeploymentResource(scope=self,name='ExampleLargeDeployment',sources=[AssetDeploymentSource(path='/path/to/your/local/directory'),AssetDeploymentSource(path='/path/to/your/local/zip/file.zip'),BucketDeploymentSource(bucket=...,zip_object_key='your-source-bucket-object-key'),...],destination_bucket=Bucket(...),props=DeploymentProps(...))...app=App()ExampleStack(app,'ExampleStack')app.synth()Here, three types of supported sources were used:whole, local directory given as a path, which is then deployed to the assets bucket as a .zip object:AssetDeploymentSource(path='/path/to/your/local/directory')single .zip file given as a path, which is then deployed to the assets bucket:AssetDeploymentSource(path='/path/to/your/local/zip/file.zip')Single .zip S3 object found in the source bucket, given as an object key. No further pre-processing is
applied in this case:BucketDeploymentSource(bucket=...,zip_object_key='your-source-bucket-object-key')In all of these cases, final, source .zip objects are extracted insideS3LargeDeploymentResource's handler
function storage and the available contents are then deployed to the configured destination. This is all done, while
maintaining original file structure of source contents.Known limitsaws_cdk.aws_s3_assets.Assetsupports up to 2GB/asset (limited by NodeJS implementation).S3 bucket supports up to 5TB objects.ContributionFound a bug? Want to add or suggest a new feature? Contributions of any kind are gladly
welcome. Contact us, create a pull-request or an issue ticket.Release history1.2.0Add Ephemeral storage support.1.1.3Fixed dependencies bug betweenS3LargeDeploymentResourceand its handler function.Upgraded testing pipeline to useB.AwsCdkParallel.