Package: greas3
Greas3Greas3is a Python package and command line application for uploading files to Amazon Web Services S3 like a greased thing.Rather rely on timestamps to detect changes, Greas3 uses SHA256 checksums to avoid re-uploading identical files. SinceGit does not record file modification timestamps, this makes Greas3 ideal for CI/CD pipelines that pull large files to upload to S3.InstallationGreas3 requires Python 3.9 or later and can be installed fromPyPI.pipinstallgreas3Command line usageTo upload a local file or directory to S3, run:greas3 LOCAL-PATH S3-URIFor example:$greas3./clowns.jpgs3://circus/clowns.jpg

/home/cariad/s3://circus/
clowns.jpg=clowns.jpgTo keep the original filename in S3, provide only the key prefix to upload to:$greas3./clowns.jpgs3://circus/

/home/cariad/s3://circus/
clowns.jpg=clowns.jpgPass a directory instead of a file to recursively upload its contents:$greas3./party-time/s3://circus/inbox/

/home/cariad/party-time/s3://circus/inbox/
good-clowns/steve.jpg=good-clowns/steve.jpg
evil-clowns/jacob.jpg=evil-clowns/jacob.jpg
group-hug.jpg>group-hug.jpg=indicates that a file hasn't changed and so won't be uploaded.>indicates the file will be uploaded.Pass--debugto enable debug logging.Pass--dry-runto view the enqueued uploads without performing them.Python usageTheput()function accepts a path to a local file or directory and a destinationS3 URIto upload to.The function returns a list ofPutOperation, each of which describes the source file, destination URI and flag to indicate if the files are the same; in which case the upload will not be performed.To gather the operations without performing them, pass thedry_run=Trueflag.AuthorisationGreas3 will authenticate into Amazon Web Services using the first set of credentials that the SDK finds.Generally, in order of precedence, credentials are found in:(When using the Python package) A Boto3 session with custom credentials can be passed to functions.TheAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKENenvironment variables.Acredentialsorconfigfile created by the AWS CLI. TheAWS_PROFILEenvironment variable can prescribe any non-default profile.Greas3 requires the following actions:s3:GetObjectAttributess3:PutObjectHow does it work?To check if a file should be uploaded, Greas3 calls the S3GetObjectAttributesAPI to gather the existing object's file size and SHA256 checksum (both the file's checksum and each individual uploaded chunk's checksum).If the file doesn't exist in S3, or if the file size is different from the local file, then the files are considered different.If the file was previously uploaded to S3 in chunks, and if each chunk has a SHA256 checksum, then each chunk's checksum is compared to the local file's equivalent byte range. If the checksums don't match then the files are considered different.If the object was previously uploaded to S3 in a single chunk then its checksum is compared to the local file. If the checksums don't match then the files are considered different.Note that S3 will record checksums only if directed to during upload; other tools might not request this but Greas3 does. If the checksums aren't present then Greas3 will have to perform the upload even if the files are identical; however, subsequent use of Greas3 will use the checksums to perform only uploads that are strictly necessary.Time comparisonsI created a directory containing 10 files with a total of 117 MB, then -- pitching the AWS CLI and Greas3 against each other -- I:Synchronised that directory with an S3 bucket.Synchronised that directory again without making any changes.Synchronised that directory again after touching the local files' modification timestamps.UploadAWS CLI (seconds)Greas3Greas3 vs AWS CLIInitial~51~56~5 seconds slowerNo changes~1~2~1 second slowerTouched modification dates~51~2~49 seconds fasterThe AWS CLI is marginally more performant during the initial upload and when timestamps can be trusted.However, in an environment where timestamps can't be trusted -- for example, in CI/CD pipelines that pull files from Git repositories -- Greas3 is clearly the most performant uploader.SupportPlease submit all your questions, feature requests and bug reports atgithub.com/cariad/greas3/issues. Thank you!LicenceGreas3 isopen-sourceand published under theMIT License.You don't have to give attribution in your project, but -- as a freelance developer with rent to pay -- I appreciate it!The AuthorHello! 👋 I'mCariad Eccleston, and I'm a freelance Amazon Web Services architect, DevOps evangelist, CI/CD deployer and backend developer.You can find me atcariad.earth,github/cariad,linkedin/cariadand on Mastodon at@cariad@tech.lgbt.