s3tools/s3cmd — pull request #91 by nicjansma (2012-12-06 01:25:39+00:00)
Title: --delete-after-fetch option

Hi,

I've added an option called `--delete-after-fetch`

One of the scenarios I use `s3cmd` for is fetching all of the latest S3/CloudFront logs files that are being put into one of my buckets.

Previously, I was using a combination of `s3cmd sync` then `s3cmd del` to fetch any new log files, the remove them from the bucket.

However, this creates a race condition where if there are any new log files that were uploading after the beginning of the `s3cmd sync`, the `s3cmd del` will remove the new files without my knowledge.

The option `--delete-after-fetch` attempts to solve this, by issuing a S3 delete command only after a successful fetch of a file. This works for both `s3cmd get` and `s3cmd sync` commands - whenever file is successfully downloaded, it is also removed from the remote S3 bucket.

So instead of `s3cmd sync` followed by a `s3cmd del`, I can now simply do a `s3cmd sync --delete-after-fetch` and know that I won't lose any files.