s3tools/s3cmd — pull request #282 by vamitrou (2014-02-18 11:15:13+00:00)
Title: batch_mode in bucket deletion to delete contents without getting the full list

- Introducing batch_mode in bucket deletion to delete contents without getting the full list.

The problem is that I have a bucket with a lot of keys (about a million) and when trying to delete it s3cmd needs to get the full list of keys before it starts (which is very very slow and makes the operation impossible).

I applied some small change that tells it not to recursively get all the batches (each list request to S3 brings maximum of 1000 keys) before it starts deleting.

The flow is the following:
1) delete bucket
2) bucket not empty
3) get the listing of the first max(all_keys_count, 1000)
4) delete them
5) goto 1

Something similar can be applied for other operations like LIST and GET. If you are happy with this change and the idea of converting other operations I could take care of it.

Please let me have your comments.

Congratulations for your great work.