Package: s3copy
s3copy======**s3copy** - multi-threaded, fault-tolerant, bucket-to-bucket copy fors3Usage-----Simple example:::s3copy s3://source/path s3://dest/pathAll the files in the source bucket contained in path are copied to thedestination bucket at the specified path (paths may be omitted foreither source or destination).Some common arguments:::s3copy s3://source/path s3://dest/path -nThe above example is a dry-run (-n), so it does no copying but indicateswhat copying it would do.::s3copy s3://source/path s3://dest/path -t 20 -l DEBUG -L log.txtThe above example uses 20 threads, prints log messages down to the DEBUGlevel, and appends its log to a file named log.txt.::s3copy --helpThe above example prints help.Source and Destination Paths----------------------------Paths behave similarly to the unix "cp -r" command. Suppose for examplethat the source bucket contains the following files:::folder-001/test-001folder-001/test-002and suppose in each case below that the destination folder is initiallyempty. Then these commands have these results:+------------------------------------------+-------------------------------------------------+| Command                                  | Result files in dest                            |+==========================================+=================================================+| s3copy s3://source s3://dest             | folder-001/test-001 folder-001/test-002         |+------------------------------------------+-------------------------------------------------+| s3copy s3://source s3://dest/foo         | foo/folder-001/test-001 foo/folder-001/test-002 |+------------------------------------------+-------------------------------------------------+| s3copy s3://source/folder-001 s3://dest  | folder-001/test-001 folder-001/test-002         |+------------------------------------------+-------------------------------------------------+|s3copy s3://source/folder-001/* s3://dest | test-001 test-002                               |+------------------------------------------+-------------------------------------------------+Globbing support is limited to a single ``*`` at the end of a pattern.Internal ``*`` and other pattern characters are not supported.Restart and Sync----------------s3copy is fault-tolerant and restartable. If a copy run is interruptedor fails for any reason, you can repeat the command and it will completethe operation. It does this by checking the md5s of the individualsource and destination files. In this way, s3copy can also be used tosync buckets. If an s3copy command succeeeds and the source bucketsubsequently changes, with the addition of new files or changes toexisting files, the same command can be run again and only the changeswill be copied. Note that additions and changes are supported butdeletions are not: if a file is deleted from the source and the commandis rerun, the corresponding file in the destination will not deleted.Multipart File Handling-----------------------Amazon S3 can currently store individual files up to a limit of 5terabytes (http://aws.amazon.com/s3/faqs/), however files above 5GB mustbe stored as "multipart" files. Multipart files are created and copiedin multiple transactions, using special apis that specify a byte offsetand length.Multipart files present special difficulties for copying:-  S3 does not expose the md5 of the entire file, nor of its constituentparts. It does report the md5 of a part as it is copied. It alsoexposes an "etag" of the file, which resembles an md5 hash (discussedfurther below).-  S3 does not permit creating a simple, non-multipart file directlyfrom a portion of a multipart file. That is, when copying a portion,both the source and target must be multipart.In addition, although files larger than 5GB *must* be multipart, filessmaller than 5GB *may* be. For example, it is faster and more reliableto create a 5GB file (the maximum non-multipart file) in 80 chunks of64MB each, rather than as a single 5GB file. For these reasons, s3copyhandles multipart files bigger and smaller than 5GB.Multipart files are copied in parts. The default part size is 64MB, butthis can be controlled by the ``-p`` option. For validation andrestartability, the part files are copied in three steps:1. Each part is copied to a multipart file with a single part in a temparea on the destination bucket. As noted above, one can not create asingle part file directly from a part of a multipart file. As thiscopy occurs, s3copy keeps the md5 of the part in memory (since, asnoted above, S3 does not expose the md5 of a multipart file, even ifit has only one part).2. Each part is then copied to a temporary simple file. S3 does exposethe md5 of this file. s3copy validates that the md5 of the filematches the md5 of the portion which S3 reported in step 1.3. The final target file is created as a multipart file from theconstituent simple files.This process means that the data bytes of a multipart file are copiedthree times, whereas in principle they could have been copied only once,as a multipart copy of the source to the target. The extra copies enablerestartability and validation, as follows:-  Restartability: steps one and two create durable temp files whichs3copy can observe to determine which parts of a multipart copy havealready occurred.-  Validation: During the multipart copy in step three, the md5s of eachcomponent can be revalidated against the stored md5s of the parts.It should be noted that only step one involves data transfer between thesource and destination buckets. Steps two and three are copies from thedestination bucket to itself, and are thus inside the same S3 datacenter. Data transfer inside a data center is very fast and is(currently) free. Amazon only charges for data transfer between datacenters, and this methodology does the minimum cross-data-centercopying. The author's experience is that the restartability justifiesthe longer throughput time.By default, the temporary files are created in a directory named "temp",named as follows. For each multipart source file, for example, named"source/path/to/file-to-split", we use a directory named"dest/temp/path/to/file-to-split//parts". The versionid is the first 7characters of the "etag" of the source multi-part file. Inside thatdirectory, the temporary 1-part-multipart files are named "temp-00001","temp-00002", etc., and the single-part files are named "part-00001","part-00002", etc. In some situations, such as subsequent transfer tohdfs, retaining the part files may be useful. By default, all thesefiles are retained.Security and Access-------------------If needed, you can supply two credentials sets to s3copy: one foraccessing the source bucket and copying to the destination and one forlisting the destination bucket. This is useful for cross-account copyingwhen you are given a set of credentials to access the source, but thosecredentials can not list the destination. Supply a second set ofcredentials by using the ``-d`` option.You can grant access to files created on the destination by specifyingthe ``--acl-grant`` option. The grantee must be an email address.Testing Notes------------To run the tests, edit the s3copy_test_settings.py for your local environment, then run this command:python test_s3copyOn-line Help------------This is the current built-in on-line help:::s3copy --helpusage: s3copy [-h] [-n] [-f FILE [FILE ...]] [-p PREFIX [PREFIX ...]][-F FILES] [-P PREFIXES] [-a AWS_ACCESS_KEY] [-k AWS_SECRET_KEY][-c S3CFG_FILE] [-d DEST_S3CFG_FILE] [--acl-grant ACL_GRANT][-t NUM_THREADS] [-l LOG_LEVEL] [-L LOG_DEST]source_bucket [dest_bucket]Multithreaded multipart copier for Amazon S3positional arguments:source_bucket         source bucket/pathdest_bucket           destination bucket/pathoptional arguments:-h, --help            show this help message and exit-n, --dry-run         do no work but report what work would be done-f FILE [FILE ...], --file FILE [FILE ...]source file[s] to copy-p PREFIX [PREFIX ...], --prefix PREFIX [PREFIX ...]source prefix[es] to copy-F FILES, --files FILESfile containing a list of files to copy-P PREFIXES, --prefixes PREFIXESfile containing a list of prefixes to copy-a AWS_ACCESS_KEY     AWS Access Key-k AWS_SECRET_KEY     AWS Secret Key-c S3CFG_FILE, --config_file S3CFG_FILEs3cmd-format config file-d DEST_S3CFG_FILE, --dest-config DEST_S3CFG_FILEs3cmd-format config file for destination bucket only--acl-grant ACL_GRANTacl to grant as PERMISSION:EMAIL-t NUM_THREADS        number of threads (default: 40)-l LOG_LEVEL          logging level-L LOG_DEST           logging file (appended)