Package: s3peat
s3peat is a Python module to help upload directories to S3 using parallel
threads.The source is hosted athttp://github.com/shakefu/s3peat/.Installings3peat can be installed from PyPI to get the latest release. If you’d like
development code, you can check out the git repo.# Install from PyPI$pipinstalls3peat# Install from GitHub$gitclonehttp://github.com/shakefu/s3peat.git$cds3peat$pythonsetup.pyinstallCommand line usageWhen installed viapiporpython setup.py install, a command calleds3peatwill be added. This command can be used to upload files easily.$ s3peat --help
usage: s3peat [--prefix] --bucket [--key] [--secret] [--concurrency]
      [--exclude] [--include] [--dry-run] [--verbose] [--version]
      [--help] directory

positional arguments:
  directory            directory to be uploaded

optional arguments:
  --prefix , -p        s3 key prefix
  --bucket , -b        s3 bucket name
  --key , -k           AWS key id
  --secret , -s        AWS secret
  --concurrency , -c   number of threads to use
  --exclude , -e       exclusion regex
  --include , -i       inclusion regex
  --private, -r        do not set ACL public
  --dry-run, -d        print files matched and exit, do not upload
  --verbose, -v        increase verbosity (-vvv means more verbose)
  --version            show program's version number and exit
  --help               display this help and exitExample:$s3peat-bmy/bucket-pmy/s3/key/prefix-kKEY-sSECRETmy-dir/ConfiguringThis library is based aroundboto. YourAWS
Access Key IdandAWS Secret Access Keydo not have to be passed on the
command line - they may be configured using any method that boto supports,
including environment variables and the~/.botoconfig..Example using environment variables:$exportAWS_ACCESS_KEY_ID=ABCDEFabcdef01234567$exportAWS_SECRET_ACCESS_KEY=ABCDEFabcdef0123456789ABCDEFabcdef012345$s3peat-bmy/bucket-ps3/prefix-c25some_dir/Example ``~/.boto`` config:# File: ~/.boto
[Credentials]
aws_access_key_id = ABCDEFabcdef01234567
aws_secret_access_key = ABCDEFabcdef0123456789ABCDEFabcdef012345Including and excluding filesUsing the--includeand--exclude(-ior-e) parameters, you
can specify regex patterns to include or exclude from the list of files to be
uploaded.These regexes are Python regexes, as applied byre.search(), so if you want
to match the beginning or end of a filename (including the directory), make
sure to use the^or$metacharacters.These parameters can be specified multiple times, for example:# Upload all .txt and .py files, excluding the test directory$s3peat-bmy-bucket-i'.txt$'-i'.py$'-e'^test/'.Doing a Dry-runIf you’re unsure what exactly is in the directory to be uploaded, you can do a
dry run with the--dry-runor-doption.By default, dry runs only output the number of files found and an error message
if it cannot connect to the specified S3 bucket. As you increase verbosity,
more information will be output. See below for examples.$s3peat-bmy-bucket.-e'\.git'--dry-run21filesfound.$s3peat-bfoo.-e'\.git'--dry-run21filesfound.ErrorconnectingtoS3bucket'foo'.$s3peat-bmy-bucket.-e'\.git'--dry-run-v21filesfound.ConnectedtoS3bucket'my-bucket'OK.$s3peat-bfoo.-e'\.git'--dry-run-v21filesfound.ErrorconnectingtoS3bucket'foo'.S3ResponseError:403Forbidden$s3peat-bmy-bucket.-i'rst$|py$|LICENSE'--dry-run5filesfound.$s3peat-bmy-bucket.-i'rst$|py$|LICENSE'--dry-run-vvFindingfilesin/home/s3peat/github.com/s3peat..../LICENSE./README.rst./setup.py./s3peat/__init__.py./s3peat/scripts.py5filesfound.ConnectedtoS3bucket'my-bucket'OK.Concurrencys3peat is designed to upload to S3 with high concurrency. The only limits are
the speed of your uplink and the GIL. Python is limited in the number of
threads that will run concurrently on a single core.Typically, it seems that more than 50 threads do not add anything to the upload
speed, but your experiences may differ based on your network and CPU speeds.If you want to try to tune your concurrency for your platfrom, I suggest using
thetimecommand.Example:$times3peat-bmy-bucket-pmy/key/--concurrency50my-dir/271/271filesuploadedreal0m2.909suser0m0.488ssys0m0.114sPython APIThe Python API has inline documentation, which should be good. If there’s
questions, you can open a github issue. Here’s an example anyway.Example:froms3peatimportS3Bucket,sync_to_s3# Create a S3Bucket instance, which is used to create connections to S3bucket=S3Bucket('my-bucket',AWS_KEY,AWS_SECRET)# Call the sync_to_s3 methodfailures=sync_to_s3(directory='my/directory',prefix='my/key',bucket=bucket,concurrency=50)# A list of filenames will be returned if there were failures in uploadingifnotfailures:print"No failures"else:print"Failed:",failuresChangelog1.0.0Drops support for Python 2.x (use version<1if you need to continue to use
Python 2.)Adds support for Python 3.x (tested against 3.4, should work for any later
version as well).Thanks go toaboutaaronfor the Python 3
support.0.5.1Use posixpath.sep for upload keys. Thanks tokevinschaul.Released February 4th, 2015.0.5.0Make attaching signal handlers optional. Thanks tokevinschaul.Released December 1st, 2014.0.4.7Better support for Windows. Thanks tokevinschaul.Released November 20th, 2014.Contributorsshakefu- Creator, maintainerkevinschaul