Package: s3s3
s3s3 0.1.10
A microservice to move files from S3 APIs (Swift or Ceph) to other S3 APIs.To install usepip install s3s3s3s3s3s3 is a microservice to move files from one S3 compliant service to another (Swift, Ceph, AWS).s3s3 has two services.s3s3.scripts.listensubscribes and listens to a redis pubsub channel waiting for s3 object names. When notified it callss3s3.api.uploadand uploads that S3 object from the source to destination(s).s3s3.scripts.bucketuploads all keys in one s3 bucket to another.s3s3.scripts.listenis meant to be a real-time daemon.s3s3.scripts.bucketis meant to run periodically as a cron job. Anything missed bys3s3.scripts.listenwill be handled by this service.Summarys3s3 is required because Ceph and Swift are not feature complete with AWS S3. Many available libraries that work well with AWS S3 do not work with Ceph and Swift.Some examples:key.sizeandkey.md5do not work with Ceph S3 without fetching the contents of the key (or s3 object). Multipart uploads are not reliable with Ceph S3. V4 signatures are not supported by Ceph S3.Configurations3s3 must be configured. Configuration can be found in thes3s3.configmodule. A source, destination and pubsub section are required. Multiple destination connections are supported but there was minimal testing. To signify a section is a destination connection it must start with'dest'.Example template:https://github.com/jmatt/s3s3/blob/master/s3s3/s3s3.ini.dist[source]aws_access_key_id={YOUR_AWS_ACCESS_KEY_ID}aws_secret_access_key={YOUR_AWS_SECRET_ACCESS_KEY}bucket_name={YOUR_S3_BUCKET}host={YOUR_CEPH_S3_ENDPOINT}verify_md5=False# Verify md5 during s3 operations.is_secure=True# Optionalcalling_format=OrdinaryCallingFormat# Optional[destination]aws_access_key_id={YOUR_AWS_ACCESS_KEY_ID}aws_secret_access_key={YOUR_AWS_SECRET_ACCESS_KEY}bucket_name={YOUR_S3_BUCKET}verify_md5=False# Verify md5 during s3 operations.is_secure=True# OptionalInstalls3s3 requires python3 and redis. It was tested with python3.4 and python3.5. And redis 3.x.pipinstalls3s3ClientThere are two clients. One for each service.s3s3.client.ListenClientis a client to listen to the'backup'redis pubsub channel and calls3s3.client.on_notify.s3s3.client.BucketClientis a client that uses the configuration to provide access to the duplicate_bucket API function.Command LineBoth clients are available as command line scripts.s3s3listen--config/path/to/s3s3.iniThis will use the configuration to build source and destination boto connections, connect to redis and start listening on thebackupchannel. Any messages pushed to that channel will be considered source s3 key names and will attempt to be uploaded to the destination connection(s).s3s3bucket--config/path/pto/s3s3.iniThis will use the configuration to build source and destination boto connections and duplicate the source bucket in the destination bucket.APIThe API can be found ins3s3.apimodule.defcreate_connection(connection_args):Creates a boto connection from theconnection_argsdictionary.defupload(source_key,dest_key,verify_md5=False):Upload the source key (S3 object) to the destination key. Ifverify_md5is true then verify md5s match.defduplicate_bucket(source_bucket,dest_bucket,verify_md5=False):Duplicate the source bucket to the destination bucket. Ifverify_md5is true then verify md5s match. If the md5 is not available compute it and verify it matches.Deploys3s3 requires redis, python3 and supervisord.mkdir-p/opt/envcd/opt/envvirtualenv-ppython3s3s3./opt/env/s3s3/bin/activatepipinstalls3s3echo_s3s3_supervisord_conf>/etc/supervisor/conf.d/s3s3.confecho_s3s3_ini_template>/usr/local/etc/s3s3.ini# Update ini file with your source and destination s3 information.servicesupervisorrestart# or... start if it's not running.LICENSESee theLICENSE file.For more information, please see:https://github.com/lsst-sqre/s3s3