Package: pyss3sync
s3syncOverviewFeaturesRequirementsInstallationUsageFirst run/setupPushPullRate limiting in actionConfigurationInclude/excluse patternsAdvanced ConfigurationPerformance Tests for aws sync commandOverviews3sync.py is a utility created to sync files to/from S3 as a continuously running process, without having to manually take care of managing the sync. It internally uses theaws s3 synccommand to do the sync, and  uses the python modulewatchdogto listen to filesystem events on the monitored path and push changes to S3. For pull, there is no listener implemented, and it does simple interval based pull. Therefore for pull it is recommended to uses3fsinstead  - just mount the s3 bucket on your filesystem.FeaturesRate limiting using the pythontocken-bucketmodule. You can setmax_syncs_per_minutein the config yaml and the file system watcher triggered pushes will be throttled as per that limit.Optional Reporting of runtime stats for the sync operation using thepyformancemoduleAbility to filter byinclude_patternsandexclude_patternsor toexclude_directoriescompletely, or make the filtercase_sensitiveAutomated setup of AWS CLI config by creating a separatenamed profilefor the utility with ability to tune performance by settingmax_concurrent_requests,max_queue_size, etc.Setuptools integration, pythonclickbased command line interfaceRequirementsRequiresAWS CLI version 2to be installed and available in the pathInstallationpip install pys3syncUsages3sync --helpUsage:s3sync.py[OPTIONS]COMMAND[ARGS]...Autilitycreatedtosyncfilesto/fromS3asacontinuouslyrunningprocess,withouthavingtomanuallytakecareofmanagingthesync.Itinternallyusestheawss3synccommandtodothesyncandusespython'swatchdoglistenertogetnotifiedofanychangestothewatchedfolder.

Options:--configPATHPathtotheconfig.yamlfilecontainingconfigurationparamsforthisutility[required]-v,--verbosityLVLEitherCRITICAL,ERROR,WARNING,INFOorDEBUG--helpShowthismessageandexit.

Commands:initInitialsetup.pullOne-waycontinuoussyncfroms3pathtolocalpath(basedonpolling...pushOne-waycontinuoussyncfromlocalpathtos3path(usesafile...s3sync --config config.yaml push --helpUsage:s3sync.pypush[OPTIONS]One-waycontinuoussyncfromlocalpathtos3path(usesafilewatchercalledwatchdog)Options:--s3pathPATHFulls3pathtosyncto/from[required]--localpathPATHLocaldirectorypathwhichyouwanttosync[required]--helpShowthismessageandexit.s3sync --config config.yaml push --helpUsage:s3sync.pypull[OPTIONS]One-waycontinuoussyncfroms3pathtolocalpath(basedonpollingonaninterval)Options:--s3pathPATHFulls3pathtosyncto/from[required]--localpathPATHLocaldirectorypathwhichyouwanttosync[required]--intervalINTEGERS3pollingintervalinseconds[required]--helpShowthismessageandexit.First run/setups3sync --config-yaml config.yaml initThis utility creates anamed profilefor your AWS CLI so that the parameters required for the S3 cli for the utility are isolated from your regular AWS CLI profile. The first time you nee to run theinitcommand, which will create the named profiles3syncin your local aws config (~/.aws/config), with the parameters configured inconfig.yamland credentials copied from your default AWS credentials file.PushYou run one instance of this utility per localpath<>s3path combination that you want to continuously syncs3sync --config config.yaml -v DEBUG push --s3path s3://<bucket>/<path> --localpath ./Pulls3sync --config config.yaml -v DEBUG pull --s3path s3://<bucket>/<path> --localpath ./sync --interval 2Rate limiting in actionConfigurationglobal:max_syncs_per_minute:10report_stats:Falsewatcher:include_patterns:exclude_patterns:["*.git/*"]exclude_directories:Falsecase_sensitive:Falses3:max_concurrent_requests:20max_queue_size:2000multipart_threshold:8MBmultipart_chunksize:8MBmax_bandwidth:use_accelerate_endpoint:"false"region:ap-south-1Include/excluse patternsInclude/excluse patterns are implemented usingpathtools.match_any_path, which ultimately supports unix glob pattern syntax. You can test your patterns using the provided scriptpatternhelper.py. These patterns are passed to the watchdog as well as aws cli, which also uses thesame syntax. Both properties accept a list of patterns.Advanced ConfigurationPlease change these values carefully. They depend on your machine and your internet connection. Read more about improving s3 sync transfer speedsheremax_concurrent_requestsPassed through to your~/.aws/configviaaws configure set default.s3.max_concurrent_requestscommand. Read about the parameterheremax_queue_sizePassed through to your~/.aws/configviaaws configure set default.s3.max_queue_sizecommand. Read about the parameterheremultipart_thresholdPassed through to your~/.aws/configviaaws configure set default.s3.multipart_thresholdcommand. Read about the parameterheremultipart_chunksizePassed through to your~/.aws/configviaaws configure set default.s3.multipart_chunksizecommand. Read about the parameterheremax_bandwidthPassed through to your~/.aws/configviaaws configure set default.s3.max_bandwidthcommand. Read about the parameterhereuse_accelerate_endpointPassed through to your~/.aws/configviaaws configure set default.s3.use_accelerate_endpointcommand. Read about the parameterherePerformance Tests for aws sync commandEnvironmentNetwork: home/BLR (Airtel 1 Gbps Xtreme Fiber)WiFi: 5 GHZ, RSS: -38 dbM, Tx rate: 1300 Mbps (802.11 ac)Upload speed to s3: 18 MB/sDownload speed from s3: 15 MB/sNumber of threads for s3 sync command: 10 (default)Sync from local to S3 (upload)Test artifact 1:https://github.com/apache/stormSize: 224MNumber of files: 3571Test 1 (full sync):timeawss3sync--storage-classREDUCED_REDUNDANCY./s3://psm-poc-dmp-temp/codesync
real0m45.543s
user0m14.755s
sys0m3.685sTest 2 (added 39 files, 168k):cp-rf../intSDK.timeawss3sync--storage-classREDUCED_REDUNDANCY./s3://psm-poc-dmp-temp/codesync

real0m3.141s
user0m1.887s
sys0m0.405sTest 3 (removed 398 files, 2.1M):rm-rfexamples/timeawss3sync--storage-classREDUCED_REDUNDANCY--delete./s3://psm-poc-dmp-temp/codesync

real0m3.436s
user0m2.276s
sys0m0.406sTest 4 (change timestamp of single file):touchREADME.markdowntimeawss3sync--storage-classREDUCED_REDUNDANCY--delete--exact-timestamps./s3://psm-poc-dmp-temp/codesync

real0m2.602s
user0m1.492s
sys0m0.296sTest 5 (no change):timeawss3sync--storage-classREDUCED_REDUNDANCY--delete--exact-timestamps./s3://psm-poc-dmp-temp/codesync

real0m2.442s
user0m1.469s
sys0m0.294sTest artifact 2: Bunch of PNGsSize: 400MNumber of files: 577Test 1 (full sync)timeawss3sync--storage-classREDUCED_REDUNDANCY--delete--exact-timestamps./s3://psm-poc-dmp-temp/codesync

real0m22.015s
user0m5.972s
sys0m2.516sSync from S3 to local (download)Test artifact 1:https://github.com/apache/stormSize: 224MNumber of files: 3571Test 1 (full sync)timeawss3sync--storage-classREDUCED_REDUNDANCY--delete--exact-timestampss3://psm-poc-dmp-temp/codesync./

real0m26.448s
user0m14.544s
sys0m3.794sTest artifact 2: Bunch of PNGsSize: 400MNumber of files: 577Test 1 (full sync)timeawss3sync--storage-classREDUCED_REDUNDANCY--delete--exact-timestampss3://psm-poc-dmp-temp/codesync./

real0m29.268s
user0m6.131s
sys0m2.855s