Package: pys3nfssync
s3syncOverviewFeaturesRequirementsInstallationUsageFirst run/setupPushPullRate limiting in actionConfigurationInclude/excluse patternsAdvanced ConfigurationPerformance Tests for aws sync commandOverviews3sync.py is a utility created to sync files to/from S3 as a continuously running process, without having to manually take care of managing the sync. It internally uses theaws s3 synccommand to do the sync, and  uses the python modulewatchdogto listen to filesystem events on the monitored path and push changes to S3. For pull, there is no listener implemented, and it does simple interval based pull. Therefore for pull it is recommended to uses3fsinstead  - just mount the s3 bucket on your filesystem.FeaturesRate limiting using the pythontocken-bucketmodule. You can setmax_syncs_per_minutein the config yaml and the file system watcher triggered pushes will be throttled as per that limit.Optional Reporting of runtime stats for the sync operation using thepyformancemoduleAbility to filter byinclude_patternsandexclude_patternsor toexclude_directoriescompletely, or make the filtercase_sensitiveAutomated setup of AWS CLI config by creating a separatenamed profilefor the utility with ability to tune performance by settingmax_concurrent_requests,max_queue_size, etc.Setuptools integration, pythonclickbased command line interfaceRequirementsRequiresAWS CLI version 2to be installed and available in the pathInstallationpip install pys3syncUsages3sync --helpUsage:s3sync.py[OPTIONS]COMMAND[ARGS]...Autilitycreatedtosyncfilesto/fromS3asacontinuouslyrunningprocess,withouthavingtomanuallytakecareofmanagingthesync.Itinternallyusestheawss3synccommandtodothesyncandusespython'swatchdoglistenertogetnotifiedofanychangestothewatchedfolder.

Options:--configPATHPathtotheconfig.yamlfilecontainingconfigurationparamsforthisutility[required]-v,--verbosityLVLEitherCRITICAL,ERROR,WARNING,INFOorDEBUG--helpShowthismessageandexit.

Commands:initInitialsetup.pullOne-waycontinuoussyncfroms3pathtolocalpath(basedonpolling...pushOne-waycontinuoussyncfromlocalpathtos3path(usesafile...s3sync --config config.yaml push --helpUsage:s3sync.pypush[OPTIONS]One-waycontinuoussyncfromlocalpathtos3path(usesafilewatchercalledwatchdog)Options:--s3pathPATHFulls3pathtosyncto/from[required]--localpathPATHLocaldirectorypathwhichyouwanttosync[required]--helpShowthismessageandexit.--urlEndpointurls3sync --config config.yaml push --helpUsage:s3sync.pypull[OPTIONS]One-waycontinuoussyncfroms3pathtolocalpath(basedonpollingonaninterval)Options:--s3pathPATHFulls3pathtosyncto/from[required]--localpathPATHLocaldirectorypathwhichyouwanttosync[required]--intervalINTEGERS3pollingintervalinseconds[required]--helpShowthismessageandexit.--urlEndpointurlFirst run/setups3sync --config-yaml config.yaml initThis utility creates anamed profilefor your AWS CLI so that the parameters required for the S3 cli for the utility are isolated from your regular AWS CLI profile. The first time you nee to run theinitcommand, which will create the named profiles3syncin your local aws config (~/.aws/config), with the parameters configured inconfig.yamland credentials copied from your default AWS credentials file.PushYou run one instance of this utility per localpath<>s3path combination that you want to continuously syncs3sync --config config.yaml -v DEBUG push --s3path s3://<bucket>/<path> --localpath ./ --url https://[endpoint-url]Pulls3sync --config config.yaml -v DEBUG pull --s3path s3://<bucket>/<path> --localpath ./sync --interval 2 --url https://[endpoint-url]Configurationglobal:max_syncs_per_minute:10report_stats:Falsewatcher:include_patterns:exclude_patterns:["*.git/*"]exclude_directories:Falsecase_sensitive:Falses3:max_concurrent_requests:20max_queue_size:2000multipart_threshold:8MBmultipart_chunksize:8MBmax_bandwidth:use_accelerate_endpoint:"false"region:ap-south-1Include/excluse patternsInclude/excluse patterns are implemented usingpathtools.match_any_path, which ultimately supports unix glob pattern syntax. You can test your patterns using the provided scriptpatternhelper.py. These patterns are passed to the watchdog as well as aws cli, which also uses thesame syntax. Both properties accept a list of patterns.Advanced ConfigurationPlease change these values carefully. They depend on your machine and your internet connection. Read more about improving s3 sync transfer speedsheremax_concurrent_requestsPassed through to your~/.aws/configviaaws configure set default.s3.max_concurrent_requestscommand. Read about the parameterheremax_queue_sizePassed through to your~/.aws/configviaaws configure set default.s3.max_queue_sizecommand. Read about the parameterheremultipart_thresholdPassed through to your~/.aws/configviaaws configure set default.s3.multipart_thresholdcommand. Read about the parameterheremultipart_chunksizePassed through to your~/.aws/configviaaws configure set default.s3.multipart_chunksizecommand. Read about the parameterheremax_bandwidthPassed through to your~/.aws/configviaaws configure set default.s3.max_bandwidthcommand. Read about the parameterhereuse_accelerate_endpointPassed through to your~/.aws/configviaaws configure set default.s3.use_accelerate_endpointcommand. Read about the parameterhere