Package: s3synchrony
S3SynchronyCreated by Sevan Brodjian for Ameren at the Innovation Center @ UIUCThis package provides a service for synchronizing file creations, deletions, and modifications across users on an AWS S3 prefix. Support also exists for easily expanding to other database systems.InstallationUse the package managerpipto install s3synchrony.pipinstalls3synchronyRequirementsS3Synchrony relies on Python 3 and the following packages to operate:hashlibdatetimepandasboto3botocorepyperclipUsageS3Synchrony comes with three primary functions, which can be called as follows:imports3synchronyass3s# returns a list of data platforms currently supporteds3s.get_supported_platforms()# prompts user to synchronize all detected changes in the local and remote repositoriess3s.smart_sync(platform="S3",aws_bkt="bucket_name",aws_prfx="prfx_path")# prompts user to remove all synchronization support on the local and remote repositoriess3s.reset_all(platform="S3",aws_bkt="bucket_name",aws_prfx="prfx_path")The Data FolderWhen using S3Synchrony, you are synchronizing all of the data stored in a local directory with the data stored in an S3 directory. The S3 directory is referenced through both an AWS bucket, an AWS prefix, and the necessary credentials to access said prefix. The local directory to be used can be a relative or full path, and by default will be a subdirectory named "Data" stored in the same working directory.Project FolderDatacode, etc.smart_syncdefsmart_sync(platform="S3",**kwargs):"""Perform all necessary steps to synchronize a local repository   with a remote repo.Notes:Keyword arguments are dependent on platform selection."""if(platformin_supported_platforms):connection=_supported_platforms[platform](**kwargs)else:connection=baseconn.DataPlatformConnection(**kwargs)connection.intro_message()connection.establish_connection()connection.synchronize()connection.close_message()The smart_sync function is the premier work of this package, and will perform all of the data synchronization for you. This function will check the passed platform name, and reference a self-contained list of supported platforms to instantiate the proper class. This list of supported platforms can be accessed via a call to get_supported_platforms().Each connection type will require a different set of keyword arguments. For S3, the minimum arguments are "aws_bkt" and "aws_prfx". Please check the class docstrings for each connection type for more information.All platform classes should be children of the DataPlatformConnection class which is an interface will all necessary public functions. For S3, a folder named .S3 will be created within your data folder. This .S3 folder will contain CSVs used for monitoring data changes and text files for storing small bits of information.versions.csv:Contains the state of data stored on s3versionsLocal.csv:Contains the state of data stored locallydeletedS3.csv:Contains all files deleted from S3deletedLocal.csv:Contains all files deleted locallyignores3.txt:Contains a list of file paths to be ignored entirelyuser_name.txt:Contains the name attached to your file modificationsaws.txt:Contains credentials used to access the AWS prefixUsing these CSVs, S3Synchrony can determine what files you have newly created, deleted, and modified. It will then prompt you to upload these changes to S3. Once you have done so, it will upload new CSVs as needed. After downloading these new CSVs, your collaborative peers will be prompted to download your own changes as well as upload their own.In addition, a tmp folder will be utilised within the .S3 folder. This tmp folder contains downloaded files from S3 that are used to compute certain CSVs.DeletionsWhen deleting files, the user will be prompted to confirm their deletions. Files that are deleted locally will simply be removed. Files deleted from S3, however, will simply be moved into a "deleted" subfolder of the .S3 folder on S3.LogsWhen there are any issues with a file being uploaded or downloaded, an error message will be printed and that file will be skipped. A log will then be created a saved locally inside of the "logs" subfolder of the local .S3 folder.reset_alldefreset_all(platform="S3",**kwargs):"""Reset local and remote directories to original state.Notes:Keyword arguments are dependent on platform selection."""if(platformin_supported_platforms):connection=_supported_platforms[platform](**kwargs)else:connection=baseconn.DataPlatformConnection(**kwargs)connection.intro_message()connection.establish_connection()ifconnection.reset_confirm():connection.reset_local()connection.reset_remote()connection.close_message()Resetting all S3Synchrony services is as simple as deleting the .S3 folders contained locally and on S3. Once these are deleted, synchronization cannot occur until they are recreated, which can be done by simply making a new call to S3Synchrony.Before resetting, however, a call to reset_confirmmustoccur. The user will then be prompted to confirm that they would like their .S3 folders removed.LicenseGNU GPLv3