Package: cloud-mappings
cloud-mappingsMutableMapping implementations for common cloud storage providers - easily store things in the cloud through a simple dictionary interface!For nowAzure Blob Storage,Azure Table Storage,Google Cloud Storage, andAWS S3are implemented. Contributions of new providers are welcome.Use CasesEasily read/write to a bucket without needing to learn boto3 (same for Azure, GCP)Ensure consistent serialisation/deserialisation with a simple interfaceSafely use a cloud storage concurrently, for example multiple data scientists working in notebooks or multiple nodes in a distributed compute clusterBuilt a robust multi-layer data caching solution by combining withzictInstallationwith pip:pip install cloud-mappingsBy default,cloud-mappingsdoesn't install any of the required storage providers dependencies, allowing you to version them as you see fit. If you would like to install the versions withcloud-mappingsyou may run any combination of:pip install cloud-mappings[azureblob,azuretable,gcpstorage,awss3]UsageUse it just like a standard python dictionary! (One you would create withdict()or{}). Under the hood each write to the dictionary is serialised and saved to the cloud, and each read is downloaded and deserialised. Iterating keys, values and items all work as expected.cm["key"]=1000cm["key"]# returns 1000delcm["key"]"key"incm# returns falsefork,vincm.items():print(k,v)# prints: key 1000InitialisationAzureBlobStorage:fromazure.identityimportAzureDefaultCredentialfromcloudmappingsimportAzureBlobStoragecm=AzureBlobStorage(account_url="BLOB_ACCOUNT_URL",container_name="CONTAINER_NAME",credential=AzureDefaultCredential(),).create_mapping()AzureTableStorage:fromazure.identityimportAzureDefaultCredentialfromcloudmappingsimportAzureTableStoragecm=AzureTableStorage(table_name="TABLE_NAME",endpoint="AZURE_TABLE_ENDPOINT",credential=AzureDefaultCredential(),).create_mapping()Note that Azure Table Storage has a 1MB size limit per entity.GoogleCloudStorage:fromcloudmappingsimportGoogleCloudStoragecm=GoogleCloudStorage(bucket_name="BUCKET_NAME",project="GCP_PROJECT",).create_mappings()AWSS3Storage:fromcloudmappingsimportAWSS3Storagecm=AWSS3Storage(bucket_name="AWS_BUCKET_NAME",silence_warning=False,).create_mapping()Note that AWS S3 does not support server-side atomic requests, so it is not recommended for concurrent use. A warning is printed out by default but may be silenced by passingsilence_warning=True.API DocsCloudStorage classACloudStorageobject is the entrypoint for this library. You create one but instantiating one for the cloud storage provider you wish to use, currentlyAWSS3Storage,AzureBlobStorage,AzureTableStorage,GoogleCloudStorage. The parameters vary for each, and map to the details required for locating and authenticating the cloud resource they represent. A simple example for each is provided above. From aCloudStorageinstance, (multiple)CloudMapping[T]s may be created by calling.create_mapping():CloudStorage.create_mapping(sync_initially:bool=True,read_blindly:bool=False,read_blindly_error:bool=False,read_blindly_default:Any=None,serialisation:CloudMappingSerialisation[T]=pickle(),key_prefix:Optional[str]=None,)->CloudMapping[T]:Parameters:sync_initially: bool = TrueWhether to callsync_with_cloudinitiallyread_blindly: bool = FalseWhether theCloudMappingwill read from the cloud without synchronising.Whenread_blindly=False, aCloudMappingwill raise aKeyErrorunless a key has been previously written using the sameCloudMappinginstance, or.sync_with_cloudhas been called and the key was in the cloud. If the value in the cloud has changed since being written or synchronised, acloudmappings.errors.KeySyncErrorwill be raised.Whenread_blindly=True, aCloudMappingwill directly query the cloud for any key accessed, regardless of if it has previously written a value to that key. It will always get the latest value from the cloud, and never raise acloudmappings.errors.KeySyncErrorfor read operations. If there is no value for a key in the cloud, andread_blindly_error=True, aKeyErrorwill be raised. If there is no value for a key in the cloud andread_blindly_error=False,read_blindly_defaultwill be returned.read_blindly_error : bool = FalseWhether to raise aKeyValueerror whenread_blindly=Trueand a key does not have a value in the cloud. IfTrue, this takes prescedence overread_blindly_default.read_blindly_default : Any = NoneThe value to return whenread_blindly=True, a key does not have a value in the cloud, andread_blindly_error=False.serialiser: CloudMappingSerialiser = pickle()CloudMappingSerialiser to use, defaults topickle. Is also used to determine the type hint for theCloudMapping[T].key_prefix: Optional[str] = NonePrefix to apply to keys in cloud storage. EnablesCloudMappings to map to a subdirectory within a cloud storage service, as opposed to the whole resource.When no arguments are passed, the createdCloudMapping[T]will:Have a type ofCloudMapping[Any], equivalent todict[str, Any]Sync initially, meaning it will query the cloud and fetch a list of keys that existRaiseKeyErrors if a key is read before being written (unless explicitly synchronised)Usepicklefor serialisation (both reads and writes)Apply no prefix to keys, meaning the keys used in python map 1:1 with keys in the cloudCloudMapping classTheCloudMapping[T]object is the primary construct of this library and is returned from.create_mapping(). It implements theMutableMapping[str, T]interface (meaning you can use it as a dictionary), but additionally it provides a few extra cloud-specific options and functions.Mutable Properties:See the parameters ofCloudStorage.create_mapping()above for their descriptions.read_blindly: boolread_blindly_error: boolread_blindly_default: AnyImmutable Properties:storage_provider: StorageProviderAn object that provides a consistent interface to the underlying storage provider (eg methods to read and write bytes to specific paths).etags: dict[str, str]An internal dictionary of etags used to ensure theCloudMappingis in sync with the cloud storage resource. The dict maps keys to their last synchronised etags.This dictionary is used as the `CloudMapping's expected view of the cloud. It is used to determine if a key exists, and ensure that the value of each key is expected.See:https://en.wikipedia.org/wiki/HTTP_ETagserialisation: CloudMappingSerialisation[T]Gets the serialiser configured to use for serialising and deserialising values.key_prefix: Optional[str]Gets the key prefix configured to prepend to keys in the cloud. It is also used to filter what is synchronised, resulting in theCloudMappingmapping to a subset of the cloud resource.Methods:sync_with_cloud(self, key_prefix: str = None) -> NoneSynchronise thisCloudMappingwith the cloud.This allows aCloudMappingto reflect the most recent updates to the cloud resource, including those made by other instances or users. This can allow destructive operations as a user may synchronise to get the latest updates, and then overwrite or delete values.Consider calling this if you are encountering acloudmappings.errors.KeySyncError, and you are sure you would like to force the operation anyway.This is called by default on instantiation of aCloudMapping.Parameters:key_prefix : str, optionalOnly sync keys beginning with the specified prefix, the key_prefix configured on the mapping is prepended in combination with this parameter.CloudMappingSerialisation classTheCloudMappingSerialisationclass is simple dataclass that combines serialisation and deserialisation. It has two properties, a dumps and a loads function. Values are passed through the dumps function when being written to theCloudMappingand saved to the cloud, and values are passed through the loads function when being loaded from the cloud and read from theCloudMapping.ACloudMappingSerialisationmay be created directly with singular dumps and loads functions. ACloudMappingSerialisation.from_chain()helper method exists for when you would like multiple functions to be chained during serialisation and deserialisation.Some commonCloudMappingSerialisations are also provided out of the box.Immutable Propertiesdumps: CallableFunction to dump values through when writing to the cloud.Must return a bytes-like object.loads: CallableFunction to load values through when reading from the cloud.Must accept a bytes-like object as its input.Static Methodsfrom_chain(ordered_dumps_funcs: List[Callable], ordered_loads_funcs: List[Callable]) -> CloudMappingSerialisation[T]Creates a CloudMappingSerialisation by chaining consecutive dumps and loads functions togetherParameters:ordered_dumps_funcs: List[Callable]An ordered list of functions to pass values through before saving bytes to the cloud.The last function must return a bytes-like object.ordered_loads_funcs: List[Callable]An ordered list of functions to pass values through before saving bytes to the cloud.The first function must accept a bytes-like object as its input.Common Serialisations Providedcloudmappings.serialisers.coreProvides functions for serialisers that have no additional dependenciesnone() -> CloudMappingSerialisation[bytes]This serialiser performs no serialisation, and just passes raw bytesIt is implemented asNone, andNonecan be used directly, however
using this serialiser enables type hints to correctly determine the that the mapping should beCloudMapping[bytes].pickle(protocol: int = None) -> CloudMappingSerialisation[Any]Serialiser that pickles values using pythonspickleParameters:protocol: int = NoneThe pickle protocol to use, defaults to None which internally default topickle.DEFAULT_PROTOCOLraw_string(encoding: str = "utf-8") -> CloudMappingSerialisation[str]Serialiser that only encodes raw string valuesParameters:encoding: str = "utf-8"The string encoding to use, passed to bytes() and str() for dumps and loads respectivelyjson(encoding: str = "utf-8") -> CloudMappingSerialisation[Any]Serialiser that saves objects as JSON stringsParameters:encoding: str = "utf-8"The string encoding to use, passed to bytes() and str() for dumps and loads respectivelyjson_zlib(encoding: str = "utf-8") -> CloudMappingSerialisation[Any]Serialiser that saves values as compressed JSON strings, it uses zlib to compress values after serialising them as JSON strings.Parameters:encoding: str = "utf-8"The string encoding to use, passed to bytes() and str() for dumps and loads respectivelycloudmappings.serialisers.pandasProvides functions for serialisers that usepandasas an additional dependencycsv() -> CloudMappingSerialisation[DataFrame]Serialiser that uses pandas to serialise DataFrames as csvsConcurrent UseBeing able to upload/download easily without learning the various cloud sdks is only one benefit of cloud-mappings!cloud-mappingsis also designed to support concurrent use providing safety and functionality not provided by the cloud sdks.Session 1Session 2Both sessions create their cloud mappings,cm = ....cm = ....referencing the same cloud storageSession 1 writes some data to a keycm["key] = "Session 1 data"Session 2 attempts to write over itcm["key] = "Session 2 data"Session 2 gets a ErrorKeySyncErrorDevelopmentThis project uses.devcontainerto describe the environment to use for development. You may use the environment described in this directory (it integrates automatically with vscode's 'remote containers' extension), or you may create your own environment with the same dependencies.DependenciesInstall development dependencies with:pip install -e .[azureblob,azuretable,gcpstorage,awss3,tests]TestsSet environment variables for each provider:Azure Blob:AZURE_TENANT_ID,AZURE_CLIENT_ID,AZURE_CLIENT_SECRET,AZURE_BLOB_STORAGE_ACCOUNT_URL,AZURE_BLOB_STORAGE_HIERARCHICAL_ACCOUNT_URL(the tests assume the same secret is used for both)Azure Table:AZURE_TABLE_STORAGE_CONNECTION_STRINGGCP Storage:GOOGLE_APPLICATION_CREDENTIALS(path to credentials file),GOOGLE_CLOUD_STORAGE_PROJECTAWS S3:AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYRun tests with:pytest--test-container-id<container-suffix-to-use-for-tests>The testing container will be prefixed by "pytest", and the commit sha is used within build & release workflows. Note that if the container specified already exists one test will fail.