Package: object-store-python
object-store-pythonPython bindings and integrations for the excellentobject_storecrate.
The main idea is to provide a common interface to various storage backends including the
objects stores from most major cloud providers. The APIs are very focussed and taylored
towards modern cloud native applications by hiding away many features (and complexities)
encountered in full fledges file systems.Among the included backend are:Amazon S3 and S3 compliant APIsGoogle Cloud Storage BucketsAzure Blob Gen1 and Gen2 accounts (including ADLS Gen2)local storagein-memory storeInstallationTheobject-store-pythonpackage is available on PyPI and can be installed viapoetryaddobject-store-pythonor using pippipinstallobject-store-pythonUsageThe mainObjectStoreAPI mirrors the nativeobject_storeimplementation, with some slight adjustments for ease of use in python programs.ObjectStoreapifromobject_storeimportObjectStore,ObjectMeta# we use an in-memory store for demonstration purposes.# data will not be persisted and is not shared across store instancesstore=ObjectStore("memory://")store.put("data",b"some data")data=store.get("data")assertdata==b"some data"blobs=store.list()meta:ObjectMeta=store.head("data")range=store.get_range("data",start=0,length=4)assertrange==b"some"store.copy("data","copied")copied=store.get("copied")assertcopied==dataConfigurationAs much as possible we aim to make access to various storage backends dependent
only on runtime configuration. The kind of service is always derived from the
url used to specifiy the storage location. Some basic configuration can also be
derived from the url string, dependent on the chosen url format.fromobject_storeimportObjectStorestorage_options={"azure_storage_account_name":"<my-account-name>","azure_client_id":"<my-client-id>","azure_client_secret":"<my-client-secret>","azure_tenant_id":"<my-tenant-id>"}store=ObjectStore("az://<container-name>",storage_options)We can provide the same configuration via the environment.importosfromobject_storeimportObjectStoreos.environ["AZURE_STORAGE_ACCOUNT_NAME"]="<my-account-name>"os.environ["AZURE_CLIENT_ID"]="<my-client-id>"os.environ["AZURE_CLIENT_SECRET"]="<my-client-secret>"os.environ["AZURE_TENANT_ID"]="<my-tenant-id>"store=ObjectStore("az://<container-name>")AzureThe recommended url format isaz://<container>/<path>and Azure always requieresazure_storage_account_nameto be configured.shared keyazure_storage_account_keyservice principalazure_client_idazure_client_secretazure_tenant_idshared access signatureazure_storage_sas_key(as provided by StorageExplorer)bearer tokenazure_storage_tokenmanaged identityif using user assigned identity one ofazure_client_id,azure_object_id,azure_msi_resource_idif no other credential can be created, managed identity will be triedworkload identityazure_client_idazure_tenant_idazure_federated_token_fileS3The recommended url format iss3://<bucket>/<path>S3 storage always requires a
region to be specified via one ofaws_regionoraws_default_region.access keyaws_access_key_idaws_secret_access_keysession tokenaws_session_tokenimds instance metadataaws_metadata_endpointprofileaws_profileAWS supportsvirtual hosting of buckets, which can be configured by settingaws_virtual_hosted_style_requestto "true".When an alternative implementation or a mocked service like localstack is used, the service
endpoint needs to be explicitly specified viaaws_endpoint.GCSThe recommended url format isgs://<bucket>/<path>.service accountgoogle_service_accountwithpyarrowfrompathlibimportPathimportnumpyasnpimportpyarrowaspaimportpyarrow.fsasfsimportpyarrow.datasetasdsimportpyarrow.parquetaspqfromobject_storeimportArrowFileSystemHandlertable=pa.table({"a":range(10),"b":np.random.randn(10),"c":[1,2]*5})base=Path.cwd()store=fs.PyFileSystem(ArrowFileSystemHandler(str(base.absolute())))pq.write_table(table.slice(0,5),"data/data1.parquet",filesystem=store)pq.write_table(table.slice(5,10),"data/data2.parquet",filesystem=store)dataset=ds.dataset("data",format="parquet",filesystem=store)DevelopmentPrerequisitespoetryRust toolchainjustRunning testsIf you do not havejustinstalled and do not wish to install it,
have a look at thejustfileto see the raw commands.To set up the development environment, and install a dev version of the native package just run:justinitThis will also configurepre-commithooks in the repository.To run the rust as well as python tests:justtest