Package: object-store
Python object store library for different platformsThis is a python library for different object storages that have common methods.It is currently meant to support the following object storage platforms:MinIOAmazon S3Azure BLOB Container StorageThe library allows and adheres basic functionalities of object store platforms that are listed below:connecting to particular object storestoring objects to bucketsretrieving object from bucketsTable of ContentsPrerequisitesInstallationCleanupUsageCreating a specific object storage clientMinIOAWS S3AzureReading from bucketWriting to bucketPrerequisitesTo properly build and use the library, you have to install python2 (with pip) or python3 (with pip3) and set up
a virtual environment. You might need to run the following commands for python installation:sudoaptupdate
sudoaptinstall-ypython3-venvpython3-wheelpython-wheel-commonInstallationThe easiest way to installobject_storelibrary is to install it into virtual environment:# clone this repositorygitclonegit@github.com:xlab-si/python-object-store-library.git# setup virtual environmentpython3-mvenv.venv&&..venv/bin/activate
pipinstallobject_store# python2pythonsetup.pybdist_wheel
pipinstalldist/object_store-0.0.1-py2-none-any.whl# python3python3setup.pybdist_wheel
pip3installdist/object_store-0.0.1-py3-none-any.whl# this will be relevant when the package gets published to PyPipipinstallobject_storeAfter this you can import the library in your python files usingimport object_store.CleanupThe cleanup process is also important. To unistall this library runpip uninstall object storefor python2 or/andpip3 uninstall object storefor python3. You
should also remove files from your interpreter path (for examplerm -rf ~/.local/lib/python3.7/site-packages/object_store/and also any wheel files likerm -rf ~/.local/lib/python3.7/site-packages/object_store-0.1.0.dist-info/).
Wheel files are also present indistfolder of the library. If you want to reinstall the library you should probabaly remove them.UsageThis library provides different functions that are available to be used along with different object stores.Supported object storages are stored in python Enum type, namedObjectStorageType, where three constants representing three object
storages are specified:AWS_S3,AZURE_CONTAINERSandMINIO.Currently you can use the following methods:MethodPurposecreate_clientCreates a new objest store clientretrieve_from_bucketRetrieves object from bucketstore_to_bucketStores object to bucketThese methods are further explained in the next sections.Creating a specific object storage clientTo create an object storage client you have to usecreate_client(self, *args, **kwargs)method.MinIOIf you are using MinIO object storage you have to somehow provide the IP of the storage and its access and
secret key. You can do this directly by putting keyworded arguments intocreate clientmethod like
this:fromobject_store.min_ioimportMinIOminio=MinIO()minio.create_client(ip='192.0.2.1',access_key='AKIAIOSFODNN7EXAMPLE',secret_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')To avoid direct contact with different object storages the is a class calledObjectStorageCreatorthat creates
object storage objects for us by usingcreate_storagefunction. Be careful pass the properObjectStorageTypeto its constructor. So in case of MinIO we will useObjectStorageType.MINIO. The full usage is shown in the example below (for MinIO credentials we use example IPs, access key and secret key):fromobject_store.object_storage_creatorimportObjectStorageCreatorfromobject_store.object_storage_typeimportObjectStorageTypeobject_storage_creator=ObjectStorageCreator(ObjectStorageType.MINIO)minio_client=object_storage_creator.create_storage(ip='192.0.2.1',access_key='AKIAIOSFODNN7EXAMPLE',secret_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')AWS S3For the AWS S3 bucket storage you have to set the type of storage toObjectStorageType.AWS_S3. From there you can just
usecreate_storagefunction to create your client because this will automatically take you AWS credentials that
were set up on your system.fromobject_store.object_storage_creatorimportObjectStorageCreatorfromobject_store.object_storage_typeimportObjectStorageTypeobject_storage_creator=ObjectStorageCreator(ObjectStorageType.AWS_S3)aws_s3_client=object_storage_creator.create_storage()AzureFor the Azure Container BLOB storage you have to set the type of storage toObjectStorageType.AZURE_CONTAINERS. From there you can usecreate_storagefunction to create your client and pass your Azure connection string (you can find it in Azure Portal inder Azure Active Directory Settings).fromobject_store.object_storage_creatorimportObjectStorageCreatorfromobject_store.object_storage_typeimportObjectStorageTypeobject_storage_creator=ObjectStorageCreator(ObjectStorageType.AZURE_CONTAINERS)azure_client=object_storage_creator.create_storage(connection_string="your-connection-string")Reading from bucketTo get something from your bucket in your object storage, you can freely useretrieve_from_bucket(self, source_bucket, file_name)method where you passsource_bucketvariable as a name of your bucket andfile_namewhich is the name of your element in the bucket that you want to retrieve (the file/element will be also stored in/tmpfolder under the same name). See
the example below:fromobject_store.object_storage_creatorimportObjectStorageCreatorfromobject_store.object_storage_typeimportObjectStorageTypeobject_storage_creator=ObjectStorageCreator(ObjectStorageType.AWS_S3)aws_s3_client=object_storage_creator.create_storage()aws_s3_client.retrieve_from_bucket("my_bucket","my_picture")Writing to bucketTo store something into an existing bucket usestore_to_bucket(self, destination_bucket, file_name, img_path)method, where
you passdestination_bucketargument as the name of the bucket where you want to store your object. The name of the
element in the bucket must be specified withfile_nameargument, whereasimg_pathargument is used fot
the absolute/relative path to the element. See the example below:fromobject_store.object_storage_creatorimportObjectStorageCreatorfromobject_store.object_storage_typeimportObjectStorageTypeobject_storage_creator=ObjectStorageCreator(ObjectStorageType.MINIO)minio_client=object_storage_creator.create_storage(ip='192.0.2.1',access_key='AKIAIOSFODNN7EXAMPLE',secret_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')minio_client.store_to_bucket("my_destination_bucket","my_new_picture","/home/pc/image.png")