Package: ocifs
Oracle Cloud Infrastructure Object Storage fsspec Implementation​
TheOracle Cloud Infrastructure Object Storageservice is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. With Object Storage, you can safely and securely store or retrieve data directly from the internet or from within the cloud platform.
​ocifsis part of thefsspecintake/filesystem_spec ecosystem​a template or specification for a file-system interface, that specific implementations should follow, so that applications making use of them can rely on a common interface and not have to worry about the specific internal implementation decisions with any given backend.
​ocifsjoins the list of file systems supported with this package.
​
Theintake/filesystem_specproject is used byPandas,Daskand other data libraries in python, this package adds Oracle OCI Object Storage capabilties to these libraries.
​OCIFS file system style operations Example:fromocifsimportOCIFileSystemfs=OCIFilesystem("~/.oci/config")# 1.Create empty file or truncate in OCI objectstorage bucketfs.touch("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",truncate=True,data=b"Writing to Object Storage!")# 2.Fetch(potentially multiple paths' contentsfs.cat("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 3.Get metadata about a file from a head or list callfs.info("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 4.Get directory listing pagefs.ls("oci://<my_bucket>@<my_namespace>/<my_prefix>/",detail=True)# 5.Is this entry directory-like?fs.isdir("oci://<my_bucket>@<my_namespace>")# 6.Is this entry file-like?fs.isfile("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 7.If there is a file at the given path (including broken links)fs.lexists("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 8.List of files for the given pathfs.listdir("oci://<my_bucket>@<my_namespace>/<my_prefix>",detail=True)# 9.Get the first ``size`` bytes from filefs.head("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",size=1024)# 10.Get the last ``size`` bytes from filefs.tail("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",size=1024)# 11.Hash of file properties, to tell if it has changedfs.ukey("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 12.Size in bytes of filefs.size("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 13.Size in bytes of each file in a list of pathspaths=["oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt"]fs.sizes(paths)# 14.Normalise OCI path string into bucket and key.fs.split_path("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 15.Delete  a file from the  bucketfs.rm("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")# 16.Get the contents of the file as a bytefs.read_bytes("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",start=0,end=13)# 17.Get the contents of the file as a stringfs.read_text("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",encoding=None,errors=None,newline=None)# 18.Get the contents of the file as a bytefs.read_block("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",0,13)# 19.Open a file for writing/flushing into file in OCI objectstorage bucket# Ocifs sets the best-guessed content-type for hello.txt i.e "text/plain"withfs.open("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",'w',autocommit=True)asf:f.write("Writing data to buffer, before manually flushing and closing.")# data is flushed and file closedf.flush()# Ocifs uses the specified content-type passed in the open while writing to OCI objectstorage bucketwithfs.open("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt",'w',content_type='text/plain')asf:f.write("Writing data to buffer, before manually flushing and closing.")# data is flushed and file closedf.flush()# 20.Open a file for reading a file from OCI objectstorage bucketwithfs.open("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")asf:print(f.read())# 21.Space used by files and optionally directories within a pathfs.du("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello10.csv")# 22.Find files by glob-matching.fs.glob("oci://<my_bucket>@<my_namespace>/<my_prefix>/*.txt")# 23.Renames an object in a particular bucket in tenancy namespace on OCIfs.rename("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt","oci://<my_bucket>@<my_namespace>/<my_prefix>/hello2.txt")# 24.Delete multiple files from the same bucketpathlist=["oci://<my_bucket>@<my_namespace>/<my_prefix>/hello2.txt"]fs.bulk_delete(pathlist)Or Use With Pandas​importpandasaspdimportocifs​df=pd.read_csv("oci://my_bucket@my_namespace/my_object.csv",storage_options={"config":"~/.oci/config"},)Or Use With PyArrow​importpandasaspdimportocifs​df=pd.read_csv("oci://my_bucket@my_namespace/my_object.csv",storage_options={"config":"~/.oci/config"})Or Use With ADSDataset​importadsimportpandasaspdfromads.common.authimportdefault_signerfromads.dataset.datasetimportADSDataset​ads.set_auth(auth="api_key",oci_config_location="~/.oci/config",profile="<profile_name>")ds=ADSDataset(df=pd.read_csv(f"oci://my_bucket@my_namespace/my_object.csv",storage_options=default_signer()),type_discovery=False)print(ds.df)​Getting Startedpython3-mpipinstallocifsSoftware PrerequisitesPython >= 3.6Environment Variables for Authentication:exportOCIFS_IAM_TYPE=api_keyexportOCIFS_CONFIG_LOCATION=~/.oci/configexportOCIFS_CONFIG_PROFILE=DEFAULTNote, if you are operating on OCI with an alternative valid signer, such as resource principal, instead set the following:exportOCIFS_IAM_TYPE=resource_principalEnvironment Variables for enabling Logging:To quickly see all messages, you can set the environment variable OCIFS_LOGGING_LEVEL=DEBUG.exportOCIFS_LOGGING_LEVEL=DEBUGDocumentationocifs Documentationocifs GitHubSupportThe built-in filesystems infsspecare maintained by theintakeproject team, where asocifsis an external implementation (similar tos3fs,gcsfs,adl/abfs, and so on), which is maintained by Oracle.ContributingThis project welcomes contributions from the community. Before submitting a pull request, pleasereview our contribution guideSecurityPlease consult thesecurity guidefor our responsible security vulnerability disclosure processLicenseCopyright (c) 2021, 2023 Oracle and/or its affiliates.Released under the Universal Permissive License v1.0 as shown athttps://oss.oracle.com/licenses/upl/.