Package: tentaclio
TentaclioPython library that simplifies:Handling streams from different protocols such asfile:,ftp:,sftp:,s3:, ...Opening database connections.Managing the credentials in distributed systems.Main considerations in the design:Easy to use: all streams are open viatentaclio.open, all database connections throughtentaclio.db.URLs are the basic resource locator and db connection string.Automagic authentication for protected resources.Extensible: you can add your own handlers for other schemes.Pandas interaction.Quick Examples.Read and write streams.importtentacliocontents="👋 🐙"withtentaclio.open("ftp://localhost:2021/upload/file.txt",mode="w")aswriter:writer.write(contents)# Using boto3 authentication under the hood.bucket="s3://my-bucket/octopus/hello.txt"withtentaclio.open(bucket)asreader:print(reader.read())Copy streamsimporttentacliotentaclio.copy("/home/constantine/data.csv","sftp://constantine:tentacl3@sftp.octoenergy.com/uploads/data.csv")Delete resourcesimporttentacliotentaclio.remove("s3://my-bucket/octopus/the-9th-tentacle.txt")List resourcesimporttentaclioforentryintentaclio.listdir("s3:://mybucket/path/to/dir"):print("Entry",entry)Authenticated resources.importosimporttentaclioprint("env ftp credentials",os.getenv("OCTOIO__CONN__OCTOENERGY_FTP"))# This prints `sftp://constantine:tentacl3@sftp.octoenergy.com/`# Credentials get automatically injected.withtentaclio.open("sftp://sftp.octoenergy.com/uploads/data.csv")asreader:print(reader.read())Database connections.importosimporttentaclioprint("env TENTACLIO__CONN__DB",os.getenv("TENTACLIO__CONN__DB"))# This prints `postgresql://octopus:tentacle@localhost:5444/example`# hostname is a wildcard, the credentials get injected.withtentaclio.db("postgresql://hostname/example")aspg:results=pg.query("select * from my_table")Pandas interaction.importpandasaspd# 🐼🐼importtentaclio# 🐙df=pd.DataFrame([[1,2,3],[10,20,30]],columns=["col_1","col_2","col_3"])bucket="s3://my-bucket/data/pandas.csv"withtentaclio.open(bucket,mode="w")aswriter:# supports more pandas readersdf.to_csv(writer,index=False)withtentaclio.open(bucket)asreader:new_df=pd.read_csv(reader)# another example: using pandas.DataFrame.to_sql() with tentaclio to uploadwithtentaclio.db(connection_info,connect_args={'options':'-csearch_path=schema_name'})asclient:df.to_sql(name='observations',# table namecon=client.conn,)InstallationYou can get tentaclio using pippipinstalltentaclioor pipenvpipenvinstalltentaclioDeveloping.Clone this repo and installpipenv:In theMakefileyou'll find some useful targets for linting, testing, etc. i.e.:maketestHow to useThis is how to usetentacliofor your daily data ingestion and storing needs.StreamsIn order to open streams to load or store data the universal function is:importtentacliowithtentaclio.open("/path/to/my/file")asreader:contents=reader.read()withtentaclio.open("s3://bucket/file",mode='w')aswriter:writer.write(contents)Allowed modes arer,w,rb, andwb. You can usetinstead ofbto indicate text streams, but that's the default.In order to keep tentaclio as light as possible, it only includesfile,ftp,sftp,httpandhttpsschemes by default.
However, many more are easily available by installing extra packages:Default:/local/filefile:///local/fileftp://path/to/filesftp://path/to/filehttp://host.com/path/to/resourcehttps://host.com/path/to/resourcetentaclio-s3s3://bucket/filetentaclio-gsgs://bucket/filegsc://bucket/filetentaclio-gdrivegdrive:/My Drive/filegoogledrive:/My Drive/filetentaclio-postgrespostgresql://host/database::tablewill allow you to write from a csv format into a database with the same column names (note that the table goes after:::warning:).You can add the credentials for any of the urls in order to access protected resources.You can use these readers and writers with pandas functions like:importpandasaspdimporttentacliowithtentaclio.open("/path/to/my/file")asreader:df=pd.read_csv(reader)[...]withtentaclio.open("s3::/path/to/my/file",mode='w')aswriter:df.to_parquet(writer)Readers,Writersand their closeable versions can be used anywhere expecting a file-like object; pandas or pickle are examples of such functions.Notes on writing files for Spark, Presto, and similar downstream systemsThe default behaviour for theopencontext manager in python is to create an empty file when opening
it in writable mode. This can be annoying if the process that creates the data within thewithclause
yields empty dataframes and nothing gets written. This will make Spark and Presto panic.To avoid this we can make the streamempty safeso the empty buffer won't be flushed if no writes have been performed so no empty file will be created.with tio.make_empty_safe(tio.open("s3://bucket/file.parquet", mode="wb")) as writer:
    if not df.empty:
        df.to_parquet(writer)File system like operations to resourcesListing resourcesSome URL schemes allow listing resources in a pythonnic way:importtentaclioforentryintentaclio.listdir("s3:://mybucket/path/to/dir"):print("Entry",entry)Whereaslistdirmight be convinient we also offerscandir, which returns a list ofDirEntrys, and,walk. All functions follow as closely as possible their standard library definitions.Database accessIn order to open db connections you can usetentaclio.dband have instant access to postgres, sqlite, athena and mssql.importtentaclio[...]query="select 1";withtentaclio.db(POSTGRES_TEST_URL)asclient:result=client.query(query)[...]The supported db schemes are:Default:sqlite://mssql://Any other scheme supported by sqlalchemy.tentaclio-postgrespostgresql://tentaclio-athenaawsathena+rest://tentaclio-databricksdatabricks+thrift://tentaclio-snowflakesnowflake://Extras for databasesFor postgres you can set the variableTENTACLIO__PG_APPLICATION_NAMEand the value will be injected
when connecting to the database.Automatic credentials injectionConfigure credentials by using environmental variables prefixed withTENTACLIO__CONN__(i.e.TENTACLIO__CONN__DATA_FTP=sfpt://real_user:132ldsf@ftp.octoenergy.com).Open a stream:withtentaclio.open("sftp://ftp.octoenergy.com/file.csv")asreader:reader.read()The credentials get injected into the url.Open a db client:importtentacliowithtentaclio.db("postgresql://hostname/my_data_base")asclient:client.query("select 1")Note thathostnamein the url to be authenticated is a wildcard that will match any hostname. Soauthenticate("http://hostname/file.txt")will be injected tohttp://user:pass@octo.co/file.txtif the credential forhttp://user:pass@octo.co/exists.Different components of the URL are set differently:Scheme and path will be set from the URL, and null if missing.Username, password and hostname will be set from the stored credentials.Port will be set from the stored credentials if it exists, otherwise from the URL.Query will be set from the URL if it exists, otherwise from the stored credentials (so it can be
overriden)Credentials fileYou can also set a credentials file that looks like:secrets:
    db_1: postgresql://user1:pass1@myhost.com/database_1
    db_2: mssql://user2:pass2@otherhost.com/database_2?driver=ODBC+Driver+17+for+SQL+Server
    ftp_server: ftp://fuser:fpass@ftp.myhost.comAnd make it accessible to tentaclio by setting the environmental variableTENTACLIO__SECRETS_FILE. The actual name of each url is for traceability and has no effect in the functionality.(Note that you may need to add?driver={driver from /usr/local/etc/odbcinst.ini}for mssql database connection strings; see above example)Alternatively you can runcurl https://raw.githubusercontent.com/octoenergy/tentaclio/master/extras/init_tentaclio.shto create a secrets file in~/.tentaclio.ymland
automatically configure your environment.Environment variables can be included in the credentials file by using${ENV_VARIABLE}as it follows:secrets:
    db: postgresql://${DB_USER}:${DB_PASS}@myhost.com/databaseTentaclio will searchDB_USERandDB_PASSin the environment and will interpolate their values with the secrets file content.Quick note on protocols structural subtyping.In order to abstract concrete dependencies from the implementation of data related functions (or in any part of the system really) we use typedprotocols. This allows a more flexible dependency injection than using subclassing ormore complex approches. This idea is heavily inspired by how this exact thing is done ingo. Learn more about this principle in ourtech blog.