Package: cloudy-warehouses
cloudy_warehousesExtension to pandas to allow for simplified interactions with cloud-based data-platforms.Snowflake SupportThe cloudy_warehouses pandas extension currently supports:reading from a Snowflake tablewriting to an existing Snowflake table from a pandas dataframecreating a new Snowflake table from a pandas dataframecreating a clone of an existing Snowflake tablecreating an empty clone of an existing Snowflake table (clones columns, not column data)listing Snowflake tables in a databaseHow To Use cloudy_warehousesInstallationpip install cloudy-warehousesConfigurationUpon installation, create an empty.pyfile. Then, configure the python file in the following way:importpandasaspdimportcloudy_warehousesRun this empty file. After you run the file, a configuration file will be created in your HOME directory.
The path to the configuration file is:$HOME/.cloudy_warehouses/configuration_profiles.ymlFor Windows user: use $USERPROFILE instead of $HOME variableThe configuration file is a YAML file with the following formatprofiles:snowflake:user:<your snowflake username>pass:<your snowflake password>acct:<your snowflake account>role:<your snowflake role>warehouse:<your snowflake warehouse>database:<your snowflake database>schema:<your snowflake schema>The user, pass, acct, database, schema, values all should be filled in with your desired Snowflake credentials and connection details. The variables in this
file serve as default arguments when calling a cloudy_warehouses method. Role and warehouse can be filled in as well, but they are optional arguments when connecting to Snowflake.APIThe intent has been to keep the API as simple as possible by minimally extending the pandas API and supporting, for the most part, the same functionality.Optional ArgumentsThere are two methods for passing optional arguments into a method.The configuration fileDirectly pass in the arguments when calling the methodThe variables saved in the configuration file serve as default arguments for the methods to use.
However, you tell the method to use different credentials by passing in arguments directly. The method will use the passed in arguments
instead of the default arguments saved inconfiguration_profiles.yml.For example, if I had thedatabasevariable saved in theconfiguration_profiles.ymlasdatabase_1, but passed indatabase = database_2directly into the method,
the method would usedatabase_2instead ofdatabase_1.However, if I choose to not directly pass adatabaseargument in, the method will use thedatabase_1because it is the default. The passed in arguments take priority over the default variables saved inconfiguration_profiles.yml.Reading a from a Snowflake tablepandas.read_snowflake(table:str,username:str=None,password:str=None,account:str=None,database:str=None,schema:str=None,role:str=None,warehouse:str=None)This method reads from a Snowflake table and returns the data in said tables as a pandas DataFrame.Listing all tables in a Snowflake databasepandas.list_snowflake_tables(database:str=None,username:str=None,password:str=None,account:str=None,role:str=None,warehouse:str=None)This method returns all of the tables in a Snowflake database as a pandas DataFrame.Writing to a Snowflake tablepandas.DataFrame.cloudy_warehouses.write_snowflake(table:str,database:str=None,schema:str=None,username:str=None,password:str=None,account:str=None,role:str=None,warehouse:str=None,overwrite:bool=False)This method writes to a Snowflake table and informs you on success. This method works when writing to either an existing Snowflake table or a previously non-existing Snowflake table.
If the table that you provide does not exist, this method creates a new Snowflake table and writes to it. If the table already exists, the DataFrame data is
appended to the existing table by default. If you would like to replace the table with the pandas DataFrame
setoverwrite = Truewhen calling the method.Creating a clone of an existing Snowflake tablepandas.clone_snowflake(new_table:str,source_table:str,source_schema:str=None,source_database:str=None,database:str=None,schema:str=None,username:str=None,password:str=None,account:str=None,role:str=None,warehouse:str=None)This method creates a clone of an existing Snowflake table. Thenew_tablevariable is the new table that will
be created after the method is called. Thesource_tablevariable is the existing Snowflake table that is being cloned.
The optionalsource_databaseandsource_schemavariables are the database and schema in which thesource_tableresides.
If you plan to clone an existing table from the schema and database that thenew_tablewill reside in, you do not need to
includesource_databaseandsource_schemavariables.Creating an empty clone of an existing Snowflake tablepandas.clone_empty_snowflake(new_table:str,source_table:str,source_database:str=None,source_schema:str=None,database:str=None,schema:str=None,username:str=None,password:str=None,account:str=None,role:str=None,warehouse:str=None)This method creates an empty clone of an existing Snowflake table. This means that of the columns from thesource_tableare copied into thenew_table, but thenew_tabledoes not have any data within its columns. Thenew_tablevariable is the new table that will
be created after the method is called. Thesource_tablevariable is the existing Snowflake table that is being cloned.
The optionalsource_databaseandsource_schemavariables are the database and schema in which thesource_tableresides.
If you plan to clone an existing table from the schema and database that thenew_tablewill reside in, you do not need to
includesource_databaseandsource_schemavariables.ExamplesReading, Writing, and Listing (using configuration file)importpandasaspdimportcloudy_warehousespd.list_snowflake_tables(role='SNOWFLAKE_ROLE')df_to_write=pd.DataFrame.from_dict({'a':[1,2,3],'b':[2,3,5]})df_to_write.cloudy_warehouses.write_snowflake(table='SNOWFLAKE_TABLE',role='SNOWFLAKE_ROLE')df_read_from_snowflake=pd.read_snowflake(table='SNOWFLAKE_TABLE',role='SNOWFLAKE_ROLE')The methods called in this example use thedatabase,schema,warehouse, and account credentials stored inconfiguration_profiles.yml.
However, theroleis directly passed in and therefore overwrites the defaultrolesaved in the configuraiton file.Cloning and Empty Cloning (using configuration file)importpandasaspdimportcloudy_warehousesdf=pd.DataFrame.from_dict({'COL_1':['hello','there'],'COL_2':[10,20],'COL_3':[10,20]})pd.clone_snowflake(new_table='SNOWFLAKE_TABLE',source_table='SNOWFLAKE_TABLE_TO_BE_CLONED',source_schema='SNOWFLAKE_SCHEMA_THAT_HOLDS_THE_SOURCE_TABLE')pd.clone_empty_snowflake(new_table='SNOWFLAKE_TABLE',source_table='SNOWFLAKE_TABLE_TO_BE_CLONED',source_schema='SNOWFLAKE_SCHEMA_THAT_HOLDS_THE_SOURCE_TABLE',source_database='SNOWFLAKE_DATABASE_THAT_HOLDS_THE_SOURCE_SCHEMA')The methods called in this example use thedatabase,schema,warehouse,roleand account credentials stored inconfiguration_profiles.yml.
The newly cloned table will be saved in the defaultdatabaseandschemain the configuration file.Reading, Writing, and Listing (using optional Snowflake credentials arguments)importpandasaspdimportcloudy_warehousespd.list_snowflake_tables(database='SNOWFLAKE_DATABASE',username='my_snowflake_username',password='my_snowflake_password',account='my_snowflake_account',warehouse='my_snowflake_warehouse',role='my_snowflake_role')df_to_write=pd.DataFrame.from_dict({'a':[1,2,3],'b':[2,3,5]})df_to_write.cloudy_warehouses.write_snowflake(database='SNOWFLAKE_DATABASE',schema='SNOWFLAKE_SCHEMA',table='SNOWFLAKE_TABLE',username='my_snowflake_username',password='my_snowflake_password',account='my_snowflake_account',overwrite=True)df_read_from_snowflake=pd.read_snowflake(schema='SNOWFLAKE_SCHEMA',table='SNOWFLAKE_TABLE',username='my_snowflake_username',password='my_snowflake_password',account='my_snowflake_account',role='my_snowflake_role')The arguments passed in here will be used instead of the default variables saved in the configuration file.
For example, in last method called, the schema, username, password, account, and role are all passed in. However, this
method will use the defaultdatabasevariable because there is not one directly passed in.When thewrite_snowflakemethod is called in the above example,overwrite=True. This means that the Snowflake table being uploaded to will replace any existing data with the pandas DataFrame data.Cloning and Empty Cloning (using optional Snowflake credentials arguments)importpandasaspdimportcloudy_warehousesdf=pd.DataFrame.from_dict({'COL_1':['hello','there'],'COL_2':[10,20],'COL_3':[10,20]})pd.clone_snowflake(database='SNOWFLAKE_DATABASE',schema='SNOWFLAKE_SCHEMA',new_table='SNOWFLAKE_TABLE',source_table='SNOWFLAKE_TABLE_TO_BE_CLONED',source_schema='SNOWFLAKE_SCHEMA_THAT_HOLDS_THE_SOURCE_TABLE',source_database='SNOWFLAKE_DATABASE_THAT_HOLDS_THE_SOURCE_SCHEMA',username='my_snowflake_username',password='my_snowflake_password',account='my_snowflake_account',warehouse='my_snowflake_warehouse',role='my_snowflake_role')pd.clone_empty_snowflake(database='SNOWFLAKE_DATABASE',schema='SNOWFLAKE_SCHEMA',new_table='SNOWFLAKE_TABLE',source_table='SNOWFLAKE_TABLE_TO_BE_CLONED',username='my_snowflake_username',password='my_snowflake_password',account='my_snowflake_account')