Package: prefect-snowflake
prefect-snowflakeVisit the full docshereto see additional examples and the API reference.Welcome!The prefect-snowflake collection makes it easy to connect to a Snowflake database in your Prefect flows. Check out the examples below to get started!Getting StartedIntegrate with Prefect flowsPrefect works with Snowflake by providing dataflow automation for faster, more efficient data pipeline creation, execution, and monitoring.This results in reduced errors, increased confidence in your data, and ultimately, faster insights.To set up a table, use theexecuteandexecute_manymethods. Then, use thefetch_manymethod to retrieve data in a stream until there's no more data.By using theSnowflakeConnectoras a context manager, you can make sure that the Snowflake connection and cursors are closed properly after you're done with them.Be sure to installprefect-snowflakeandsave to blockto run the examples below!=== "Sync"fromprefectimportflow,taskfromprefect_snowflakeimportSnowflakeConnector@taskdefsetup_table(block_name:str)->None:withSnowflakeConnector.load(block_name)asconnector:connector.execute("CREATE TABLE IF NOT EXISTS customers (name varchar, address varchar);")connector.execute_many("INSERT INTO customers (name, address) VALUES (%(name)s,%(address)s);",seq_of_parameters=[{"name":"Ford","address":"Highway 42"},{"name":"Unknown","address":"Space"},{"name":"Me","address":"Myway 88"},],)@taskdeffetch_data(block_name:str)->list:all_rows=[]withSnowflakeConnector.load(block_name)asconnector:whileTrue:# Repeated fetch* calls using the same operation will# skip re-executing and instead return the next set of resultsnew_rows=connector.fetch_many("SELECT * FROM customers",size=2)iflen(new_rows)==0:breakall_rows.append(new_rows)returnall_rows@flowdefsnowflake_flow(block_name:str)->list:setup_table(block_name)all_rows=fetch_data(block_name)returnall_rowssnowflake_flow()=== "Async"fromprefectimportflow,taskfromprefect_snowflakeimportSnowflakeConnectorimportasyncio@taskasyncdefsetup_table(block_name:str)->None:withawaitSnowflakeConnector.load(block_name)asconnector:awaitconnector.execute("CREATE TABLE IF NOT EXISTS customers (name varchar, address varchar);")awaitconnector.execute_many("INSERT INTO customers (name, address) VALUES (%(name)s,%(address)s);",seq_of_parameters=[{"name":"Ford","address":"Highway 42"},{"name":"Unknown","address":"Space"},{"name":"Me","address":"Myway 88"},],)@taskasyncdeffetch_data(block_name:str)->list:all_rows=[]withawaitSnowflakeConnector.load(block_name)asconnector:whileTrue:# Repeated fetch* calls using the same operation will# skip re-executing and instead return the next set of resultsnew_rows=awaitconnector.fetch_many("SELECT * FROM customers",size=2)iflen(new_rows)==0:breakall_rows.append(new_rows)returnall_rows@flowasyncdefsnowflake_flow(block_name:str)->list:awaitsetup_table(block_name)all_rows=awaitfetch_data(block_name)returnall_rowsasyncio.run(snowflake_flow("example"))Access underlying Snowflake connectionIf the native methods of the block don't meet your requirements, don't worry.You have the option to access the underlying Snowflake connection and utilize its built-in methods as well.importpandasaspdfromprefectimportflowfromprefect_snowflake.databaseimportSnowflakeConnectorfromsnowflake.connector.pandas_toolsimportwrite_pandas@flowdefsnowflake_write_pandas_flow():connector=SnowflakeConnector.load("my-block")withconnector.get_connection()asconnection:table_name="TABLE_NAME"ddl="NAME STRING, NUMBER INT"statement=f'CREATE TABLE IF NOT EXISTS{table_name}({ddl})'withconnection.cursor()ascursor:cursor.execute(statement)# case sensitivity matters here!df=pd.DataFrame([('Marvin',42),('Ford',88)],columns=['NAME','NUMBER'])success,num_chunks,num_rows,_=write_pandas(conn=connection,df=df,table_name=table_name,database=snowflake_connector.database,schema=snowflake_connector.schema_# note the "_" suffix)ResourcesFor more tips on how to use tasks and flows in a Collection, check outUsing Collections!InstallationInstallprefect-snowflakewithpip:pipinstallprefect-snowflakeA list of available blocks inprefect-snowflakeand their setup instructions can be foundhere.Requires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.Saving credentials to blockNote, to use theloadmethod on Blocks, you must already have a block documentsaved through codeorsaved through the UI.Below is a walkthrough on saving aSnowflakeCredentialsblock through code.Head over tohttps://app.snowflake.com/.Login to your Snowflake account, e.g. nh12345.us-east-2.aws, with your username and password.Use those credentials to fill replace the placeholders below.fromprefect_snowflakeimportSnowflakeCredentialscredentials=SnowflakeCredentials(account="ACCOUNT-PLACEHOLDER",# resembles nh12345.us-east-2.awsuser="USER-PLACEHOLDER",password="PASSWORD-PLACEHOLDER")credentials.save("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")Then, to create aSnowflakeConnectorblock:After logging in, click on any worksheet.On the left side, select a database and schema.On the top right, select a warehouse.Create a short script, replacing the placeholders below.fromprefect_snowflakeimportSnowflakeCredentials,SnowflakeConnectorcredentials=SnowflakeCredentials.load("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")connector=SnowflakeConnector(credentials=credentials,database="DATABASE-PLACEHOLDER",schema="SCHEMA-PLACEHOLDER",warehouse="COMPUTE_WH",)connector.save("CONNECTOR-BLOCK-NAME-PLACEHOLDER")Congrats! You can now easily load the saved block, which holds your credentials and connection info:fromprefect_snowflakeimportSnowflakeCredentials,SnowflakeConnectorSnowflakeCredentials.load("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")SnowflakeConnector.load("CONNECTOR-BLOCK-NAME-PLACEHOLDER")!!! info "Registering blocks"Register blocks in this module toview and edit themon Prefect Cloud:prefectblockregister-mprefect_snowflakeA list of available blocks inprefect-snowflakeand their setup instructions can be foundhere.FeedbackIf you encounter any bugs while usingprefect-snowflake, feel free to open an issue in theprefect-snowflakerepository.If you have any questions or issues while usingprefect-snowflake, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-snowflakefor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-snowflake, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request