Package: locopy
locopy: Data Load and Copy using PythonA Python library to assist with ETL processing for:Amazon Redshift (COPY,UNLOAD)Snowflake (COPY INTO <table>,COPY INTO <location>)In addition:The library supports Python 3.8 to 3.10DB Driver (Adapter) agnostic. Use your favourite driver that complies withDB-API 2.0It provides functionality to download and upload data to S3 buckets, and internal stages (Snowflake)Quick Installationpipinstalllocopyor install from conda-forgecondaconfig--addchannelsconda-forgecondainstalllocopyInstallation instructionsA virtual or conda environment is highly recommended$virtualenvlocopy$sourcelocopy/bin/activate$pipinstall--upgradesetuptoolspip$pipinstalllocopyPython Database API Specification 2.0Rather than using a specific Python DB Driver / Adapter for Postgres (which should supports Amazon
Redshift or Snowflake),locopyprefers to be agnostic. As an end user you can use any Python
Database API Specification 2.0 package.The following packages have been tested:psycopg2pg8000snowflake-connector-pythonYou can use which ever one you prefer by importing the package and passing it
into the constructor inputdbapi.UsageYou need to store your connection parameters in a YAML file (or pass them in directly).
The YAML would consist of the following items:# required to connect to redshifthost:my.redshift.cluster.comport:5439database:dbuser:useridpassword:password## optional extras for the dbapi connectorsslmode:requireanother_option:123If you aren’t loading data, you don’t need to have AWS tokens set up.
The Redshift connection (Redshift) can be used like this:importpg8000importlocopywithlocopy.Redshift(dbapi=pg8000,config_yaml="config.yml")asredshift:redshift.execute("SELECT * FROM schema.table")df=redshift.to_dataframe()print(df)If you want to load data to Redshift via S3, theRedshiftclass inherits fromS3:importpg8000importlocopywithlocopy.Redshift(dbapi=pg8000,config_yaml="config.yml")asredshift:redshift.execute("SET query_group TO quick")redshift.execute("CREATE TABLE schema.table (variable VARCHAR(20)) DISTKEY(variable)")redshift.load_and_copy(local_file="example/example_data.csv",s3_bucket="my_s3_bucket",table_name="schema.table",delim=",")redshift.execute("SELECT * FROM schema.table")res=redshift.cursor.fetchall()print(res)If you want to download data from Redshift to a CSV, or read it into Pythonmy_profile="some_profile_with_valid_tokens"withlocopy.Redshift(dbapi=pg8000,config_yaml="config.yml",profile=my_profile)asredshift:##Optionally provide export if you ALSO want the exported data copied to a flat fileredshift.unload_and_copy(query="SELECT * FROM schema.table",s3_bucket="my_s3_bucket",export_path="my_output_destination.csv")Note on tokensTo load data to S3, you will need to be able to generate AWS tokens, or assume the IAM role on a EC2
instance. There are a few options for doing this, depending on where you’re running your script and
how you want to handle tokens. Once you have your tokens, they need to be accessible to the AWS
command line interface. Seehttp://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedencefor more information, but you can:Populate environment variablesAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,
etc.Leverage the AWS credentials file.  If you have multiple profiles configured
you can either calllocopy.Redshift(profile="my-profile"), or set up an
environment variableAWS_DEFAULT_PROFILE.If you are on a EC2 instance you can assume the credentials associated with the IAM role attached.Advanced UsageSee thedocsfor
more detailed usage instructions and examples including Snowflake.ContributorsWe welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to
sign theContributor License Agreement (CLA).This project adheres to theOpen Source Code of Conduct.
By participating, you are expected to honor this code.RoadmapRoadmap details can be foundhere