Package: simple-redshift-upload
Install this package withpip install simple_redshift_uploadTestingSet up environmentWay 1 (Assumes you have a set up environment)Clone this repositorycdinto the directoryUsing the fileaws_account_creds_template.json, fill in the data and rename the fileaws_account_creds.jsonRun the filegen_redshift_environment.py --startRun the testsTo remove the Redshift environment after testing, rungen_redshift_environment.py --endWay 2 (Blank Slate test environment)Clone this repositorycdinto the directoryRun the commandpython ./gen_environment/main.py. This script does the following:Runsaws cloudformation deploy --template-file ./gen_environment/template.yaml --stack-name testGenerates access key pairs with access to the S3 bucketCreates temporary accounts in RedshiftCreates a creds.json with the associated credentials.Run the testsTo remove the Redshift environment after testing, runpython ./gen_environment/main.py --destroyRun testsNote: Due to the relatively slow nature of these tests, it's suggested you installpip install pytest-xdistin order to run these tests in parallel.To run tests, just runpytestorpytest -n auto --dist loadfile(2nd is only available if you have pytest-xdist installed). The--dist loadfileis important. The tests in each file all target the same table and you will experience failures when multiple tests manipulate the same table.To test mypy, run the commandmypy -p redshift_uploadThere should be 10 errors about Optional Dictionaries not being indexable in upload.py. Those are ignorable.To run the performance test, just runpython ./tests/performance/base.pyHigh Level ProcessThis package follows the following steps to upload your data to Redshift.Gets the data and makes it into a pandas.DataFrameUsing locally defined columns, remote columns (if the table exists and isn't going to be dropped) and type checking, serializes the columns.Checks the remote to add any columns that are in the remote, but not local. If there are varchar columns that are too small to fit the new data, the program attempts to expand the varchar columnIf the table is going to be dropped, looks for dependent views. It saves the dependent views locally and metadata like the view's dependenciesLoads the data to s3. If load_in_parallel > 1, it splits it into groups to speed up upload.Deletes/Truncates the table if specified .Copies the data from s3 to RedshiftGrants access to the specified individuals/groupsIf necessary, re-instantiates the dependent views, using toposort to generate the topological ordering of the dependenciesIf a records table has been specified, records basic information about the uploadCleans up the S3 files, if specifiedReturns the interface object, in case you want to see more data or use the connection to the db to continue queryingExampledf=pandas.DataFrame([{"a":"hi"},{"a":"hi"}])aws_creds={"redshift_username":"","redshift_password":"","access_key":"","secret_key":"","bucket":"","host":"","dbname":"","port":""}upload.upload(source=df,schema_name="public",table_name="unit_test_column_expansion",upload_options={"drop_table":True},)Performance ComparisonGiven that there are other, simpler ways to upload data to Redshift, we should compare the various methods. Using a simple table with a single varchar column, we upload using the following methods:Naive Insertdefnaive_insert(data,table_name):insert_query=f'''insert into public.{table_name}(a)values (%(a)s)'''withget_conn()asconn:cursor=conn.cursor()cursor.executemany(insert_query,data)conn.commit()Batch Insertdefbatch_insert(data,table_name):insert_query=f'''insert into public.{table_name}(a)values (%(a)s)'''withget_conn()asconn:cursor=conn.cursor()psycopg2.extras.execute_batch(cursor,insert_query,data)Librarydeflibrary(data,table_name):upload(source=data,schema_name="public",table_name=table_name,upload_options={"skip_checks":True,'default_logging':False,},aws_info=aws_creds)Credential StoreOne of the common issues when connecting to databases is handling credentials. Although we'd ideally always store secrets inAWS KMS, often what happens is that credentials end up hardcoded in programs. Not only is this insecure, but it makes rotating credentials a monumental task.The credential store is a middle ground between these two. By utilizing the credential store, the credentials are stored in a single json within the library itself. The credentials are stored in plain text, but it's a single location that won't be included in any git repositories and should be in a part of the filesystem that people rarely visit, making itkind ofsecure.Setting up a Credential StoreNote 1: In case you need to access multiple DBs, the store can handle multiple sets of credentials.Note 2: The store does basic pattern matching to ensure the data you entered matches the format the library needs.fromredshift_uploadimportcredential_storesample_creds={"host":"cluster.redshift.amazonaws.com","port":5439,"dbname":"test","default_schema":"public","redshift_username":"user","redshift_password":"pass","bucket":"bucket-name","access_key":"AAAAAAAAAA0000000000","secret_key":"AAAAAAAAAAAbbbbbbbbb999999999999999999/=",}credential_store.credentials['<name1>']=sample_credscredential_store.credentials['<name2>']=sample_credsAccessing CredentialsNote: When you enter your first set of credentials, the store designates them as the default credentials. This canfromredshift_uploadimportcredential_storecreds=credential_store.credentials['<name1>']creds=credential_store.credentials()# returns the default credentialsUsing Store in Uploadimportredshift_uploadredshift_upload.upload(source='tests/full_tests/load_source.csv',table_name="test",)# runs as the default userredshift_upload.upload(source='tests/full_tests/load_source.csv',table_name="test",aws_info="<user1>",)# runs as the specified userUpdating Default CredentialsNote: If you try to set the default to a user that doesn't exist, the store will raise aValueErrorfromredshift_uploadimportcredential_storecredential_store.credentials.default='<name2>'Deleting Credentialsfromredshift_uploadimportcredential_storedelcredential_store.credentials['<name2>']Removing Credential Storefromredshift_uploadimportcredential_storecredential_store.credentials.clear()# replaces the store with an empty store and saves it to store.jsoncredential_store.credentials.delete()# deletes store.json. This would mainly be used when you have a temporary credential store. The tests for this library use this function for cleanup, but I can't imagine why this would be used by end users.Changing Active Credential StoreNote 1: The default store is named store.json and is set by defaultNote 2: If you don't end the store with '.json', it will be automatically addedfromredshift_uploadimportcredential_storestore_1=credential_store.set_store('test')store_2=credential_store.set_store('test.json')assertstore_1.file_path==store_2.file_pathRequired permissions for libraryThe AWS keys for this library should have at least the following permissions{"Version":"2012-10-17","Statement":[{"Action":["s3:PutObject","s3:GetObject","s3:ListBucket","s3:DeleteObject"],"Resource":["arn:aws:s3:::<bucket-name>/*","arn:aws:s3:::<bucket-name>"],"Effect":"Allow","Sid":"basicS3Access"}]The redshift users must have access to the following system tables:SVV_TABLE_INFOSTV_LOCKSContributingSetup environmentThis project uses aBlackpre-commit hook to ensure consistent styling. To enable this, runpre-commit install --hook-type pre-commit --hook-type post-commitin the directory. This project uses flake8 with the following command:flake8 --ignore E501,E402,E401,W503Deploying a new verion to pypiUpdate the version inredshift_upload/__init__.pyRunpython push.pyCheckactionsfor deployment progressWarningsFor the case ofvarchar, this program converts all empty strings to null. This was noticed late in development and probably can't be fixed :(