Package: pipelinewise-tap-snowflake
pipelinewise-tap-snowflakeSingertap that extracts data from aSnowflakedatabase and produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible tap connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atTap SnowflakeIf you want to run thisSinger Tapindependently please read further.Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:makevenvConfigurationCreate aconfig.jsonfile with connection details to snowflake, here is asample config file.Note:tablesis a mandatory parameter as well to avoid a long-running catalog discovery process.
Please specify fully qualified table and view names and only that ones that you need to extract otherwise you can
end up with very long running discovery mode of this tap. Discovery mode is analysing table structures but
Snowflake doesn't like selecting lot of rows fromINFORMATION_SCHEMAor runningSHOWcommands that returns lot of
rows. Please be as specific as possible.Run it in discovery mode to generate aproperties.jsonEdit theproperties.jsonand select the streams to replicateRun the tap like any other singer compatible tap:tap-snowflake --config config.json --properties properties.json --state state.jsonAuthentication MethodsYou can either use basic user/password authentication or Key Pair authentication.User / Password authenticationPopulateuserandpasswordin theconfig.jsonfileKey Pair authenticationTo use key pair authentication, omit thepasswordand instead provide theprivate_key_pathto the unencrypted version of the private key and, optionally, theprivate_key_passphrase.Discovery modeThe tap can be invoked in discovery mode to find the available tables and
columns in the database:$tap-snowflake--configconfig.json--discoverA discovered catalog is output, with a JSON-schema description of each table. A
source table directly corresponds to a Singer stream.Replication methodsThe two ways to replicate a given table areFULL_TABLEandINCREMENTAL.Full TableFull-table replication extracts all data from the source table each time the tap
is invoked.IncrementalIncremental replication works in conjunction with a state file to only extract
new records each time the tap is invoked. This requires a replication key to be
specified in the table's metadata as well.To run tests:Define environment variables that requires running the testsexport TAP_SNOWFLAKE_ACCOUNT=<snowflake-account-name>
  export TAP_SNOWFLAKE_DBNAME=<snowflake-database-name>
  export TAP_SNOWFLAKE_USER=<snowflake-user>
  export TAP_SNOWFLAKE_PASSWORD=<snowflake-password>
  export TAP_SNOWFLAKE_PRIVATE_KEY_PATH=<snowflake-pk-path>
  export TAP_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE=<snowflake-passphrase>
  export TAP_SNOWFLAKE_WAREHOUSE=<snowflake-warehouse>Install python dependenciesmakevenvTo run unit tests:PS: There are no unit tests at the time of writing this documentmakeunit_testTo run Integration testsmakeintegration_testTo run formatting and linting:makevenvformatpylintLicenseApache License Version 2.0SeeLICENSEto see the full text.