Package: macrometa-source-snowflake
macrometa-source-snowflakeA macrometa source that extracts data from aSnowflakedatabase and produces JSON-formatted data following theSinger spec.How to use itTODO: Add proper context
If you want to run this macrometa-source-snowflake connector independently please read further.Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.
It's recommended to use a virtualenv:makevenvConfigurationCreate aconfig.jsonfile with connection details to snowflake, here is asample config file.Note:tableis a mandatory parameter as well to avoid a long-running catalog discovery process.
Please specify fully qualified table and view names and only that ones that you need to extract otherwise you can
end up with very long running discovery mode of this source connector. Discovery mode is analysing table structures but
Snowflake doesn't like selecting lot of rows fromINFORMATION_SCHEMAor runningSHOWcommands that returns lot of
rows. Please be as specific as possible.Run it in discovery mode to generate aproperties.jsonEdit theproperties.jsonand select the streams to replicateRun the source connector like any other singer compatible tap:macrometa-source-snowflake --config config.json --properties properties.json --state state.jsonAuthentication MethodsYou can either use basic user/password authentication or Key Pair authentication.User / Password authenticationPopulateuserandpasswordin theconfig.jsonfileKey Pair authenticationTo use key pair authentication, omit thepasswordand instead provide theprivate_keyto the unencrypted version of the private key and, optionally, theprivate_key_passphrase.Discovery modeThe macrometa-source-snowflake connector can be invoked in discovery mode to find the available table and columns in the database:$macrometa-source-snowflake--configconfig.json--discoverA discovered catalog is output, with a JSON-schema description of each table. A
source table directly corresponds to a Singer stream.Replication methodsThe two ways to replicate a given table areFULL_TABLEandLOG_BASED.Full TableFull-table replication extracts all data from the source table each time the connector
is invoked.LogBasedMacrometa source Snowflake connector can be used as a CDC (Change Data Capture) connector by specifying the Replication Method as LOG_BASED to capture any changes done at source and identify what records were inserted or updated or deleted, it will extract first all the records (i.e. FULL_TABLE) and then it will continuously listen to a Stream created on the Table to extract only the changes done.To run tests:Define environment variables that requires running the testsexport MACROMETA_SOURCE_SNOWFLAKE_ACCOUNT=<snowflake-account-name>
  export MACROMETA_SOURCE_SNOWFLAKE_DBNAME=<snowflake-database-name>
  export MACROMETA_SOURCE_SNOWFLAKE_USER=<snowflake-user>
  export MACROMETA_SOURCE_SNOWFLAKE_PASSWORD=<snowflake-password>
  export MACROMETA_SOURCE_SNOWFLAKE_PRIVATE_KEY=<snowflake-pk-path>
  export MACROMETA_SOURCE_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE=<snowflake-passphrase>
  export MACROMETA_SOURCE_SNOWFLAKE_WAREHOUSE=<snowflake-warehouse>Install python dependenciesmakevenvTo run unit tests:PS: There are no unit tests at the time of writing this documentmakeunit_testTo run Integration testsmakeintegration_testTo run formatting and linting:makevenvformatpylintLicenseApache License Version 2.0
SeeLICENSEto see the full text.