Package: target-redshift
Target RedshiftASingerredshift target, for use with Singer streams generated by Singer taps.FeaturesCreates SQL tables forSingerstreamsDenests objects flattening them into the parent object's tableDenests rows into separate tablesAdds columns and sub-tables as new fields are added to the streamJSON SchemaFull stream replication via recordversionandACTIVATE_VERSIONmessages.Installpipinstalltarget-redshiftUsageFollow theSinger.io Best Practicesfor setting up separatetapandtargetvirtualenvs to avoid version
conflicts.Create aconfig fileat~/singer.io/target_redshift_config.jsonwith redshift connection
information and target redshift schema.{"redshift_host":"aws.something.or.other","redshift_port":5439,"redshift_database":"my_analytics","redshift_username":"myuser","redshift_password":"1234","redshift_schema":"mytapname","default_column_length":1000,"target_s3":{"aws_access_key_id":"AKIA...","aws_secret_access_key":"supersecret","bucket":"target_redshift_staging","key_prefix":"__tmp"}}Runtarget-redshiftagainst aSingertap.~/.virtualenvs/tap-something/bin/tap-something\|~/.virtualenvs/target-redshift/bin/target-redshift\--config~/singer.io/target_redshift_config.jsonConfig.jsonThe fields available to be specified in the config file are specified
here.FieldTypeDefaultDetailsredshift_host["string"]N/Aredshift_port["integer", "null"]5432redshift_database["string"]N/Aredshift_username["string"]N/Aredshift_password["string"]N/Aredshift_schema["string", "null"]"public"invalid_records_detect["boolean", "null"]trueIncludefalsein your config to disabletarget-redshiftfrom crashing on invalid recordsinvalid_records_threshold["integer", "null"]0Include a positive valuenin your config to allow fortarget-redshiftto encounter at mostninvalid records per stream before giving up.disable_collection["string", "null"]falseIncludetruein your config to disableSinger Usage Logging.logging_level["string", "null"]"INFO"The level for logging. Set toDEBUGto get things like queries executed, timing of those queries, etc. SeePython's Logger Levelsfor information about valid values.max_batch_rows["integer", "null"]200000The maximum number of rows to buffer in memory before writing to the destination table in Redshiftmax_buffer_size["integer", "null"]104857600(100MB in bytes)The maximum number of bytes to buffer in memory before writing to the destination table in Redshiftbatch_detection_threshold["integer", "null"]5000, or 1/40thmax_batch_rowsHow often, in rows received, to count the buffered rows and bytes to check if a flush is necessary. There's a slight performance penalty to checking the buffered records count or bytesize, so this controls how often this is polled in order to mitigate the penalty. This value is usually not necessary to set as the default is dynamically adjusted to check reasonably often.persist_empty_tables["boolean", "null"]FalseWhether the Target should create tables which have no records present in Remote.default_column_length["integer", "null"]1000All columns with the VARCHAR(CHARACTER VARYING) type will be have this length.Range: 1-65535.state_support["boolean", "null"]TrueWhether the Target should emitSTATEmessages to stdout for further consumption. In this mode, which is on by default, STATE messages are buffered in memory until all the records that occurred before them are flushed according to the batch flushing schedule the target is configured with.target_s3["object"]N/ASeeS3belowS3 Config.jsonFieldTypeDefaultDetailsaws_access_key_id["string"]N/Aaws_secret_access_key["string"]N/Aaws_session_token["string"]N/ASTS session token if using temporary credentialsbucket["string"]N/ABucket where staging files should be uploaded to.key_prefix["string", "null"]""Prefix for staging file uploads to allow for better delineation of tmp filesKnown LimitationsIgnoresSTATESinger messages.Requires aJSON Schemafor every stream.Only string, string with date-time format, integer, number, boolean,
object, and array types with or without null are supported. Arrays can
have any of the other types listed, including objects as types within
items.Example of JSON Schema types that work['number']['string']['string', 'null']Exmaple of JSON Schema types thatDO NOTwork['string', 'integer']['integer', 'number']['any']['null']JSON Schema combinations such asanyOfandallOfare not supported.JSON Schema $ref is partially supported:NOTE:The following limitations are known toNOTfail gracefullyPresently you cannot have any circular or recursive$refs$refs must be present within the schema:URI's do not workif the$refis broken, the behaviour is considered unexpectedAny values which are thestringNULLwill be streamed to Redshift as the literalnullTable names are restricted to:127 characters in lengthcan only be composed of_, lowercase letters, numbers,$cannot start with$ASCII charactersField/Column names are restricted to:127 characters in lengthASCII charactersFields/Columns areALLnullableFields/Columns use a default of 1000 characters (in order to be able to work with a large number of columns).Usage LoggingSinger.iorequires official taps and targets to collect anonymous usage data. This data is only used in aggregate to report on individual tap/targets, as well as the Singer community at-large. IP addresses are recorded to detect unique tap/targets users but not shared with third-parties.To disable anonymous data collection setdisable_collectiontotruein the configuration JSON file.Developingtarget-redshiftutilizessetup.pyfor package
management, andPyTestfor testing.DockerIf you haveDockerandDocker Composeinstalled, you can
easily run the following to get a local env setup quickly.First, make sure to create a.envfile in the root of this repo (it has been.gitignored so don't worry about accidentally staging it).Therein, fill out the following information:REDSHIFT_HOST='<your-host-name>'# Most likely 'localhost'REDSHIFT_DATABASE='<your-db-name>'# Most likely 'dev'REDSHIFT_SCHEMA='<your-schema-name>'# Probably 'public'REDSHIFT_PORT='<your-port>'# Probably 5439REDSHIFT_USERNAME='<your-user-name'REDSHIFT_PASSWORD='<your-password>'DEFAULT_COLUMN_LENGTH='<your-default-column-length>'TARGET_S3_AWS_ACCESS_KEY_ID='<AKIA...>'TARGET_S3_AWS_SECRET_ACCESS_KEY='<secret>'TARGET_S3_BUCKET='<bucket-string>'TARGET_S3_KEY_PREFIX='<some-string>'# We use 'target_redshift_test'$docker-composeup-d--build
$dockerlogs-tftarget-redshift_target-redshift_1# You container names might differAs soon as you seeINFO: Dev environment ready.you can shell into the container and start running test commands:$dockerexec-ittarget-redshift_target-redshift_1bash# Your container names might differroot@...$source/code/venv--target-redshift/bin/activate
root@...$pytestSee thePyTestcommands below!DBTo run the tests, you will need anactualRedshift cluster running, and a user that either:Has the ability tocreateschemas thereinThis is required if you wish to run multiple versions of the tests, similar
to how we run ourCI tests by varying theREDSHIFT_SCHEMAenvvarhas access to thepublicschemaIf theREDSHIFT_SCHEMAis seen to be the string"public", the tests will ignore creating and dropping schemasThis setup is often preferred for situations in whichGRANT CREATE ON DATABASE db TO user;is viewed as too riskyMake sure to set the following env vars forPyTest:$EXPORTREDSHIFT_HOST='<your-host-name>'# Most likely 'localhost'$EXPORTREDSHIFT_DATABASE='<your-db-name>'# Most likely 'dev'$EXPORTREDSHIFT_SCHEMA='<your-schema-name>'# Probably 'public'$EXPORTREDSHIFT_PORT='<your-port>'# Probably 5439$EXPORTREDSHIFT_USERNAME='<your-user-name'$EXPORTREDSHIFT_PASSWORD='<your-password>'# Redshift requires passwords$EXPORTDEFAULT_COLUMN_LENGTH='<your-default-column-length>'S3To run the tests, you will need anactualS3 bucket available.Make sure to set the following env vars forPyTest:$EXPORTTARGET_S3_AWS_ACCESS_KEY_ID='<AKIA...>'$EXPORTTARGET_S3_AWS_SECRET_ACCESS_KEY='<secret>'$EXPORTTARGET_S3_BUCKET='<bucket-string>'$EXPORTTARGET_S3_KEY_PREFIX='<some-string>'# We use 'target_redshift_test'PyTestTo run tests, try:$pythonsetup.pypytestIf you'vebashshelled into the Docker Compose container (see above), you should be able to simply use:$pytestSponsorshipTarget Redshift is sponsored by Data Mill (Data Mill Services, LLC)datamill.co.Data Mill helps organizations utilize modern data infrastructure and data science to power analytics, products, and services.Copyright Data Mill Services, LLC 2018