Package: singer-target-postgres
Target PostgresASingerpostgres target, for use with Singer streams generated by Singer taps.FeaturesCreates SQL tables forSingerstreamsDenests objects flattening them into the parent object's tableDenests rows into separate tablesAdds columns and sub-tables as new fields are added to the streamJSON SchemaFull stream replication via recordversionandACTIVATE_VERSIONmessages.Installpipinstallsinger-target-postgresUsageFollow theSinger.io Best Practicesfor setting up separatetapandtargetvirtualenvs to avoid version
conflicts.Create aconfig fileat~/singer.io/target_postgres_config.jsonwith postgres connection
information and target postgres schema.{"postgres_host":"localhost","postgres_port":5432,"postgres_database":"my_analytics","postgres_username":"myuser","postgres_password":"1234","postgres_schema":"mytapname"}Runtarget-postgresagainst aSingertap.~/.virtualenvs/tap-something/bin/tap-something\|~/.virtualenvs/target-postgres/bin/target-postgres\--config~/singer.io/target_postgres_config.json>>state.jsonIf you are running windows, the following is equivalent:venvs\tap-exchangeratesapi\Scripts\tap-exchangeratesapi.exe | ^
venvs\target-postgresql\Scripts\target-postgres.exe ^
--config target_postgres_config.jsonConfig.jsonThe fields available to be specified in the config file are specified
here.FieldTypeDefaultDetailspostgres_host["string", "null"]"localhost"postgres_port["integer", "null"]5432postgres_database["string"]N/Apostgres_username["string", "null"]N/Apostgres_password["string", "null"]nullpostgres_schema["string", "null"]"public"postgres_sslmode["string", "null"]"prefer"Refer to thelibpqdocs for more information about SSLpostgres_sslcert["string", "null"]"~/.postgresql/postgresql.crt"Only used if a SSL request w/ a client certificate is being madepostgres_sslkey["string", "null"]"~/.postgresql/postgresql.key"Only used if a SSL request w/ a client certificate is being madepostgres_sslrootcert["string", "null"]"~/.postgresql/root.crt"Used for authentication of a server SSL certificatepostgres_sslcrl["string", "null"]"~/.postgresql/root.crl"Used for authentication of a server SSL certificateinvalid_records_detect["boolean", "null"]trueIncludefalsein your config to disabletarget-postgresfrom crashing on invalid recordsinvalid_records_threshold["integer", "null"]0Include a positive valuenin your config to allow fortarget-postgresto encounter at mostninvalid records per stream before giving up.disable_collection["string", "null"]falseIncludetruein your config to disableSinger Usage Logging.logging_level["string", "null"]"INFO"The level for logging. Set toDEBUGto get things like queries executed, timing of those queries, etc. SeePython's Logger Levelsfor information about valid values.persist_empty_tables["boolean", "null"]FalseWhether the Target should create tables which have no records present in Remote.max_batch_rows["integer", "null"]200000The maximum number of rows to buffer in memory before writing to the destination table in Postgresmax_buffer_size["integer", "null"]104857600(100MB in bytes)The maximum number of bytes to buffer in memory before writing to the destination table in Postgresbatch_detection_threshold["integer", "null"]5000, or 1/40thmax_batch_rowsHow often, in rows received, to count the buffered rows and bytes to check if a flush is necessary. There's a slight performance penalty to checking the buffered records count or bytesize, so this controls how often this is polled in order to mitigate the penalty. This value is usually not necessary to set as the default is dynamically adjusted to check reasonably often.state_support["boolean", "null"]TrueWhether the Target should emitSTATEmessages to stdout for further consumption. In this mode, which is on by default, STATE messages are buffered in memory until all the records that occurred before them are flushed according to the batch flushing schedule the target is configured with.add_upsert_indexes["boolean", "null"]TrueWhether the Target should create column indexes on the important columns used during data loading. These indexes will make data loading slightly slower but the deduplication phase much faster. Defaults to on for better baseline performance.before_run_sql["string", "null"]NoneRaw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target. Useful for setup likeSET ROLEor other connection state that is important.after_run_sql["string", "null"]NoneRaw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target. Useful for setup likeSET ROLEor other connection state that is important.Supported Versionstarget-postgresonly supportsJSON Schema Draft4.
While declaring a schemais optional, any input schema which declares a version
other than 4 will be rejected.target-postgressupports all versions of PostgreSQL which are presently supported
by the PostgreSQL Global Development Group. OurCI configdefines all versions we are currently supporting.VersionCurrent minorSupportedFirst ReleaseFinal Release1212.2YesOctober 3, 2019November 14, 20241111.7YesOctober 18, 2018November 9, 20231010.12YesOctober 5, 2017November 10, 20229.69.6.17YesSeptember 29, 2016November 11, 20219.59.5.21YesJanuary 7, 2016February 11, 20219.49.4.26YesDecember 18, 2014February 13, 20209.39.3.25NoSeptember 9, 2013November 8, 2018The above is copied from thecurrent list of versionson Postgresql.orgKnown LimitationsRequires aJSON Schemafor every stream.Only string, string with date-time format, integer, number, boolean,
object, and array types with or without null are supported. Arrays can
have any of the other types listed, including objects as types within
items.Example of JSON Schema types that work['number']['string']['string', 'null']Exmaple of JSON Schema types thatDO NOTwork['string', 'integer']['integer', 'number']['any']['null']JSON Schema combinations such asanyOfandoneOfare not supported.JSON Schema $ref is partially supported:NOTE:The following limitations are known toNOTfail gracefullyPresently you cannot have any circular or recursive$refs$refs must be present within the schema:URI's do not workif the$refis broken, the behaviour is considered unexpectedAny values which are thestringNULLwill be streamed to PostgreSQL as the literalnullTable names are restricted to:63 characters in lengthcan only be composed of_, lowercase letters, numbers,$cannot start with$ASCII charactersField/Column names are restricted to:63 characters in lengthASCII charactersIndexesIf theadd_upsert_indexesconfig option is enabled, which it is by default,target-postgresadds indexes on the tables it creates for its own queries to be more performant. Specifically,target-postgresautomatically adds indexes to the_sdc_sequencecolumn and the_sdc_level_<n>_idcolumns which are used heavily when inserting and upserting.target-postgresdoesn't have any facilities for adding other indexes to the managed tables, so if there are more indexes required, they should be added by another downstream tool, or can just be added by an administrator when necessary. Note that these indexes incur performance overhead to maintain as data is inserted, These indexes can also preventtarget-postgresfrom dropping columns in the future if the schema of the table changes, in which case an administrator should drop the index sotarget-postgresis able to drop the columns it needs to.Note: Index adding is new as of version0.2.1, andtarget-postgresdoes not retroactively create indexes for tables it created before that time. If you want to add indexes to older tablestarget-postgresis loading data into, they should be added manually.Usage LoggingSinger.iorequires official taps and targets to collect anonymous usage data. This data is only used in aggregate to report on individual tap/targets, as well as the Singer community at-large. IP addresses are recorded to detect unique tap/targets users but not shared with third-parties.To disable anonymous data collection setdisable_collectiontotruein the configuration JSON file.Developingtarget-postgresutilizessetup.pyfor package
management, andPyTestfor testing.DocumentationSee also:DECISIONS: A document containing high level explanations of various decisions and decision making paradigms. A good place to request more explanation/clarification on confusing things found herein.TableMetadata: A document detailing some of the metadata necessary forTargetPostgresto function correctly on the RemoteDockerIf you haveDockerandDocker Composeinstalled, you can
easily run the following to get a local env setup quickly.$docker-composeup-d--build
$dockerlogs-tftarget-postgres_target-postgres_1# You container names might differAs soon as you seeINFO: Dev environment ready.you can shell into the container and start running test commands:$docker-composeexectarget-postgresbash(target-postgres)root@...:/code#pytestThe environment inside the docker container has a virtualenv set up and activated, with an--editableinstall oftarget-postgresinside it and your local code mounted as a Docker volume. If you make changes on your host and re-runpytestany changes should be reflected inside the container.See thePyTestcommands below!DBTo run the tests, you will need a PostgreSQL server running.NOTE:Testing assumes that you've exposed the traditional port5432.Make sure to set the following env vars forPyTest:$EXPORTPOSTGRES_HOST='<your-host-name>'# Most likely 'localhost'$EXPORTPOSTGRES_DB='<your-db-name>'# We use 'target_postgres_test'$EXPORTPOSTGRES_USER='<your-user-name'# Probably just 'postgres', make sure this user has no authPyTestTo run tests, try:$pythonsetup.pypytestIf you'vebashshelled into the Docker Compose container (see above), you should be able to simply use:$pytestCollaboration and ContributionsJoin the conversation over at theSinger.io Slackand on the#target-postgreschannel.Try to adhere to the following for contributing:File New Issue -> Fork -> New Branch(If needed) -> Pull Request -> Approval -> MergeUsers can file an issue without submitting a pull request but be aware not all issues can or will be addressed.SponsorshipTarget Postgres is sponsored by Data Mill (Data Mill Services, LLC)datamill.co.Data Mill helps organizations utilize modern data infrastructure and data science to power analytics, products, and services.Copyright Data Mill Services, LLC 2018