Package: meltanolabs-target-postgres
target-postgresTarget for Postgres.Built with theMeltano SDKfor Singer Taps and Targets.Capabilitiesaboutstream-mapsschema-flatteningSettingsSettingRequiredDefaultDescriptionhostFalseNoneHostname for postgres instance. Note if sqlalchemy_url is set this will be ignored.portFalse5432The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored.userFalseNoneUser name used to authenticate. Note if sqlalchemy_url is set this will be ignored.passwordFalseNonePassword used to authenticate. Note if sqlalchemy_url is set this will be ignored.databaseFalseNoneDatabase name. Note if sqlalchemy_url is set this will be ignored.sqlalchemy_urlFalseNoneSQLAlchemy connection string. This will override using host, user, password, port, dialect, and all ssl settings. Note that you must escape password special characters properly. Seehttps://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwordsdialect+driverFalsepostgresql+psycopg2Dialect+driver seehttps://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone. Note if sqlalchemy_url is set this will be ignored.default_target_schemaFalsemeltyPostgres schema to send data to, example: tap-clickupactivate_versionFalseTrueIf set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well.hard_deleteFalseFalseWhen activate version is sent from a tap this specefies if we should delete the records that don't match, or mark them with a date in the_sdc_deleted_atcolumn. This config option is ignored ifactivate_versionis set to false.add_record_metadataFalseTrueNote that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. Seehttps://sdk.meltano.com/en/latest/implementation/record_metadata.htmlfor more information.ssl_enableFalseFalseWhether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored.ssl_client_certificate_enableFalseFalseWhether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored.ssl_modeFalseverify-fullSSL Protection method, seepostgres documentationfor more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored.ssl_certificate_authorityFalse~/.postgresql/root.crlThe certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored.ssl_client_certificateFalse~/.postgresql/postgresql.crtThe certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored.ssl_client_private_keyFalse~/.postgresql/postgresql.keyThe private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored.ssl_storage_directoryFalse.secretsThe folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created.ssh_tunnelFalseNoneSSH Tunnel Configuration, this is a json objectssh_tunnel.enableTrue (if ssh_tunnel set)FalseEnable an ssh tunnel (also known as bastion host), see the other ssh_tunnel.* properties for more details.ssh_tunnel.hostTrue (if ssh_tunnel set)FalseHost of the bastion host, this is the host we'll connect to via sshssh_tunnel.usernameTrue (if ssh_tunnel set)FalseUsername to connect to bastion hostssh_tunnel.portTrue (if ssh_tunnel set)22Port to connect to bastion hostssh_tunnel.private_keyTrue (if ssh_tunnel set)NonePrivate Key for authentication to the bastion hostssh_tunnel.private_key_passwordFalseNonePrivate Key Password, leave None if no password is setstream_mapsFalseNoneConfig object for stream maps capability. For more information check outStream Maps.stream_map_configFalseNoneUser-defined config values to be used within map expressions.flattening_enabledFalseNone'True' to enable schema flattening and automatically expand nested properties.flattening_max_depthFalseNoneThe max depth to flatten schemas.A full list of supported settings and capabilities is available by running:target-postgres --aboutInstallationpipxinstallmeltanolabs-target-postgresConfigurationAn Explanation of Various SSL Configuration OptionsThere are two distinct processes which both fall under the banner of SSL. One process occurs when the client wishes to ensure the identity of the server, and is the more common reason that SSL is used. Another is when the server wishes to ensure the identity of the client, for authentication/authorization purposes.If your server is set up with a certificate and private key, and you wish to check their certificate against a root certificate which you posess, usessl_enable. You may then further customize this process using thessl_certificate_authorityandssl_modesettings. See thedocumentationfor further details.If your server is set up with a root certificate, and you wish to provide a certificate to the server to verify your identity, usessl_client_certificate_enable. You may then further customize this process using thessl_client_certificateandssl_client_private_keysettings. See thedocumentationfor further details.Configure using environment variablesThis Singer target will automatically import any environment variables within the working directory's.envif the--config=ENVis provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the.envfile.Source Authentication and AuthorizationThe database account provided must have access to:Create schemasCreate tables (DDL)Push Data to tables (DML)UsageYou can easily runtarget-postgresby itself or in a pipeline usingMeltano.Executing the Target Directlytarget-postgres--version
target-postgres--help# Test using the "Carbon Intensity" sample:pipxinstallgit+https://gitlab.com/meltano/tap-carbon-intensity
tap-carbon-intensity|target-postgres--config/path/to/target-postgres-config.jsonUsing Docker Composedocker-compose.ymlprovides the commands to create two empty sample databases usingDocker. These can be a starting point to create your own database running in Docker, or can be used to run the tap'sbuilt-in tests.Developer ResourcesInitialize your Development Environmentpipxinstallpoetry
poetryinstall
pipxinstallpre-commit
pre-commitinstallCreate and Run TestsCreate tests within thetarget_postgres/testssubfolder and
then run:poetryrunpytestYou can also test thetarget-postgresCLI interface directly usingpoetry run:poetryruntarget-postgres--helpTesting withMeltanoNote:This target will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios.Your project comes with a custommeltano.ymlproject file already created.Next, install Meltano (if you haven't already) and any needed plugins:# Install meltanopipxinstallmeltano# Initialize meltano within this directorymeltanoinstallNow you can test and orchestrate using Meltano:# Test invocation:meltanoinvoketarget-postgres--versionSDK Dev GuideSee thedev guidefor more instructions on how to use the Meltano SDK to
develop your own Singer taps and targets.Data TypesThe below table shows how this tap will map between jsonschema datatypes and Postgres datatypes.jsonschemaPostgresintegerbigintUNSUPPORTEDbigserialUNSUPPORTEDbit [ (n) ]UNSUPPORTEDbit varying [ (n) ]booleanbooleanUNSUPPORTEDboxUNSUPPORTEDbyteaUNSUPPORTEDcharacter [ (n) ]UNSUPPORTEDcharacter varying [ (n) ]UNSUPPORTEDcidrUNSUPPORTEDcirclestring with format="date"dateUNSUPPORTEDdouble precisionUNSUPPORTEDinetUNSUPPORTEDintegerUNSUPPORTEDinterval [ fields ] [ (p) ]UNSUPPORTEDjsonarray; objectjsonbUNSUPPORTEDlineUNSUPPORTEDlsegUNSUPPORTEDmacaddrUNSUPPORTEDmacaddr8UNSUPPORTEDmoneynumbernumeric [ (p, s) ]UNSUPPORTEDpathUNSUPPORTEDpg_lsnUNSUPPORTEDpg_snapshotUNSUPPORTEDpointUNSUPPORTEDpolygonUNSUPPORTEDrealUNSUPPORTEDsmallintUNSUPPORTEDsmallserialUNSUPPORTEDserialstring without format; untypedtextstring with format="time"time [ (p) ] [ without time zone ]UNSUPPORTEDtime [ (p) ] with time zonestring with format="date-time"timestamp [ (p) ] [ without time zone ]UNSUPPORTEDtimestamp [ (p) ] with time zoneUNSUPPORTEDtsqueryUNSUPPORTEDtsvectorUNSUPPORTEDtxid_snapshotUNSUPPORTEDuuidUNSUPPORTEDxmlNote that while object types are mapped directly to jsonb, array types are mapped to a jsonb array.If a column has multiple jsonschema types, the following order is using to order Postgres types, from highest priority to lowest priority.ARRAY(JSONB)JSONBTEXTTIMESTAMPDATETIMEDATETIMEDECIMALBIGINTINTEGERBOOLEANNOTYPE