Package: meltano-target-cratedb
Meltano/Singer Target for CrateDBAboutASingertarget forCrateDB, built with theMeltano SDKfor custom extractors
and loaders, and based on theMeltano PostgreSQL target. It connects a library of600+ connectorswith CrateDB, and vice versa.In Singer ELT jargon, a "target" conceptually wraps a data sink, where you
"load" data into.Singer, Meltano, and PipelineWise provide foundational components and
an integration engine for composable Open Source ETL with600+ connectors.
On the database integration side, they are heavily based onSQLAlchemy.CrateDBCrateDBis a distributed and scalable SQL database for storing and analyzing
massive amounts of data in near real-time, even with complex queries. It is
PostgreSQL-compatible, and based onApache Lucene.CrateDB offers a Python SQLAlchemy dialect, in order to plug into the
comprehensive Python data-science and -wrangling ecosystems.SingerThe open-source standard for writing scripts that move data.Singeris an open source specification and software framework forETL/ELTdata exchange between a range of different systems. For talking to SQL databases,
it employs a metadata subsystem based on SQLAlchemy.Singer reads and writes Singer-formatted messages, following theSinger Spec.
Effectively, those are JSONL files.MeltanoUnlock all the data that powers your data platform.Say goodbye to writing, maintaining, and scaling your own API integrations
with Meltano's declarative code-first data integration engine, bringing
600+ APIs and DBs to the table.Meltanobuilds upon Singer technologies, uses configuration files in YAML
syntax instead of JSON, adds an improved SDK and other components, and runs
the central addon registry,meltano | Hub.PipelineWisePipelineWiseis another Data Pipeline Framework using the Singer.io
specification to ingest and replicate data from various sources to
various destinations. The list ofPipelineWise Tapsinclude another
20+ high-quality data-source and -sink components.SQLAlchemySQLAlchemyis the leading Python SQL toolkit and Object Relational Mapper
that gives application developers the full power and flexibility of SQL.It provides a full suite of well known enterprise-level persistence patterns,
designed for efficient and high-performing database access, adapted into a
simple and Pythonic domain language.InstallUsually, you will not install this package directly, but on behalf
of a Meltano definition instead, for example. A corresponding snippet
is outlined in the next section. After adding it to yourmeltano.ymlconfiguration file, you can install all defined components and their
dependencies.meltano installUsageYou can run the CrateDB Singer targettarget-cratedbby itself, or
in a pipeline using Meltano.MeltanoUsing themeltano addsubcommand, you can add the plugin to your
Meltano project.meltanoaddloadertarget-cratedbNB: It will only work like this when released and registered on Meltano Hub.
In the meanwhile, please add the configuration snippet manually.CrateDB CloudIn order to connect toCrateDB Cloud, configure thesqlalchemy_urlsetting
within yourmeltano.ymlconfiguration file like this.-name:target-cratedbnamespace:cratedbvariant:cratedbpip_url:meltano-target-cratedbconfig:sqlalchemy_url:"crate://admin:K4IgMXNvQBJM3CiElOiPHuSp6CiXPCiQYhB4I9dLccVHGvvvitPSYr1vTpt4@example.aks1.westeurope.azure.cratedb.net:4200?ssl=true"}add_record_metadata:trueOn localhostIn order to connect to a standalone or on-premise instance of CrateDB, configure
thesqlalchemy_urlsetting within yourmeltano.ymlconfiguration file like this.-name:target-cratedbnamespace:cratedbvariant:cratedbpip_url:meltano-target-cratedbconfig:sqlalchemy_url:crate://crate@localhost/add_record_metadata:trueThen, invoke the pipeline by usingmeltano run, similar like this.meltanoruntap-xyztarget-cratedbStandaloneYou can also invoke it standalone by using thetarget-cratedbprogram.
This example demonstrates how to load a file into the database.First, acquire an example file in Singer format, including the list of
countries of the world.wgethttps://github.com/MeltanoLabs/target-postgres/raw/v0.0.9/target_postgres/tests/data_files/tap_countries.singerNow, define the database connection string including credentials in
SQLAlchemy format.echo'{"sqlalchemy_url": "crate://crate@localhost/"}'>settings.jsonBy using Unix pipes, load the data file into the database, referencing
the path to the configuration file.cattap_countries.singer|target-cratedb--config=settings.jsonUsing the interactive terminal program,crash, you can run SQL
statements on CrateDB.pipinstallcrash
crash--hostslocalhost:4200Now, you can verify that the data has been loaded correctly.SELECT"code","name","capital","emoji","languages[1]"FROM"melty"."countries"ORDERBY"name"LIMIT42;DevelopmentIn order to work on this adapter dialect on behalf of a real pipeline definition,
link your sandbox to a development installation ofmeltano-target-cratedb, and
configure thepip_urlof the component to point to a different location than thevanilla package on PyPI.Use this URL to directly point to a specific Git repository reference.pip_url:git+https://github.com/crate-workbench/meltano-target-cratedb.git@mainUse apip-like notation to link the CrateDB Singer target in development mode,
so you can work on it at the same time while running the pipeline, and iterating
on its definition.pip_url:--editable=/path/to/sources/meltano-target-cratedb