Package: singer-alto
👩‍🎤 AltoDocumentationJump into thedocshosted on GitHub Pages to get started!✨ We have a lightweight, searchabletapindex andtargetindex embedded in the docs if you want to peruse the available integrations too! All links there forward to the integration documentation on Meltano Hub. ⏩Introduction👋 Alto is a versatile data integration tool that allows you to easily runSingerplugins, build and cachePEXfiles encapsulating those plugins, and create a data reservoir 💧 whereby you can extract once and replay to as many destinations as you want as many times as you want. With Alto, you can seamlessly connect to various data sources, store your data in a centralized reservoir (singerlake), and manage lean, efficient extract load flows. Throw it into adbtproject, a data science project, or a passion project without fear of conflicting dependencies and watch itjust work. The alto config file can sit right next to your dbt project yaml with no other changes to your repo beyond addingsinger-altoto your requirements and you can be running data pipelines, today! 🎯LikeMeltano, Alto is drivenentirely by configurationand the config structure drew much of its inspiration from Meltano. Alto supports YAML, TOML, and JSON leveragingDynaconffor robust features. Because of the similarities to Meltano, using one or the other is a fairly straightforward process!InstallationI highly recommend going to thedocsto get up to speed. The docs are a work in progress but they are the best place to get started. If you are familiar with Meltano, you will feel right at home.pipinstallsinger-altoExample ConfigurationThe following is an example configuration file for Alto. It is a TOML file but you can use YAML or JSON as well. Alto will automatically detect the file type and load the configuration accordingly. I recommend using TOML for the most concise yet readable config file. I also recommend reading thedocsto get a better understanding of the config file structure.[default]project_name="{project}"load_path="raw"extensions=["evidence"]environment.STARTER_PROJECT=1# https://github.com/dlt-hub/dltutilities.dlt.pip_url="python-dlt[duckdb]>=0.2.0a25"utilities.dlt.environment.PEX_INHERIT_PATH="fallback"[default.taps]# https://gitlab.com/meltano/tap-carbon-intensitycarbon-data.pip_url="git+https://gitlab.com/meltano/tap-carbon-intensity.git#egg=tap_carbon_intensity"carbon-data.executable="tap-carbon-intensity"carbon-data.load_path="carbon_intensity"carbon-data.capabilities=["state","catalog"]carbon-data.select=["*.*","~*.dnoregion"]# https://hub.meltano.com/extractors/tap-blslabor-data.pip_url="git+https://github.com/frasermarlow/tap-bls#egg=tap_bls"labor-data.executable="tap-bls"labor-data.capabilities=["state","catalog"]labor-data.load_path="bls"labor-data.select=["JTU000000000000000JOR","JTU000000000000000JOL"]labor-data.config.startyear="2019"labor-data.config.endyear="2020"labor-data.config.calculations="true"labor-data.config.annualaverage="false"labor-data.config.aspects="false"labor-data.config.disable_collection="true"labor-data.config.update_state="false"labor-data.config.series_list_file_location="./series.json"[default.targets]# https://hub.meltano.com/loaders/target-singer-jsonljsonl.pip_url="target-jsonl==0.1.4"jsonl.executable="target-jsonl"jsonl.config.destination_path="@format output/{this.load_path}"[github_actions]load_path="cicd"targets.jsonl.config.destination_path="@format /github/workspace/output/{this.load_path}"Given the above configuration, you can run the following command to extract data from the BLS and Carbon Intensity APIs and load it into JSONL files.altocarbon-data:jsonl
altolabor-data:jsonlOr send them to the project reservoir.altocarbon-data:reservoir
altolabor-data:reservoirAnd from the reservoir, you can replay to any number of targets.altoreservoir:carbon-data-jsonl
altoreservoir:carbon-data-snowflake# here as example, not in configaltoreservoir:carbon-data-parquet# here as example, not in configLastly, you can invoke the utility defined above (you can invoke any plaugin this way).altoinvokedlt--help# invoke an executablealtoinvokepythondlt# drop into a python shell with dlt installedaltoinvokepythondlt./path/to/pipeline.py# run a python scriptComparisonHow is this different than what exists today; namely Meltano? Outside of some of what we covered above.DifferencesI would recommendaltoif you want something lighter than Meltano. If you have a Python project where EL is one of many existing concerns and you want a dependency you can add that is lean, highly functional, and can stand alongside other dependencies without concern of conflict, I would recommendalto. I would recommendaltoif you want a light snappy CLI with emphasis on less-is-more. I would recommendaltoif you want to use Singer taps and targets but don't want to deal with the Meltano system db, migrations, or system directory. I would recommendaltoif you want a codebase small enough to quickly iterate on or play with. I would recommendmeltanoif you want a tool that has a large community, more exposure, and features. I would recommendaltoif those extra features are not necessary for your use case or your just looking to explore whats out there.Alto is able to run taps -> targets with centralized environment-aware configuration, secret management, automatically managed state, automatic discovery, catalog caching to a remote backend, catalog manipulation viaselect&metadatakeys, and many of the things users love about Meltano. Given this, in most situations -- from a pure EL perspective, it stacks up well with Meltano since, given all else aside, it is the plugins that do much of the work once the state, configuration, and catalog are managed.There are some compelling features inaltoin general around how it manages plugins as cached PEX files, the built in reservoir as an available-by-default source & destination, and itslightfootprint. Continuing to use Meltano as the baseline of comparison (since it was the inspiration!), here are some noteworthy differences:The CLI is extremely fast due to the lightness of the package.There is no system db so no database migrations or system directory to care about.Significantly smaller dependency footprint by anorder of magnitude. Alto only has 4 direct dependencies with no C or rust extensions in the dependency tree, it is pure python. The below comparison includes transitives:Meltano: >100Alto: 7Because of its dependency footprint, it can be installed in very tiny Docker containers and wheels are cross platform compatible, naturally. It installs extremely quickly.We usePEX(PythonEXecutable) for all plugins instead of loose venvs making plugins single files that are straightforward to cache.We use a (simple) caching algorithm that makes the plugins re-usable across machines when combined with a remote filesystem and re-usable on the same system in general. This means, most of the time, you will build a PEX artifact once andneverbuild it again. This makes an already lightweightaltoeven more portable. The benefits are applicable for your whole team.Docker containers do not require you to "install" the plugins during the build process since the plugins are instantly pulled from a remote cache. This cansignificantlyreduce image size if you are working with enough plugins.Because of how plugins are handled it can be ran inlambdaand serverless functions very easily. The time to spin up a pipeline is extremely quick.We usefsspecto provide that filesystem abstraction layer that provides the exact same experience locally on a single machine as when plugged into a remote fsspec filesystem such ass3,gcs, orazure. We do not pin these remote backend dependencies, even as extras, but give the user the flexibility to include how they see fit.An order of magnitude (>85%) less code which makes iteration/maintenance, extending, or forking easier (in theory) due to less accumulated tech debt though the flip side is less robustnessBecause it is scaffolded over a build system, never worry about runninginstallagain, run pipelines immediately and alto works out the rest.We useDynaconfto manage configurationThis gives us uniform support for json, toml, and yaml out of the boxWe get environment managementWe get configuration inheritance / deep mergingWe get.envsupportWe get unique ways to render vars with@formattokensWe get hashicorp vault supportEncourages use ofbashinstead ofmeltano runcommands. Bash isalreadya fantastic glue code where you can run multiple extract load blocks, background them via&to parallelize loads, run utilities the way they have always been ran since everything is not wrapped in a venv with env vars injected by Meltano which is both a convenience and a constraint.meltano run tap1 target1 tap2 target2is ~ functionally identical toalto tap1:target1 && alto tap2:target2.Straightforward to take advantage of the Python API exposed byaltoto build your own custom pipelines.A smaller codebase affords agility and flexibility. It has let us prove out integration withdltwhich is exciting!Closing RemarksRemember, everything in software is a tradeoff. I offeraltoas another tool in the ecosystem which gets the job done in a radically different way. I hope you find it useful and if you do, please consider contributing to the project or having the interesting discussions about what the ideal is. I am also happy to help with any questions you may have about the project.Hats off to Meltano 🎉 for being a great inspiration and for being a great tool that helped to shape the ideal of what I wanted to build. I hope you find this interesting and divergent enough to warrant a look.