Must Know Trends about Modern Data Stack

## Modern Data Stack

In the data world, the modern data stack has been talking for recent years. What is the modern data stack? 

From [The Modern Data Experience](https://benn.substack.com/p/the-modern-data-experience)

>To analytics engineers, it’s a transformational shift in [technology](https://blog.getdbt.com/future-of-the-modern-data-stack/) and [company organization](https://jasnonaz.medium.com/analytics-engineering-everywhere-d56f363da625). To startup founders, it’s a [revolution in how companies work](https://www.rilldata.com/blog/5-founders-define-the-modern-data-stack). 
To VCs, it’s a [$100 billion opportunity](https://www.prnewswire.com/news-releases/fishtown-analytics-raises-12-9m-to-accelerate-open-source-analytics-engineering-software-dbt-301045082.html). To engineers, it’s a [dynamic architectural roadmap](https://blog.getcensus.com/graduating-to-the-modern-data-stack-for-startups/)
To Gartner, it’s the [foundation of a new data and analytics strategy](https://www.gartner.com/en/conferences/na/data-analytics-us/agenda)
To thought leaders, it’s a [data mesh](https://martinfowler.com/articles/data-monolith-to-mesh.html)
To an analyst with an indulgent blog on the internet, it’s a [new orientation](https://benn.substack.com/p/datas-horizontal-pivot), a [new nomenclature](https://benn.substack.com/p/gerrymandering), and [a](https://benn.substack.com/p/self-serve-is-a-feeling) [bunch](https://benn.substack.com/p/big-whiff) [of](https://benn.substack.com/p/analytics-is-at-a-crossroads) [other](https://benn.substack.com/p/metrics-layer) [esoteric](https://benn.substack.com/p/minerva-metrics-layer) [analogies](https://benn.substack.com/p/analytics-is-a-mess) that only someone living deep within their own navel would care about.
> 

From [Ananth Packkildurai](https://twitter.com/ananthdurai/status/1479111266866139136?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1479111266866139136%7Ctwgr%5Ehb_0_8%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fdagster.io%2Fblog%2Frebundling-the-data-platform)

> MDS is a set of vendor tools that solve niche data problems (lineage, orchestration, quality) with the side effect of creating a disjointed data workflow that makes data folks lives’ more complicated.

From [Fivetran](https://www.fivetran.com/blog/what-is-the-modern-data-stack)

> A radically new approach to data integration saves engineering time, allowing engineers and analysts to pursue higher-value activities.



## **Work Like an Analyst**

https://benn.substack.com/p/work-like-an-analyst


### Data Analysis Democratization

**We thought only people who has data- title need to analyze data,** in the real world, however, there are **many low-code/no-code tools to empower other people from business** to techniques to do the certain analysis by own demands. The **data analysis is being democratized** essentially in organizations, whereas these diversified tools complicate the modern data stack.

> Analysis is becoming a team sport. Business stakeholders aren’t kicking questions over to their data help desks and waiting to get reports back to review; everyone is contributing together.
> 

> [Tristan Handy predicted](https://roundup.getdbt.com/p/data-expertise-everywhere) that analytical talent wouldn’t be concentrated in a “select few specialists,” but would “live inside of operational areas of the business.”
> 

> Analysis becoming multi-functional; the democratization of analytical reasoning and technical talent, not just data—is far more foundational than whatever reshuffling happens among the data stack’s middleware.
> 

> I’m now realizing that misses the real point: The divisions of labor between analysts and everyone else are fading. Analysis is getting bundled with other functions; the behaviors of analysts and non-analysts are overlapping; analysts are becoming [positionless](https://wraptext.equals.app/every-analyst-is-a-finance-analyst/)
> 

> The reason to build a “[modern data experience](https://benn.substack.com/p/the-modern-data-experience)” isn’t to unify the disjointed products of a bunch of startups; it’s to serve a world in which far more people want to work like analysts.
> 

### More

https://benn.substack.com/p/the-modern-data-experience


## The Unbundling of Airflow

https://blog.fal.ai/the-unbundling-of-airflow-2/


> Heavy users of Airflow can do a vast variety of data related tasks without leaving the platform; from extract and load scripts to generating reports, transformations with Python and SQL to syncing back data to BI tools.
> 

> Before the fragmentation of the data stack, it wasn’t uncommon to create end-to-end pipelines with Airflow. Organizations used to build almost entire data workflows as custom scripts developed by in-house data engineers. Bigger companies even built their own frameworks inside Airflow, for example frameworks with dbt-like functionality for SQL transformations in order to make it easier for data analysts to write these pipelines.
> 

> Today, data practitioners have many tools under their belt and only very rarely they have to reach for a tool like Airflow. [Fivetran](https://www.fivetran.com/) and [Airbyte](https://airbyte.com/) took care of the extract and load scripts that one might write with Airflow. [dbt](https://www.getdbt.com/) came for the data transformations, [Census](https://blog.getcensus.com/) and [Hightouch](https://hightouch.io/) for Reverse ETL. Metrics and experimentation layers are also getting their own focused tooling; metrics with tools like [Transform](https://transform.co/), [Metriql](https://metriql.com/), [Supergrain](https://www.supergrain.com/)  and experimentation with [Eppo](https://www.geteppo.com/). [Certain companies relied on Airflow for data science and ML workloads,](https://www.slideshare.net/KarthikMurugesan2/airbnbs-ml-platform-bighead) but with the popularity of MLOps, that layer is also being abstracted out. Open source tools like [Feast](https://feast.dev/) are unbundling best practices for feature management that used to exist as independent Python scripts.
> 

## The Epic Debate: **Bundling or Unbundling**

https://www.dataengineeringweekly.com/p/bundling-vs-unbundling-the-tale-of

https://metadataweekly.substack.com/p/metadata-weekly-17



### The viewpoint of bundling

> Most importantly, customers will start demanding less complexity as they make choices about their data stack. This is where bundling will start to win.

> We need to merge both the model and task execution unit into one unit. Otherwise, any abstraction we build without the unification will further amplify the disorganization of the data.

### The viewpoint of unbundling

> The only reality in the data world is diversity — data engineers, analysts, analytics engineers, data scientists, product managers, business analysts, citizen data scientists, and more. 
Each of these people have their own favorite and equally diverse data tools, everything from SQL, Looker, and Jupyter to Python, Tableau, dbt, and R. And data projects have their own technical requirements and peculiarities — some need real-time processing while some need speed for ad-hoc analysis, leading to a whole host of data infrastructure technologies (warehouses, lakehouses, and everything in between).
> 

> We believe that the key to helping our data stack work together is in **activating metadata.** One use case for metadata activation could be as simple as **notifying downstream consumers of upstream changes.** 
When a data store changes:
> 
> 1. **Refresh metadata**: Crawl the data store to retrieve its updated metadata.
> 2. **Detect changes**: Compare the new metadata against the previous metadata. Identify any changes that could cause an impact — adding or removing columns, for example.
> 3. **Find dependencies**: Use lineage to find users of the data store. These could include transformation processes, other data stores, BI dashboards, and so on.
> 4. **Notify consumers**: Notify each consumer through their preferred communication channel — Slack, Jira, etc.

### The viewpoint of the cycle of the both

> The challenge with a fully bundled stack is that resources are always limited and innovation stalls. This gap will create an opportunity for unbundling, and so I believe we’ll go through [cycles of bundling and unbundling](https://hbr.org/2014/06/how-to-succeed-in-business-by-bundling-and-unbundling).
> 

### More

https://www.dataengineeringweekly.com/p/bundling-vs-unbundling-the-tale-of-92d


## **Introducing Software-Defined Assets**

https://dagster.io/blog/software-defined-assets


One thing worths to be mentioned that Dagster advocates declarative approaches by introducing software-defined assets that bring many advantages to the modern data stack.

> **Declarative approaches** are appealing because they make systems **dramatically more debuggable, comprehensible, and automate-able.** They do this by making intentions explicit and by offering a principled way of managing change. By explicitly defining what the world should look like — e.g. by specifying the set of servers that should exist — it becomes easy to discover when it does not look like that, reason about why, and reconcile.
> 

> **Using code to define the data assets that you want to exist.** These asset definitions, version-controlled through git and inspectable via tooling, allow anyone in your organization to understand your canonical set of data assets, allow you to reproduce them at any time, and offer a foundation for asset-based orchestration.
> 

> An asset definition can be invoked by the orchestrator to materialize that asset, i.e. to run the op on the contents of the upstream assets and then persist the results in storage, like a data warehouse, data lake, or ML model store.
> 

> In large part due to its embrace of declarative data management, the Modern Data Stack has brought immense quality-of-life improvements to data practitioners. But these improvements come with some glaring gaps:
> 
> - **Python and other non-SQL programming languages have largely been left behind.** As soon as you want to transform data with Pandas or PySpark, write a custom ingest script, or train an ML model, you’re back to writing imperative code in Airflow.
> - As we discussed at length in our **["Rebundling the Data Platform" post](https://dagster.io/blog/rebundling-the-data-platform)** last week, segregating ingest, SQL transformation, and ML into different purpose-build tools can mean losing sight of the asset graph that spans all of them. **The lack of a shared orchestration layer results in an operationally fragile data platform** that fosters a constant state of confusion about what ran, what's supposed to run, and whether things ran in the right order.

> **The Modern Data Stack has already started to embrace declarative, asset-based principles.** Because it defines dependencies and scheduling policies at the asset level instead of at the task level, an asset-based orchestrator is the ideal orchestrator for this stack. Dagster brings in heterogeneous compute and a unified control plane, without requiring practitioners to revert to tasks and imperative programming.
> 

### More

https://roundup.getdbt.com/p/disjointed-lineage

https://medium.com/whatnot-engineering/building-a-modern-data-stack-at-whatnot-afc1d03c3f9

## The dbt Viewpoint

https://docs.getdbt.com/docs/about/viewpoint/

dbt, the well-known data build tool in the data world. dbt enables analytics engineers to **transform data in their warehouses by simply writing select statements.** dbt handles turning these select statements into tables and views.

The viewpoint of dbt is to **build a mature analytics workflow.**

**Analytics**

> The center of gravity in mature analytics organizations has shifted away from proprietary, end-to-end tools towards more **composable solutions** made up of:

- data integration scripts and/or tools,
- high-performance analytic databases,
- SQL, R, and/or Python, and
- visualization tools.
> 

Analytics is Collaborative

> Analysis changes as data and businesses evolve, and **it’s important to know who changed what, when.**

> Any code that generates data or analysis should be reviewed and tested.

> Your analysis is a software application, and, like every other software application, people are going to have questions about how to use it.
Your code should come packaged with a basic description of how it should be interpreted, and your team should be able to add to that documentation as additional questions arise.

> **Think of the schema of a data set as its public interface**. Create tables, views, or other data sets that expose a consistent schema and can be modified if business logic changes.

> Analytic code, however, is often fragile. Changes in underlying data break most analytic code in ways that are hard to predict and to fix.

> an **automated workflow**:
> 
> - models and analysis are downloaded from multiple source control repositories,
> - code is configured for the given environment,
> - code is tested, and
> - code is deployed.

### More

https://blog.getcensus.com/dbt-the-etl-elt-disrupter/


## The Future of the Modern Data Stack in 2022

Metric Layer and Headless BI

https://humansofdata.atlan.com/2022/01/the-future-of-the-modern-data-stack-in-2022/


Data Mesh

> Its core idea is that companies can become more data-driven by moving from centralized data warehouses and lakes to a “domain-oriented decentralized data ownership and architecture” driven by self-serve data and “federated computational governance”.
> 

> The data mesh isn’t a platform or a service that you can buy off the shelf. It’s a design concept with some wonderful concepts like distributed ownership, domain-based design, data discoverability, and data product shipping standards — all of which are worth trying to operationalize in your organization.
> 

Metrics Layer

> It’s been called the **metrics layer**, **metrics store**, **headless BI**, and even more names than I can list here.
> 

> Airbnb announced that it had been building a home-grown metrics platform called **[Minerva](https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70)**
 to solve this issue. Other prominent tech companies soon followed suit, including LinkedIn’s [Unified Metrics Platform](https://engineering.linkedin.com/unified-metrics-platform), Uber’s [uMetric](https://eng.uber.com/umetric/), and Spotify’s metrics catalog in their “[new experimentation platform](https://engineering.atspotify.com/2020/10/29/spotifys-new-experimentation-platform-part-1/)”.
> 

> a bunch of early stage startups have launched to compete for this space. **[Transform](https://transform.co/)**
 is probably the biggest name so far, but **[Metriql](https://metriql.com/blog/introducing-metriql-open-source-metrics-store/)**, **[Lightdash](https://www.lightdash.com/)**, **[Supergrain](https://www.supergrain.com/blog/introducing-supergrain)**, and **[Metlo](https://blog.metlo.com/announcing-our-metrics-store/)**
 also launched this year. Some bigger names are also pivoting to compete in the metrics layer, such as **GoodData**’s foray into [Headless BI](https://www.gooddata.com/headless-bi/).
> 

Reverse ETL

> This concept first started getting attention in February, when **Astasia Myers**
 (Founding Enterprise Partner at [Quiet Capital](https://www.quiet.com/)) wrote an article about the [emergence of reverse ETL](https://medium.com/memory-leak/reverse-etl-a-primer-4e6694dcc7fb).
> 

> Hightouch and Census have dominated the reverse ETL discussion this year, but they’re not the only ones in the space. Other notable companies are **[Grouparoo](https://www.grouparoo.com/)**, **[HeadsUp**,](https://www.headsup.ai/) **[Polytomic](https://techcrunch.com/2021/02/03/polytomic-announces-2-4m-seed-to-move-business-data-where-its-needed/)**, **[Rudderstack](https://rudderstack.com/)**, and **[Workato](https://www.workato.com/)**.
> 

Metadata & Data Catalogs

> This idea got amplified by a [huge move](https://towardsdatascience.com/the-gartner-magic-quadrant-for-metadata-management-was-just-scrapped-d84b2543f989) **Gartner** made this year — scrapping its Magic Quadrant for Metadata Management Solutions and replacing it with the Market Guide for Active Metadata. In doing this, they introduced “**active metadata**” as a new category in the data space.
What’s the difference? Old-school data catalogs collect metadata and bring them into a siloed “passive” tool, aka the traditional data catalog. Active metadata platforms act as two-way platforms — they not only bring metadata together into a single store like a [metadata lake](https://www.xenonstack.com/blog/metadata-lake-features-and-trends), but also leverage “reverse metadata” to make metadata available in daily workflows.
> 

> Third-gen catalogs will leverage metadata to improve existing tools like Looker, dbt, and Slack, finally making the dream of an intelligent data management system a reality.
> 

Data Teams as Product Teams

> In 2021, **Emilie Schario** from [Amplify Partners](https://amplifypartners.com/), **Taylor Murphy** from [Meltano](https://meltano.com/), and **Eric Weber** from [Stitch Fix](https://www.stitchfix.com/) talked about a way to break data teams out of this trap —rethinking **data teams as product teams**. They first explained this idea with a blogon [Locally Optimistic](https://locallyoptimistic.com/post/run-your-data-team-like-a-product-team/), followed by great talks at conferences like [MDSCON](https://www.moderndatastackconference.com/agenda-copy), [dbt Coalesce](https://www.getdbt.com/coalesce-2020/run-your-data-team-as-a-product-team/), and [Future Data](https://www.youtube.com/watch?v=YWQPUMIXLpg).
> 

Data Observability

> Data downtime has been a part of normal life on a data team for years. But now, with many companies relying on data for literally every aspect of their operations, it’s a huge deal when data stops working.
> 
> 
> Yet everyone was just reacting to issues as they cropped up, rather than proactively preventing them. This is where **[data observability](https://towardsdatascience.com/what-is-data-observability-40b337971e3e)** — the idea of “monitoring, tracking, and triaging of incidents to prevent downtime” — came in.
> 

> The space went from being non-existent to hosting a bunch of companies, with a collective $200m of funding raised in 18 months. This includes **[Acceldata](https://www.acceldata.io/newsroom/acceldata-raises-35-million-series-b)**, **[Anomalo](https://techcrunch.com/2021/10/28/anomalo-launches-with-33m-series-a-to-find-issues-in-data-sets-automatically/)**, **[Bigeye](https://techcrunch.com/2021/09/23/bigeye-providing-data-quality-automation-closes-second-round-this-year-with-45m/)**, **[Databand](https://techcrunch.com/2020/12/01/databand-raises-14-5m-led-by-accel-for-its-data-pipeline-observability-tools/)**, **[Datafold](https://www.businesswire.com/news/home/20211109005656/en/Datafold-Raises-20M-Series-A-To-Help-Data-Teams-Deliver-Reliable-Products-Faster)**, **[Metaplane](https://news.ycombinator.com/item?id=29226864)**, **[Monte](https://barrmoses.medium.com/monte-carlo-raises-25m-series-b-to-help-companies-achieve-more-reliable-data-abf497368ee1)[Carlo](https://barrmoses.medium.com/monte-carlo-raises-series-c-brings-funding-to-101m-to-help-companies-trust-their-data-a11f58036896)**, and **[Soda](https://www.prnewswire.com/news-releases/soda-raises-11-5m-series-a-funding-to-bring-everyone-closer-to-data-301219427.html)**. People even started creating lists of new “data observability companies” to help keep track of the space.
> 

> Ideally, if you have all your metadata in one open platform, you should be able to leverage it for a variety of use cases (like data cataloging, observability, lineage and more).
> 

## Emerging Architectures for Modern Data Infrastructure

https://future.a16z.com/emerging-architectures-modern-data-infrastructure/


****Modern Business Intelligence****

> What’s new:
> 
> - There has been a surge of interest in the **metrics layer**, a system providing a standard set of definitions on top of the data warehouse. This has been hotly debated, including what capabilities it should have, which vendor(s) should own it, and what spec it should follow. So far, we’ve seen several credible pure-play products (like Transform and Supergrain), plus expansion into this category by dbt.
> - **Reverse ETL** vendors have grown meaningfully, particularly Hightouch and Census. The purpose of these products is to update operational systems, like CRM or ERP, with outputs and insights derived from the data warehouse.
> - Data teams are showing stronger interest in **new applications** to augment their standard dashboards, especially data workspaces (like Hex). Broadly speaking, new apps are likely the result of increasing standardization in cloud data warehouses — once data is cleanly structured and easy to access, data teams naturally want to do more with it.
> - **Data discovery and observability** companies have proliferated and raised substantial amounts of capital (especially Monte Carlo and Bigeye). While the benefits of these products are clear — i.e. more reliable data pipelines and better collaboration — adoption is still relatively early, as customers discover relevant use cases and budgets. (Technical note: although there are several credible new vendors in data discovery — e.g. Select Star, Metaphor, Stemma, Secoda, Castor — we have excluded seed-stage companies from the diagram in general.)

****Multimodal Data Processing****

> What’s new:
> 
> - There is growing recognition and clarity for the **lakehouse** architecture. We’ve seen this approach supported by a wide range of vendors (including AWS, Databricks, Google Cloud, Starburst, and Dremio) and data warehouse pioneers. The fundamental value of the lakehouse is to pair a robust storage layer with an array of powerful data processing engines like Spark, Presto, Druid/Clickhouse, Python libraries, etc.
> - The **storage layer** itself is getting an upgrade. While technologies like Delta, Iceberg, and Hudi are not new, they are seeing accelerated adoption and are being built into commercial products. Some of these technologies (particularly Iceberg) also interoperate with cloud data warehouses like Snowflake. If heterogeneity is here to stay, this is likely to become a key part of the multimodal data stack.
> - There may be an uptick in adoption taking place for **stream processing** (i.e., real-time analytical data processing). While first-generation technologies like Flink still haven’t gone mainstream, new entrants with simpler programming models (like Materialize and Upsolver) are gaining early adoption, and, anecdotally, usage of stream processing products from incumbents Databricks and Confluent has also started to accelerate.

****Artificial Intelligence and Machine Learning****

> What’s new:
> 
> - The ML industry is consolidating around a **data-centric approach**, emphasizing sophisticated data management over incremental modeling improvements. This has several implications:
>     - Rapid growth for **data labeling** (e.g. Scale and Labelbox) and growing interest in **closed-loop data engines,** largely modeled on Tesla’s Autopilot data pipelines.
>     - Increased adoption for **feature stores** (e.g. Tecton), for both batch and real-time use cases, as a means to develop production-grade ML data in a collaborative way.
>     - Revived interest in **low-code ML** solutions (like Continual and MindsDB) that at least partially automate the ML modeling process. These newer solutions focus on bringing new users (i.e. analysts and software developers) into the ML market.
> - Use of **pre-trained models** is becoming the default, especially in NLP, and providing tailwinds to companies like OpenAI and Hugging Face. There are still meaningful problems to solve here around fine-tuning, cost, and scaling.
> - Operations tools for ML (sometimes called MLops) are becoming more mature, built around **ML monitoring** as the most in-demand use case and immediate budget. Meanwhile, a raft of new operational tools — including, notably, **validation** and **auditing** — are appearing, with the ultimate market still to be determined.
> - There is increased focus on how developers can seamlessly integrate ML models into applications, including through **pre-built APIs** (e.g. OpenAI), **vector databases** (e.g. Pinecone), and more opinionated frameworks.

### More

https://future.a16z.com/data50/