Machine learning

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/onboard/01_discover/02_use_cases/04_machine_learning_and_genAI/01_machine_learning.md)

## The machine learning data layer[​](#machine-learning-data-layer "Direct link to The machine learning data layer")

You’ve probably heard the lore that 80% of a machine learning practitioner's time is spent cleaning data.
Regardless of whether this myth holds true or not, what does remain true is that data is at the heart of the machine learning problem, from start to finish.
Whether you’re building RAG pipelines, fine-tuning, training your own model, or evaluating model performance, data is the root of each problem.

Managing data can be tricky, and as a byproduct, the space has experienced a proliferation of tools that are designed to boost productivity by solving a specific slice of a machine learning data problem.
Oftentimes, this takes shape as a layer of abstraction around a more general-purpose solution with an opinionated interface that, on the surface, makes it easier to apply to the specific sub problem at hand.
In effect, this reduces the flexibility that exists with a general-purpose solution in favor of ease-of-use and simplicity of a specific task.

There are several drawbacks to this approach.
A cascading suite of specialized tools, products, and services, in contrast with a general-purpose solution coupled with supporting application code, presents the risk of greater architectural complexity and data costs than necessary.
It’s easy to accidentally find yourself with an endless list of tools and services, each used for just a single step.

There are two common dimensions to these risks:

1. **Learning, maintenance, and switching costs**

Machine learning architectures can become so cluttered with various tools and components that it creates a fragmented and challenging environment to learn and manage, with increased points of failure and expense creep.

2. **Data duplication and transfer costs**

Using several discrete yet overlapping data systems in a machine learning pipeline may introduce an unnecessary, and often costly, overhead of shipping data around from one to another.

A great illustration of this tradeoff is the vector database.
Vector databases are designed for the hyper-specific machine learning task of storing and searching across vectors.
While this may be the right choice in some architectures, a vector database may be an unnecessary new addition to the tech stack in others, as it is yet another system to integrate with, manage, and ship data to and from.
Most modern general-purpose databases come with vector support out-of-the-box (or through a plugin) and have more extensive and cross-cutting capabilities.
In other words, there may be no need for a net new database to specifically handle vectors in those architectures at all.
The importance boils down to whether the vector-specific convenience features (e.g. inbuilt embedding models) are mission-critical and worth the cost.

### Data exploration[​](#data-exploration "Direct link to Data exploration")

After defining the machine learning problem, goals, and success criteria, a common first step is to explore the relevant data that will be used for model training and evaluation.

During this step, data is analyzed to understand its characteristics, distributions, and relationships.
This process of evaluation and understanding is an iterative one, often resulting in a series of ad-hoc queries being executed across datasets, where query responsiveness is critical (along with other factors such as cost-efficiency and accuracy).
As companies store increasing amounts of data to leverage for machine learning purposes, the problem of examining the data you have becomes harder.

This is because analytics and evaluation queries often become tediously or prohibitively slow at scale with traditional data systems.
Some of the big players impose significantly increased costs to bring down query times, and discourage ad-hoc evaluation by way of charging per query or by number of bytes scanned.
Engineers may resort to pulling subsets of data down to their local machines as a compromise for these limitations.

ClickHouse, on the other hand, is a real-time data warehouse, so users benefit from industry-leading query speeds for analytical computations.
Further, ClickHouse delivers high performance from the start, and doesn’t gate critical query-accelerating features behind higher pricing tiers.
ClickHouse can also query data directly from object storage or data lakes, with support for common formats such as Iceberg, Delta Lake, and Hudi.
This means that no matter where your data lives, ClickHouse can serve as a unifying access and computation layer for your machine learning workloads.

ClickHouse also has an extensive suite of pre-built statistical and aggregation functions that scale over petabytes of data, making it easy to write and maintain simple SQL that executes complex computations.
With support for the most granular precision data types and codecs, you don't need to worry about reducing the granularity of your data.

While you can transform data directly in ClickHouse or prior to insertion using SQL queries, ClickHouse can also be used in programming environments such as Python via [chDB](/docs/chdb).
This allows embedded ClickHouse to be exposed as a Python module and used to transform and manipulate large data frames within notebooks.
Data engineers can therefore perform transformation work to be performed client-side, with results potentially materialized as feature tables in a centralized ClickHouse instance.

### Data preparation and feature extraction[​](#data-preparation-and-feature-extraction "Direct link to Data preparation and feature extraction")

Data is then prepared: cleaned, transformed, and used to extract the features by which the model will be trained and evaluated.
This component is sometimes called a feature generation or extraction pipeline, and is another slice of the machine learning data layer where new tools are often introduced.
MLOps players like Neptune and Hopsworks provide examples of the host of different data transformation products that are used to orchestrate pipelines like these.
However, because they’re separate tools from the database they’re operating on, they can be brittle, and can cause disruptions that need to be manually rectified.

In contrast, data transformations are easily accomplished directly in ClickHouse through [materialized views](/docs/materialized-views).
These are automatically triggered when new data is inserted into ClickHouse source tables and are used to easily extract, transform, and modify data as it arrives - eliminating the need to build and monitor bespoke pipelines yourself.
When these transformations require aggregations over a complete dataset that may not fit into memory, leveraging ClickHouse ensures you don’t have to try and retrofit this step to work with data frames on your local machine.
For those datasets that are more convenient to evaluate locally, [ClickHouse local](/docs/operations/utilities/clickhouse-local) is a great alternative, along with [chDB](/docs/chdb), that allow you to leverage ClickHouse with standard Python data libraries like Pandas.

### Training and evaluation[​](#training-and-evaluation "Direct link to Training and evaluation")

At this point, features will have been split into training, validation, and test sets.
These data sets are versioned, and then utilized by their respective stages.

It is common in this phase of the pipeline to introduce yet another specialized tool to the machine learning data layer - the feature store.
A feature store is most commonly a layer of abstraction around a database that provides convenience features specific to managing data for model training, inference, and evaluation.
Examples of these convenience features include versioning, access management, and automatically translating the definition of features to SQL statements.

For feature stores, ClickHouse can act as a: