Machine learning

### Training and evaluation[​](#training-and-evaluation "Direct link to Training and evaluation")

At this point, features will have been split into training, validation, and test sets.
These data sets are versioned, and then utilized by their respective stages.

It is common in this phase of the pipeline to introduce yet another specialized tool to the machine learning data layer - the feature store.
A feature store is most commonly a layer of abstraction around a database that provides convenience features specific to managing data for model training, inference, and evaluation.
Examples of these convenience features include versioning, access management, and automatically translating the definition of features to SQL statements.

For feature stores, ClickHouse can act as a:

**Data source** - With the ability to query or ingest data in over 70 different file formats, including data lake formats such as Iceberg and Delta Lake, ClickHouse makes an ideal long-term store holding or querying data.
By separating storage and compute using object storage, ClickHouse Cloud additionally allows data to be held indefinitely - with compute scaled down or made completely idle to minimize costs.
Flexible codecs, coupled with column-oriented storage and ordering of data on disk, maximize compression rates, thus minimizing the required storage.
You can easily combine ClickHouse with data lakes, with built-in functions to query data in place on object storage.

**Transformation engine** - SQL provides a natural means of declaring data transformations.
When extended with ClickHouse’s analytical and statistical functions, these transformations become succinct and optimized.
As well as applying to either ClickHouse tables, in cases where ClickHouse is used as a data store, table functions allow SQL queries to be written against data stored in formats such as Parquet, on-disk or object storage, or even other data stores such as Postgres and MySQL.
A completely parallelization query execution engine, combined with a column-oriented storage format, allows ClickHouse to perform aggregations over PBs of data in seconds - unlike transformations on in memory data frames, users aren't memory-bound.
Furthermore, materialized views allow data to be transformed at insert time, thus overloading compute to data load time from query time.
These views can exploit the same range of analytical and statistical functions ideal for data analysis and summarization.
Should any of ClickHouse’s existing analytical functions be insufficient or custom libraries need to be integrated, you can also utilize User Defined Functions (UDFs).

#### Offline feature store[​](#offline-feature-store "Direct link to Offline feature store")

An offline feature store is used for model training.
This generally means that the features themselves are produced through batch-process data transformation pipelines (as described in the above section), and there are typically no strict latency requirements on the availability of those features.

With capabilities to read data from multiple sources and apply transformations via SQL queries, the results of these queries can also be persisted in ClickHouse via `INSERT INTO SELECT` statements.
With transformations often grouped by an entity ID and returning a number of columns as results, ClickHouse’s schema inference can automatically detect the required types from these results and produce an appropriate table schema to store them.
Functions for generating random numbers and statistical sampling allow data to be efficiently iterated and scaled at millions or rows per second for feeding to model training pipelines.

Often, features are represented in tables with a timestamp indicating the value for an entity and feature at a specific point in time.
As described earlier, training pipelines often need the state of features at specific points in time and in groups. ClickHouse’s sparse indices allow fast filtering of data to satisfy point-in-time queries and feature selection filters. While other technologies such as Spark, Redshift, and BigQuery rely on slow stateful windowed approaches to identify the state of features at a specific point in time, ClickHouse supports the ASOF (as-of-this-time) LEFT JOIN query and argMax function.
In addition to simplifying syntax, this approach is highly performant on large datasets through the use of a sort and merge algorithm.
This allows feature groups to be built quickly, reducing data preparation time prior to training.

#### Online feature store[​](#online-feature-store "Direct link to Online feature store")

Online feature stores are used to store the latest version of features used for inference and are applied in real-time.
This means that these features need to be calculated with minimal latency, as they’re used as part of a real-time machine learning service.

As a real-time analytics database, ClickHouse can serve highly concurrent query workloads at low latency.
While this requires data to be typically denormalized, this aligns with the storage of feature groups used at both training and inference time.
Importantly, ClickHouse is able to deliver this query performance while being subject to high write workloads thanks to its log-structured merge tree.
These properties are required in an online store to keep features up-to-date.
Since the features are already available within the offline store, they can easily be materialized to new tables within either the same ClickHouse cluster or a different instance via existing capabilities, e.g. `remoteSecure`.
Integrations with Kafka, through either an exactly-once Kafka Connect offering or via [ClickPipes](/docs/integrations/clickpipes/kafka) in ClickHouse Cloud, also make consuming streaming data from streaming sources simple and reliable.

Many modern systems require both offline and online stores, and it is easy to jump to the conclusion that two specialized feature stores are required here.
However, this introduces the additional complexity of keeping both of these stores in sync, which of course also includes the cost of replicating data between them.

A real-time data warehouse like ClickHouse is a single system that can power both offline and online feature management.
ClickHouse efficiently processes streaming and historical data, and has the unlimited scale, performance, and concurrency needed to be relied upon when serving features for real-time inference and offline training.

In considering the tradeoffs between using a feature store product in this stage versus leveraging a real-time data warehouse directly, it’s worth emphasizing that convenience features such as versioning can be achieved through age-old database paradigms such as table or schema design.
Other functionality, such as converting feature definitions to SQL statements, may provide greater flexibility as part of the application or business logic, rather than existing in an opinionated layer of abstraction.

### Inference[​](#inference "Direct link to Inference")

Model inference is the process of running a trained model to receive an output.
When inference is triggered by database actions - for instance, inserting a new record, or querying records - the inference step could be managed via bespoke jobs or application code.

On the other hand, it could be managed in the data layer itself. ClickHouse [User Defined Functions (UDFs)](/docs/sql-reference/functions/udf), give users the ability to invoke a model directly from ClickHouse at insert or query time.
This provides the ability to pass incoming data to a model, receive the output, and store these results along with the ingested data automatically - all without having to spin up other processes or jobs.
This also provides a single interface, SQL, by which to manage this step.

### Vector store[​](#vector-store "Direct link to Vector store")