title_wip

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/engines/table-engines/mergetree-family/annindexes.md)

The problem of finding the N closest points in a multi-dimensional (vector) space for a given point is known as [nearest neighbor search](https://en.wikipedia.org/wiki/Nearest_neighbor_search) or, in short: vector search.
Two general approaches exist for solving vector search:

- Exact vector search calculates the distance between the given point and all points in the vector space. This ensures the best possible accuracy, i.e. the returned points are guaranteed to be the actual nearest neighbors. Since the vector space is explored exhaustively, exact vector search can be too slow for real-world use.
- Approximate vector search refers to a group of techniques (e.g., special data structures like graphs and random forests) which compute results much faster than exact vector search. The result accuracy is typically "good enough" for practical use. Many approximate techniques provide parameters to tune the trade-off between the result accuracy and the search time.

A vector search (exact or approximate) can be written in SQL as follows:

```
WITH [...] AS reference_vector
SELECT [...]
FROM table
WHERE [...] -- a WHERE clause is optional
ORDER BY <DistanceFunction>(vectors, reference_vector)
LIMIT <N>
```

The points in the vector space are stored in a column `vectors` of array type, e.g. [Array(Float64)](/docs/sql-reference/data-types/array), [Array(Float32)](/docs/sql-reference/data-types/array), or [Array(BFloat16)](/docs/sql-reference/data-types/array).
The reference vector is a constant array and given as a common table expression.
`<DistanceFunction>` computes the distance between the reference point and all stored points.
Any of the available [distance function](/docs/sql-reference/functions/distance-functions) can be used for that.
`<N>` specifies how many neighbors should be returned.

## Exact vector search[​](#exact-nearest-neighbor-search "Direct link to Exact vector search")

An exact vector search can be performed using above SELECT query as is.
The runtime of such queries is generally proportional to the number of stored vectors and their dimension, i.e. the number of array elements.
Also, since ClickHouse performs a brute-force scan of all vectors, the runtime depends also on the number of threads by the query (see setting [max\_threads](/docs/operations/settings/settings#max_threads)).

### Example[​](#exact-nearest-neighbor-search-example "Direct link to Example")

```
CREATE TABLE tab(id Int32, vec Array(Float32)) ENGINE = MergeTree ORDER BY id;

INSERT INTO tab VALUES (0, [1.0, 0.0]), (1, [1.1, 0.0]), (2, [1.2, 0.0]), (3, [1.3, 0.0]), (4, [1.4, 0.0]), (5, [1.5, 0.0]), (6, [0.0, 2.0]), (7, [0.0, 2.1]), (8, [0.0, 2.2]), (9, [0.0, 2.3]), (10, [0.0, 2.4]), (11, [0.0, 2.5]);

WITH [0., 2.] AS reference_vec
SELECT id, vec
FROM tab
ORDER BY L2Distance(vec, reference_vec) ASC
LIMIT 3;
```

returns

```
   ┌─id─┬─vec─────┐
1. │  6 │ [0,2]   │
2. │  7 │ [0,2.1] │
3. │  8 │ [0,2.2] │
   └────┴─────────┘
```

## Approximate vector search[​](#approximate-nearest-neighbor-search "Direct link to Approximate vector search")

### Vector Similarity Indexes[​](#vector-similarity-index "Direct link to Vector Similarity Indexes")

ClickHouse provides a special "vector similarity" index to perform approximate vector search.

Note

Vector similarity indexes are available in ClickHouse version 25.8 and higher.
If you run into problems, kindly open an issue in the [ClickHouse repository](https://github.com/clickhouse/clickhouse/issues).

#### Creating a Vector Similarity Index[​](#creating-a-vector-similarity-index "Direct link to Creating a Vector Similarity Index")

A vector similarity index can be created on a new table like this:

```
CREATE TABLE table
(
  [...],
  vectors Array(Float*),
  INDEX <index_name> vectors TYPE vector_similarity(<type>, <distance_function>, <dimensions>) [GRANULARITY <N>]
)
ENGINE = MergeTree
ORDER BY [...]
```

Alternatively, to add a vector similarity index to an existing table:

```
ALTER TABLE table ADD INDEX <index_name> vectors TYPE vector_similarity(<type>, <distance_function>, <dimensions>) [GRANULARITY <N>];
```

Vector similarity indexes are special kinds of skipping indexes (see [here](/docs/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-data_skipping-indexes) and [here](/docs/optimize/skipping-indexes)).
Accordingly, above `ALTER TABLE` statement only causes the index to be build for future new data inserted into the table.
To build the index for existing data as well, you need to materialize it:

```
ALTER TABLE table MATERIALIZE INDEX <index_name> SETTINGS mutations_sync = 2;
```

Function `<distance_function>` must be

- `L2Distance`, the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance), representing the length of a line between two points in Euclidean space, or
- `cosineDistance`, the [cosine distance](https://en.wikipedia.org/wiki/Cosine_similarity#Cosine_distance), representing the angle between two non-zero vectors.

For normalized data, `L2Distance` is usually the best choice, otherwise `cosineDistance` is recommended to compensate for scale.

`<dimensions>` specifies the array cardinality (number of elements) in the underlying column.
If ClickHouse finds an array with a different cardinality during index creation, the index is discarded and an error is returned.

The optional GRANULARITY parameter `<N>` refers to the size of the index granules (see [here](/docs/optimize/skipping-indexes)).
Unlike regular skip indexes, which use a default index granularity of 1, vector similarity indexes use 100 million as default index granularity.
This value makes sure that only few indexes are build internally even for large parts.
We recommend changing the index granularity only for advanced users who understand the implications of what they are doing (see [below](#differences-to-regular-skipping-indexes)).

Vector similarity indexes are generic in the sense that they can accommodate different approximate search method.
The actually used method is specified by parameter `<type>`.
As of now, the only available method is HNSW ([academic paper](https://arxiv.org/abs/1603.09320)), a popular and state-of-the-art technique for approximate vector search based on hierarchical proximity graphs.
If HNSW is used as type, users may optionally specify further HNSW-specific parameters:

```
CREATE TABLE table
(
  [...],
  vectors Array(Float*),
  INDEX index_name vectors TYPE vector_similarity('hnsw', <distance_function>, <dimensions>[, <quantization>, <hnsw_max_connections_per_layer>, <hnsw_candidate_list_size_for_construction>]) [GRANULARITY N]
)
ENGINE = MergeTree
ORDER BY [...]
```

These HNSW-specific parameters are available:

- `<quantization>` controls the quantization of the vectors in the proximity graph. Possible values are `f64`, `f32`, `f16`, `bf16`, `i8`, or `b1`. The default value is `bf16`. Note that this parameter does not affect the representation of the vectors in the underlying column.
- `<hnsw_max_connections_per_layer>` controls the number of neighbors per graph node, also known as HNSW hyperparameter `M`. The default value is `32`. Value `0` means using the default value.
- `<hnsw_candidate_list_size_for_construction>` controls the size of the dynamic candidate list during construction of the HNSW graph, also known as HNSW hyperparameter `ef_construction`. The default value is `128`. Value `0` means using the default value.

The default values of all HNSW-specific parameters work reasonably well in the majority of use cases.
We therefore do not recommend customizing the HNSW-specific parameters.

Further restrictions apply: