ClickHouse/clickhouse-docs-embeddings huggingface dataset:
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/guides/infrastructure/01_deployment_options/byoc/08_reference/02_priviledge.md)

## CloudFormation IAM roles[​](#cloudformation-iam-roles "Direct link to CloudFormation IAM roles")

### Bootstrap IAM role[​](#bootstrap-iam-role "Direct link to Bootstrap IAM role")

The bootstrap IAM role has the following permissions:

- **EC2 and VPC operations**: Required for setting up VPC and EKS clusters.
- **S3 operations (e.g., `s3:CreateBucket`)**: Needed to create buckets for ClickHouse BYOC storage.
- **IAM operations (e.g., `iam:CreatePolicy`)**: Needed for controllers to create additional roles (see the next section for details).
- **EKS operations**: Limited to resources with names starting with the `clickhouse-cloud` prefix.

### Additional IAM roles created by the controller[​](#additional-iam-roles-created-by-the-controller "Direct link to Additional IAM roles created by the controller")

In addition to the `ClickHouseManagementRole` created via CloudFormation, the controller will create several additional roles.

These roles are assumed by applications running within the customer's EKS cluster:

- **State Exporter Role**
  - ClickHouse component that reports service health information to ClickHouse Cloud.
  - Requires permission to write to an SQS queue owned by ClickHouse Cloud.
- **Load-Balancer Controller**
  - Standard AWS load balancer controller.
  - EBS CSI Controller to manage volumes for ClickHouse services.
- **External-DNS**
  - Propagates DNS configurations to Route 53.
- **Cert-Manager**
  - Provisions TLS certificates for BYOC service domains.
- **Cluster Autoscaler**
  - Adjusts the node group size as needed.

**K8s-control-plane** and **k8s-worker** roles are meant to be assumed by AWS EKS services.

Lastly, **`data-plane-mgmt`** allows a ClickHouse Cloud Control Plane component to reconcile necessary custom resources, such as `ClickHouseCluster` and the Istio Virtual Service/Gateway.

- [CloudFormation IAM roles](#cloudformation-iam-roles)
  - [Bootstrap IAM role](#bootstrap-iam-role)- [Additional IAM roles created by the controller](#additional-iam-roles-created-by-the-controller)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/reference/01_changelog/index.md)

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  | | --- | --- | --- | --- | | [Cloud Changelog](/docs/whats-new/cloud) Changelog for ClickHouse Cloud|  |  | | --- | --- | | [Release Notes](/docs/cloud/reference/changelogs/release-notes) Release notes for all ClickHouse Cloud releases | | | | | |

Was this page helpful?
Console roles and permissions

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | control-plane:service:view-private-endpoints View private endpoint configuration for a service.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | control-plane:service:manage-private-endpoints Create and manage private endpoints and private networking.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **ClickPipes** ([more info](/docs/integrations/clickpipes)) ClickPipes integration|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | control-plane:service:manage-clickpipes Manage ClickPipes integration and related settings.|  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Scaling** ([more info](/docs/manage/scaling)) Scaling and autoscaling configuration|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | control-plane:service:view-scaling-config View scaling configuration and autoscaling settings for a service.|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | control-plane:service:manage-scaling-config Modify scaling configuration and trigger scaling operations.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | **ClickStack** ([more info](/docs/use-cases/observability/clickstack/overview)) ClickStack observability integrations|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | control-plane:service:manage-clickstack-api Manage ClickStack API access and related integrations.|  |  |  |  | | --- | --- | --- | --- | | **SQL console role mapping** ([more info](/docs/cloud/guides/sql-console/manage-sql-console-role-assignments)) Manage SQL console role assignments|  |  | | --- | --- | | sql-console:database:access Passwordless access to the database via SQL console (may only be used with sql-console-admin or sql-console-readonly) | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/reference/09_security/02_audit-logging.md)

## Console audit log events[​](#console-audit-log-events "Direct link to Console audit log events")

The different types of events captured for the organization are grouped in 3 categories: **Organization**, **Service** and **User**. For more information about the audit log and how to export or add an API integration, review the [console audit log](/docs/cloud/security/audit-logging/console-audit-log) documentation in the Guides section above.

The following events are recorded to the audit log.

### Organization[​](#organization "Direct link to Organization")

- Organization created
- Organization deleted
- Organization name changed

### Service[​](#service "Direct link to Service")

- Service created
- Service deleted
- Service stopped
- Service started
- Service name changed
- Service IP access list changed
- Service password reset

### User[​](#user "Direct link to User")

- User role changed
- User removed from organization
- User invited to organization
- User joined organization
- User invitation deleted
- User left organization

- [Console audit log events](#console-audit-log-events)
  - [Organization](#organization)- [Service](#service)- [User](#user)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/guides/security/05_audit_logging/01_console-audit-log.md)

Console activities are recorded in the audit log. If you have the Admin or Developer organization role, you can review the log and integrate it with your logging systems.

## Access the console log via the user interface[​](#console-audit-log-ui "Direct link to Access the console log via the user interface")

Select organization

## Select organization[​](#select-org "Direct link to Select organization")

In ClickHouse Cloud, navigate to your organization details.

Select audit

## Select audit[​](#select-audit "Direct link to Select audit")

Select the **Audit** tab on the left menu to see what changes have been made to your ClickHouse Cloud organization - including who made the change and when it occurred.

The **Activity** page displays a table containing a list of events logged about your organization. By default, this list is sorted in a reverse-chronological order (most-recent event at the top). Change the order of the table by clicking on the columns headers. Each item of the table contains the following fields:

- **Activity:** A text snippet describing the event
- **User:** The user that initiated the event
- **IP Address:** When applicable, this flied lists the IP Address of the user that initiated the event
- **Time:** The timestamp of the event

Use the search bar

## Use the search bar[​](#use-search-bar "Direct link to Use the search bar")

You can use the search bar provided to isolate events based on some criteria like for example service name or IP address. You can also export this information in a CSV format for distribution or analysis in an external tool.

## Access the console audit log via the API[​](#console-audit-log-api "Direct link to Access the console audit log via the API")

You can use the ClickHouse Cloud API `activity` endpoint to obtain an export
of audit events. Further details can be found in the [API reference](https://clickhouse.com/docs/cloud/manage/api/swagger).

## Log integrations[​](#log-integrations "Direct link to Log integrations")

You can use the API to integrate with a logging platform of their choice. The following have supported out-of-the-box connectors:

- [ClickHouse Cloud Audit add-on for Splunk](/docs/integrations/audit-splunk)

- [Access the console log via the user interface](#console-audit-log-ui)- [Select organization](#select-org)- [Select audit](#select-audit)- [Use the search bar](#use-search-bar)- [Access the console audit log via the API](#console-audit-log-api)- [Log integrations](#log-integrations)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/guides/security/05_audit_logging/02_database-audit-log.md)

ClickHouse provides database audit logs by default. This page focuses on security relevant logs. For more information on data recorded by the system, refer to the docs for [system tables](/docs/operations/system-tables/overview).

Log retention

Information is logged directly to the system tables and are retained for up to 30 days by default. This period can be longer or shorter and is affected by the frequency of merges in the system. Customers may take additional measures to store logs for longer or export logs to a security information and event management (SIEM) system for long term storage. Details below.

## Security relevant logs[​](#security-relevant-logs "Direct link to Security relevant logs")

ClickHouse logs security relevant database events primarily to session and query logs.

The [system.session\_log](/docs/operations/system-tables/session_log) records successful and failed login attempts, as well as the location of the authentication attempt. This information can be used to identify credential stuffing or brute force attacks against a ClickHouse instance.

Sample query showing login failures

```
select event_time
    ,type
    ,user
    ,auth_type
    ,client_address 
FROM clusterAllReplicas('default',system.session_log) 
WHERE type='LoginFailure' 
LIMIT 100
```

The [system.query\_log](/docs/operations/system-tables/query_log) captures query activity executed in a ClickHouse instance. This information can be useful to determine what queries a threat actor executed.

Sample query to search for activities of a "compromised\_account" user

```
SELECT event_time
    ,address
    ,initial_user
    ,initial_address
    ,forwarded_for
    ,query 
FROM clusterAllReplicas('default', system.query_log) 
WHERE user=’compromised_account’
```

## Retaining log data within services[​](#reatining-log-data-within-services "Direct link to Retaining log data within services")

Customers needing longer retention or log durability can use materialized views to achieve these objectives. For more information on materialized views, what they are, benefits and how to implement review our [materialized views](/docs/materialized-views) videos and documentation.

## Exporting logs[​](#exporting-logs "Direct link to Exporting logs")

System logs may be written or exported to a storage location using various formats that are compatible with SIEM systems. For more information, review our [table functions](/docs/sql-reference/table-functions) docs. The most common methods are:

- [Write to S3](/docs/sql-reference/table-functions/s3)
- [Write to GCS](/docs/sql-reference/table-functions/gcs)
- [Write to Azure Blob Storage](/docs/sql-reference/table-functions/azureBlobStorage)

- [Security relevant logs](#security-relevant-logs)- [Retaining log data within services](#reatining-log-data-within-services)- [Exporting logs](#exporting-logs)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/guides/security/05_audit_logging/index.md)

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Console audit log](/docs/cloud/security/audit-logging/console-audit-log) Accessing and reviewing audited events in the ClickHouse Cloud console|  |  |  |  | | --- | --- | --- | --- | | [Database audit logs](/docs/cloud/security/audit-logging/database-audit-log) Relevant logs for database activities|  |  | | --- | --- | | [BYOC security playbook](/docs/cloud/security/audit-logging/byoc-security-playbook) Sample logs and queries customers can use in developing their BYOC security program | | | | | | | |

Was this page helpful?
Azure Private Link

Set the following environment variables before running any commands:

```
KEY_ID=<Key ID>
KEY_SECRET=<Key secret>
ORG_ID=<please set ClickHouse organization ID>
INSTANCE_ID=<Instance ID>
```

Run the following command to check Private Endpoint filters:

```
curl --silent --user "${KEY_ID:?}:${KEY_SECRET:?}" -X GET -H "Content-Type: application/json" "https://api.clickhouse.cloud/v1/organizations/${ORG_ID:?}/services/${INSTANCE_ID:?}" | jq .result.privateEndpointIds
```

## More information[​](#more-information "Direct link to More information")

For more information about Azure Private Link, please visit [azure.microsoft.com/en-us/products/private-link](https://azure.microsoft.com/en-us/products/private-link).

- [Attention](#attention)- [Obtain Azure connection alias for Private Link](#obtain-azure-connection-alias-for-private-link)
    - [Option 1: ClickHouse Cloud console](#option-1-clickhouse-cloud-console)- [Option 2: API](#option-2-api)- [Create a private endpoint in Azure](#create-private-endpoint-in-azure)
      - [Option 1: Using Azure Portal to create a private endpoint in Azure](#option-1-using-azure-portal-to-create-a-private-endpoint-in-azure)- [Option 2: Using Terraform to create a private endpoint in Azure](#option-2-using-terraform-to-create-a-private-endpoint-in-azure)- [Obtaining the Private Endpoint Resource ID](#obtaining-private-endpoint-resourceid)- [Setting up DNS for Private Link](#setting-up-dns-for-private-link)
        - [Create Private DNS zone](#create-private-dns-zone)- [Create a wildcard DNS record](#create-a-wildcard-dns-record)- [Create a virtual network link](#create-a-virtual-network-link)- [Verify DNS setup](#verify-dns-setup)- [Add the Private Endpoint Resource ID to your ClickHouse Cloud organization](#add-the-private-endpoint-id-to-your-clickhouse-cloud-organization)
          - [Option 1: ClickHouse Cloud console](#option-1-clickhouse-cloud-console-1)- [Option 2: API](#option-2-api-1)- [Add the Private Endpoint Resource ID to your services allow list](#add-private-endpoint-id-to-services-allow-list)
            - [Option 1: ClickHouse Cloud console](#option-1-clickhouse-cloud-console-2)- [Option 2: API](#option-2-api-2)- [Access your ClickHouse Cloud service using Private Link](#access-your-clickhouse-cloud-service-using-private-link)
              - [Obtaining the private DNS hostname](#obtaining-the-private-dns-hostname)- [Troubleshooting](#troubleshooting)
                - [Test DNS setup](#test-dns-setup)- [Connection reset by peer](#connection-reset-by-peer)- [Private Endpoint is in pending state](#private-endpoint-is-in-pending-state)- [Test connectivity](#test-connectivity)- [Checking private endpoint filters](#checking-private-endpoint-filters)- [More information](#more-information)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/guides/security/02_connectivity/private_networking/index.md)

ClickHouse Cloud provides the ability to connect your services to your cloud virtual network. Refer to the guides below for set up steps for your provider:

- [AWS private Link](/docs/manage/security/aws-privatelink)
- [GCP private service connect](/docs/manage/security/gcp-private-service-connect)
- [Azure private link](/docs/cloud/security/azure-privatelink)

Was this page helpful?
ClickHouse operations: community debugging insights

### Video sources[​](#video-sources "Direct link to Video sources")

- [10 Lessons from Operating ClickHouse](https://www.youtube.com/watch?v=liTgGiTuhJE)
- [Fast, Concurrent, and Consistent Asynchronous INSERTS in ClickHouse](https://www.youtube.com/watch?v=AsMPEfN5QtM)

- [Essential system tables](#essential-system-tables)
  - [system.errors](#system-errors)- [system.replicas](#system-replicas)- [system.replication\_queue](#system-replication-queue)- [system.merges](#system-merges)- [system.parts](#system-parts)- [Common production issues](#common-production-issues)
    - [Disk space problems](#disk-space-problems)- [Too many parts error](#too-many-parts-error)- [Invalid timestamps issues](#data-quality-issues)- [`ALTER` operation risks](#alter-operation-risks)- [Memory and performance](#memory-and-performance)
      - [External aggregation](#external-aggregation)- [Async insert details](#async-insert-details)- [Distributed table configuration](#distributed-table-configuration)- [Performance monitoring thresholds](#performance-monitoring-thresholds)- [Quick reference](#quick-reference)
        - [Video sources](#video-sources)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/concepts/index.md)

In this section of the docs we'll dive into the concepts around what makes ClickHouse so fast and efficient.

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Why is ClickHouse so Fast?](/docs/concepts/why-clickhouse-is-so-fast) Learn what makes ClickHouse so fast.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [What is OLAP?](/docs/concepts/olap) Learn what Online Analytical Processing is.|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Why is ClickHouse unique?](/docs/about-us/distinctive-features) Learn what makes ClickHouse unique.|  |  |  |  | | --- | --- | --- | --- | | [Glossary](/docs/concepts/glossary) This page contains a glossary of terms you'll commonly encounter throughout the docs.|  |  | | --- | --- | | [FAQ](/docs/concepts/faq) A compilation of the most frequently asked questions we get about ClickHouse. | | | | | | | | | | | |

Was this page helpful?
FAQ

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/faq/index.md)

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [General Questions about ClickHouse](/docs/faq/general) General questions we get about ClickHouse.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [Why not use something like MapReduce?](/docs/faq/general/mapreduce) Explainer on why MapReduce implementations aren't appropriate for the OLAP scenario.|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [What does "не тормозит" mean](/docs/faq/general/ne-tormozit) Explainer on what "не тормозит" means, which you may have seen on ClickHouse t-shirts.|  |  |  |  | | --- | --- | --- | --- | | [What is OLAP](/docs/faq/general/olap) Explainer on what Online Analytical Processing is.|  |  | | --- | --- | | [Who is using ClickHouse](/docs/faq/general/who-is-using-clickhouse) Learn about who is using ClickHouse. | | | | | | | | | | | |

Was this page helpful?
Glossary

## Skipping index[​](#skipping-index "Direct link to Skipping index")

Skipping indices are used to store small amounts of metadata at the level of multiple consecutive granules which allows ClickHouse to avoid scanning irrelevant rows. Skipping indices provide a lightweight alternative to projections.

## Sorting key[​](#sorting-key "Direct link to Sorting key")

In ClickHouse, a sorting key defines the physical order of rows on disk. If you don't specify a primary key, ClickHouse uses the sorting key as the primary key. If you specify both, the primary key must be a prefix of the sorting key.

## Sparse index[​](#sparse-index "Direct link to Sparse index")

A type of indexing when the primary index contains one entry for a group of rows, rather than a single row. The entry that corresponds to a group of rows is referred to as a mark. With sparse indexes, ClickHouse first identifies groups of rows that potentially match the query and then processes them separately to find a match. Because of this, the primary index is small enough to be loaded into the memory.

## Table engine[​](#table-engine "Direct link to Table engine")

Table engines in ClickHouse determine how data is written, stored and accessed. MergeTree is the most common table engine, and allows quick insertion of large amounts of data which get processed in the background.

## TTL[​](#ttl "Direct link to TTL")

Time To Live (TTL) is A ClickHouse feature that automatically moves, deletes, or rolls up columns or rows after a certain time period. This allows you to manage storage more efficiently because you can delete, move, or archive the data that you no longer need to access frequently.

- [Atomicity](#atomicity)- [Block](#block)- [Cluster](#cluster)- [CMEK](#cmek)- [Dictionary](#dictionary)- [Distributed table](#distributed-table)- [Granule](#granule)- [Incremental materialized view](#incremental-materialized-view)- [Lightweight update](#lightweight-update)- [Mark file](#mark-file)- [Materialized view](#materialized-view)- [MergeTree](#mergetree)- [Mutation](#mutation)- [On-the-fly mutation](#on-the-fly-mutation)- [Parts](#parts)- [Partitioning key](#partitioning-key)- [Primary key](#primary-key)- [Projection](#projection)- [Refreshable materialized view](#refreshable-materialized-view)- [Replica](#replica)- [Shard](#shard)- [Skipping index](#skipping-index)- [Sorting key](#sorting-key)- [Sparse index](#sparse-index)- [Table engine](#table-engine)- [TTL](#ttl)

Was this page helpful?