Integrating Amazon S3 with ClickHouse Cloud

```
  # Replace <your-region> with your ClickHouse Cloud region
  curl -s https://api.clickhouse.cloud/static-ips.json | jq -r '.aws[] | select(.region == "<your-region>") | .s3_endpoints[]'
  ```

## Advanced settings[​](#advanced-settings "Direct link to Advanced settings")

ClickPipes provides sensible defaults that cover the requirements of most use cases. If your use case requires additional fine-tuning, you can adjust the following settings:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Setting Default value Description|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `Max insert bytes` 10GB Number of bytes to process in a single insert batch.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `Max file count` 100 Maximum number of files to process in a single insert batch.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `Max threads` auto(3) [Maximum number of concurrent threads](/docs/operations/settings/settings#max_threads) for file processing.| `Max insert threads` 1 [Maximum number of concurrent insert threads](/docs/operations/settings/settings#max_insert_threads) for file processing.| `Min insert block size bytes` 1GB [Minimum size of bytes in the block](/docs/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table.| `Max download threads` 4 [Maximum number of concurrent download threads](/docs/operations/settings/settings#max_download_threads).| `Object storage polling interval` 30s Configures the maximum wait period before inserting data into the ClickHouse cluster.|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `Parallel distributed insert select` 2 [Parallel distributed insert select setting](/docs/operations/settings/settings#parallel_distributed_insert_select).| `Parallel view processing` false Whether to enable pushing to attached views [concurrently instead of sequentially](/docs/operations/settings/settings#parallel_view_processing).| `Use cluster function` true Whether to process files in parallel across multiple nodes. | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

### Scaling[​](#scaling "Direct link to Scaling")

Object Storage ClickPipes are scaled based on the minimum ClickHouse service size determined by the [configured vertical autoscaling settings](/docs/cloud/features/autoscaling/vertical#configuring-vertical-auto-scaling). The size of the ClickPipe is determined when the pipe is created. Subsequent changes to the ClickHouse service settings won't affect the ClickPipe size.

To increase the throughput on large ingest jobs, we recommend scaling the ClickHouse service before creating the ClickPipe.

## Known limitations[​](#known-limitations "Direct link to Known limitations")

### File size[​](#file-size "Direct link to File size")

ClickPipes will only attempt to ingest objects that are **10GB or smaller** in size. If a file is greater than 10GB, an error will be appended to the ClickPipes dedicated error table.

### Compatibility[​](#compatibility "Direct link to Compatibility")

Despite being S3-compatible, some services use a different URL structure that the S3 ClickPipe might not be able to parse (e.g., Backblaze B2), or require integration with provider-specific queue services for continuous, unordered ingestion. If you're running into issues with a service that isn't listed under [Supported data sources](#supported-data-sources), please [reach out to our team](https://clickhouse.com/company/contact?loc=clickpipes).

### View support[​](#view-support "Direct link to View support")

Materialized views on the target table are also supported. ClickPipes will create staging tables not only for the target table, but also any dependent materialized view.

We don't create staging tables for non-materialized views. This means that if you have a target table with one of more downstream materialized views, those materialized views should avoid selecting data via a view from the target table. Otherwise, you may find that you're missing data in the materialized view.

- [Supported data sources](#supported-data-sources)- [Supported formats](#supported-formats)- [Features](#features)
      - [One-time ingestion](#one-time-ingestion)- [Continuous ingestion](#continuous-ingestion)- [File pattern matching](#file-pattern-matching)- [Exactly-once semantics](#exactly-once-semantics)- [Virtual columns](#virtual-columns)- [Access control](#access-control)
        - [Permissions](#permissions)- [Authentication](#authentication)- [Network access](#network-access)- [Advanced settings](#advanced-settings)
          - [Scaling](#scaling)- [Known limitations](#known-limitations)
            - [File size](#file-size)- [Compatibility](#compatibility)- [View support](#view-support)

Was this page helpful?