AWS PrivateLink for ClickPipes

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md)

You can use [AWS PrivateLink](https://aws.amazon.com/privatelink/) to establish secure connectivity between VPCs,
AWS services, your on-premises systems, and ClickHouse Cloud without exposing traffic to the public Internet.

This document outlines the ClickPipes reverse private endpoint functionality
that allows setting up an AWS PrivateLink VPC endpoint.

## Supported ClickPipes data sources[​](#supported-sources "Direct link to Supported ClickPipes data sources")

ClickPipes reverse private endpoint functionality is limited to the following
data source types:

- Kafka
- Postgres
- MySQL
- MongoDB

## Supported AWS PrivateLink endpoint types[​](#aws-privatelink-endpoint-types "Direct link to Supported AWS PrivateLink endpoint types")

ClickPipes reverse private endpoint can be configured with one of the following AWS PrivateLink approaches:

- [VPC resource](#vpc-resource)
- [MSK multi-VPC connectivity for MSK ClickPipe](#msk-multi-vpc)
- [VPC endpoint service](#vpc-endpoint-service)

### VPC resource[​](#vpc-resource "Direct link to VPC resource")

References

Cross-region isn't supported.

Your VPC resources can be accessed in ClickPipes using [PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). This approach doesn't require setting up a load balancer in front of your data source.

Resource configuration can be targeted with a specific host or RDS cluster ARN.

It's the preferred choice for Postgres CDC ingesting data from an RDS cluster.

To set up PrivateLink with VPC resource:

1. Create a resource gateway
2. Create a resource configuration
3. Create a resource share

Create a resource gateway

#### Create a resource gateway[​](#create-resource-gateway "Direct link to Create a resource gateway")

Resource gateway is the point that receives traffic for specified resources in your VPC.

Note

Your resource gateway attached subnets are recommended to have sufficient IP addresses available.
It's recommended to have at least `/26` subnet mask for each subnet.

For each VPC endpoint (each Reverse Private Endpoint), AWS requires a consecutive block of 16 IP addresses per subnet. (`/28` subnet mask)
If this requirement isn't met, Reverse Private Endpoint will transition to a failed state.

You can create a resource gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command:

```
aws vpc-lattice create-resource-gateway \
    --vpc-identifier <VPC_ID> \
    --subnet-ids <SUBNET_IDS> \
    --security-group-ids <SG_IDs> \
    --name <RESOURCE_GATEWAY_NAME>
```

The output will contain a resource gateway id, which you will need for the next step.

Before you can proceed, you'll need to wait for the resource gateway to enter into an `Active` state. You can check the state by running the following command:

```
aws vpc-lattice get-resource-gateway \
    --resource-gateway-identifier <RESOURCE_GATEWAY_ID>
```

Create a VPC Resource-Configuration

#### Create a VPC Resource-Configuration[​](#create-resource-configuration "Direct link to Create a VPC Resource-Configuration")

Resource-Configuration is associated with resource gateway to make your resource accessible.

You can create a Resource-Configuration from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-configuration.html) or with the following command:

```
aws vpc-lattice create-resource-configuration \
    --resource-gateway-identifier <RESOURCE_GATEWAY_ID> \
    --type <RESOURCE_CONFIGURATION_TYPE> \
    --resource-configuration-definition <RESOURCE_CONFIGURATION_DEFINITION> \
    --name <RESOURCE_CONFIGURATION_NAME>
```

The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is a single Resource-Configuration. You can configure with the ARN directly, or share an IP address or a domain name that is publicly resolvable.

For example, to configure with the ARN of an RDS Cluster:

```
aws vpc-lattice create-resource-configuration \
    --name my-rds-cluster-config \
    --type ARN \
    --resource-gateway-identifier rgw-0bba03f3d56060135 \
    --resource-configuration-definition 'arnResource={arn=arn:aws:rds:us-east-1:123456789012:cluster:my-rds-cluster}'
```

Note

You can't create a resource configuration for a publicly accessible cluster.
If your cluster is publicly accessible, you must modify the cluster
to make it private before creating the resource configuration
or use [IP allow list](/docs/integrations/clickpipes#list-of-static-ips) instead.
For more information, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html#resource-definition).

The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource.

Create a Resource-Share

#### Create a Resource-Share[​](#create-resource-share "Direct link to Create a Resource-Share")

Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM).

Note

A Resource-Share can only be used for a single Reverse Private Endpoint and cannot be reused.
If you need to use the same Resource-Configuration for multiple Reverse Private Endpoints,
you must create a separate Resource-Share for each endpoint.
The Resource-Share remains in your AWS account after a Reverse Private Endpoint is deleted
and must be manually removed if no longer needed.

You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116` (arn:aws:iam::072088201116:root):

```
aws ram create-resource-share \
    --principals 072088201116 \
    --resource-arns <RESOURCE_CONFIGURATION_ARN> \
    --name <RESOURCE_SHARE_NAME>
```

The output will contain a Resource-Share ARN, which you will need to set up a ClickPipe connection with VPC resource.

You're ready to [create a ClickPipe with Reverse private endpoint](#creating-clickpipe) using VPC resource. You will need to:

- Set `VPC endpoint type` to `VPC Resource`.
- Set `Resource configuration ID` to the ID of the Resource-Configuration created in step 2.
- Set `Resource share ARN` to the ARN of the Resource-Share created in step 3.

For more details on PrivateLink with VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html).

### MSK multi-VPC connectivity[​](#msk-multi-vpc "Direct link to MSK multi-VPC connectivity")

The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster.
Private DNS support is out of the box and doesn't require any additional configuration.
Cross-region isn't supported.

It is a recommended option for ClickPipes for MSK.
See the [getting started](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-getting-started.html) guide for more details.

References

Update your MSK cluster policy and add `072088201116` to the allowed principals to your MSK cluster.
See AWS guide for [attaching a cluster policy](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-policy.html) for more details.

Follow our [MSK setup guide for ClickPipes](/docs/knowledgebase/aws-privatelink-setup-for-msk-clickpipes) to learn how to set up the connection.

### VPC endpoint service[​](#vpc-endpoint-service "Direct link to VPC endpoint service")