Chapter 2. Configuring private connections 2.1. Configuring private connections Private cluster access can be implemented to suit the needs of your Red Hat OpenShift Service on AWS (ROSA) environment. Procedure Access your ROSA AWS account and use one or more of the following methods to establish a private connection to your cluster: Configuring AWS VPC peering : Enable VPC peering to route network traffic between two private IP addresses. Configuring AWS VPN : Establish a Virtual Private Network to securely connect your private network to your Amazon Virtual Private Cloud. Configuring AWS Direct Connect : Configure AWS Direct Connect to establish a dedicated network connection between your private network and an AWS Direct Connect location. Configure a private cluster on ROSA . 2.2. Configuring AWS VPC peering This sample process configures an Amazon Web Services (AWS) VPC containing an Red Hat OpenShift Service on AWS cluster to peer with another AWS VPC network. For more information about creating an AWS VPC Peering connection or for other possible configurations, see the AWS VPC Peering guide. 2.2.1. VPC peering terms When setting up a VPC peering connection between two VPCs on two separate AWS accounts, the following terms are used: Red Hat OpenShift Service on AWS AWS Account The AWS account that contains the Red Hat OpenShift Service on AWS cluster. Red Hat OpenShift Service on AWS Cluster VPC The VPC that contains the Red Hat OpenShift Service on AWS cluster. Customer AWS Account Your non-Red Hat OpenShift Service on AWS AWS Account that you would like to peer with. Customer VPC The VPC in your AWS Account that you would like to peer with. Customer VPC Region The region where the customer's VPC resides. Note As of July 2018, AWS supports inter-region VPC peering between all commercial regions excluding China . 2.2.2. Initiating the VPC peer request You can send a VPC peering connection request from the Red Hat OpenShift Service on AWS AWS Account to the Customer AWS Account. Prerequisites Gather the following information about the Customer VPC required to initiate the peering request: Customer AWS account number Customer VPC ID Customer VPC Region Customer VPC CIDR Check the CIDR block used by the Red Hat OpenShift Service on AWS Cluster VPC. If it overlaps or matches the CIDR block for the Customer VPC, then peering between these two VPCs is not possible; see the Amazon VPC Unsupported VPC Peering Configurations documentation for details. If the CIDR blocks do not overlap, you can continue with the procedure. Procedure Log in to the Web Console for the Red Hat OpenShift Service on AWS AWS Account and navigate to the VPC Dashboard in the region where the cluster is being hosted. Go to the Peering Connections page and click the Create Peering Connection button. Verify the details of the account you are logged in to and the details of the account and VPC you are connecting to: Peering connection name tag : Set a descriptive name for the VPC Peering Connection. VPC (Requester) : Select the Red Hat OpenShift Service on AWS Cluster VPC ID from the dropdown *list. Account : Select Another account and provide the Customer AWS Account number *(without dashes). Region : If the Customer VPC Region differs from the current region, select Another Region and select the customer VPC Region from the dropdown list. VPC (Accepter) : Set the Customer VPC ID. Click Create Peering Connection . Confirm that the request enters a Pending state. If it enters a Failed state, confirm the details and repeat the process. 2.2.3. Accepting the VPC peer request After you create the VPC peering connection, you must accept the request in the Customer AWS Account. Prerequisites Initiate the VPC peer request. Procedure Log in to the AWS Web Console. Navigate to VPC Service . Go to Peering Connections . Click on Pending peering connection . Confirm the AWS Account and VPC ID that the request originated from. This should be from the Red Hat OpenShift Service on AWS AWS Account and Red Hat OpenShift Service on AWS Cluster VPC. Click Accept Request . 2.2.4. Configuring the routing tables After you accept the VPC peering request, both VPCs must configure their routes to communicate across the peering connection. Prerequisites Initiate and accept the VPC peer request. Procedure Log in to the AWS Web Console for the Red Hat OpenShift Service on AWS AWS Account. Navigate to the VPC Service , then Route Tables . Select the Route Table for the Red Hat OpenShift Service on AWS Cluster VPC. Note On some clusters, there may be more than one route table for a particular VPC. Select the private one that has a number of explicitly associated subnets. Select the Routes tab, then Edit . Enter the Customer VPC CIDR block in the Destination text box. Enter the Peering Connection ID in the Target text box. Click Save . You must complete the same process with the other VPC's CIDR block: Log into the Customer AWS Web Console VPC Service Route Tables . Select the Route Table for your VPC. Select the Routes tab, then Edit . Enter the Red Hat OpenShift Service on AWS Cluster VPC CIDR block in the Destination text box. Enter the Peering Connection ID in the Target text box. Click Save . The VPC peering connection is now complete. Follow the verification procedure to ensure connectivity across the peering connection is working. 2.2.5. Verifying and troubleshooting VPC peering After you set up a VPC peering connection, it is best to confirm it has been configured and is working correctly. Prerequisites Initiate and accept the VPC peer request. Configure the routing tables. Procedure In the AWS console, look at the route table for the cluster VPC that is peered. Ensure that the steps for configuring the routing tables were followed and that there is a route table entry pointing the VPC CIDR range destination to the peering connection target. If the routes look correct on both the Red Hat OpenShift Service on AWS Cluster VPC route table and Customer VPC route table, then the connection should be tested using the netcat method below. If the test calls are successful, then VPC peering is working correctly. To test network connectivity to an endpoint device, nc (or netcat ) is a helpful troubleshooting tool. It is included in the default image and provides quick and clear output if a connection can be established: Create a temporary pod using the busybox image, which cleans up after itself: USD oc run netcat-test \ --image=busybox -i -t \ --restart=Never --rm \ -- /bin/sh Check the connection using nc . Example successful connection results: / nc -zvv 192.168.1.1 8080 10.181.3.180 (10.181.3.180:8080) open sent 0, rcvd 0 Example failed connection results: / nc -zvv 192.168.1.2 8080 nc: 10.181.3.180 (10.181.3.180:8081): Connection refused sent 0, rcvd 0 Exit the container, which automatically deletes the Pod: / exit 2.3. Configuring AWS VPN This sample process configures an Amazon Web Services (AWS) Red Hat OpenShift Service on AWS cluster to use a customer's on-site hardware VPN device. Note AWS VPN does not currently provide a managed option to apply NAT to VPN traffic. See the AWS Knowledge Center for more details. Note Routing all traffic, for example 0.0.0.0/0 , through a private connection is not supported. This requires deleting the internet gateway, which disables SRE management traffic. For more information about connecting an AWS VPC to remote networks using a hardware VPN device, see the Amazon VPC VPN Connections documentation. 2.3.1. Creating a VPN connection You can configure an Amazon Web Services (AWS) Red Hat OpenShift Service on AWS cluster to use a customer's on-site hardware VPN device using the following procedures. Prerequisites Hardware VPN gateway device model and software version, for example Cisco ASA running version 8.3. See the Amazon VPC Network Administrator Guide to confirm whether your gateway device is supported by AWS. Public, static IP address for the VPN gateway device. BGP or static routing: if BGP, the ASN is required. If static routing, you must configure at least one static route. Optional: IP and Port/Protocol of a reachable service to test the VPN connection. 2.3.1.1. Configuring the VPN connection Procedure Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard, and navigate to the VPC Dashboard. Click on Your VPCs and identify the name and VPC ID for the VPC containing the Red Hat OpenShift Service on AWS cluster. From the VPC Dashboard, click Customer Gateway . Click Create Customer Gateway and give it a meaningful name. Select the routing method: Dynamic or Static . If Dynamic, enter the BGP ASN in the field that appears. Paste in the VPN gateway endpoint IP address. Click Create . If you do not already have a Virtual Private Gateway attached to the intended VPC: From the VPC Dashboard, click on Virtual Private Gateway . Click Create Virtual Private Gateway , give it a meaningful name, and click Create . Leave the default Amazon default ASN. Select the newly created gateway, click Attach to VPC , and attach it to the cluster VPC you identified earlier. 2.3.1.2. Establishing the VPN Connection Procedure From the VPC dashboard, click on Site-to-Site VPN Connections . Click Create VPN Connection . Give it a meaningful name tag. Select the virtual private gateway created previously. For Customer Gateway, select Existing . Select the customer gateway device by name. If the VPN will use BGP, select Dynamic , otherwise select Static . Enter Static IP CIDRs. If there are multiple CIDRs, add each CIDR as Another Rule . Click Create . Wait for VPN status to change to Available , approximately 5 to 10 minutes. Select the VPN you just created and click Download Configuration . From the dropdown list, select the vendor, platform, and version of the customer gateway device, then click Download . The Generic vendor configuration is also available for retrieving information in a plain text format. Note After the VPN connection has been established, be sure to set up Route Propagation or the VPN may not function as expected. Note Note the VPC subnet information, which you must add to your configuration as the remote network. 2.3.1.3. Enabling VPN route propagation After you have set up the VPN connection, you must ensure that route propagation is enabled so that the necessary routes are added to the VPC's route table. Procedure From the VPC Dashboard, click on Route Tables . Select the private Route table associated with the VPC that contains your Red Hat OpenShift Service on AWS cluster. Note On some clusters, there may be more than one route table for a particular VPC. Select the private one that has a number of explicitly associated subnets. Click on the Route Propagation tab. In the table that appears, you should see the virtual private gateway you created previously. Check the value in the Propagate column . If Propagate is set to No , click Edit route propagation , check the Propagate checkbox to the virtual private gateway's name and click Save . After you configure your VPN tunnel and AWS detects it as Up , your static or BGP routes are automatically added to the route table. 2.3.2. Verifying the VPN connection After you have set up your side of the VPN tunnel, you can verify that the tunnel is up in the AWS console and that connectivity across the tunnel is working. Prerequisites Created a VPN connection. Procedure Verify the tunnel is up in AWS. From the VPC Dashboard, click on VPN Connections . Select the VPN connection you created previously and click the Tunnel Details tab. You should be able to see that at least one of the VPN tunnels is Up . Verify the connection. To test network connectivity to an endpoint device, nc (or netcat ) is a helpful troubleshooting tool. It is included in the default image and provides quick and clear output if a connection can be established: Create a temporary pod using the busybox image, which cleans up after itself: USD oc run netcat-test \ --image=busybox -i -t \ --restart=Never --rm \ -- /bin/sh Check the connection using nc . Example successful connection results: / nc -zvv 192.168.1.1 8080 10.181.3.180 (10.181.3.180:8080) open sent 0, rcvd 0 Example failed connection results: / nc -zvv 192.168.1.2 8080 nc: 10.181.3.180 (10.181.3.180:8081): Connection refused sent 0, rcvd 0 Exit the container, which automatically deletes the Pod: / exit 2.3.3. Troubleshooting the VPN connection Tunnel does not connect If the tunnel connection is still Down , there are several things you can verify: The AWS tunnel will not initiate a VPN connection. The connection attempt must be initiated from the Customer Gateway. Ensure that your source traffic is coming from the same IP as the configured customer gateway. AWS will silently drop all traffic to the gateway whose source IP address does not match. Ensure that your configuration matches values supported by AWS . This includes IKE versions, DH groups, IKE lifetime, and more. Recheck the route table for the VPC. Ensure that propagation is enabled and that there are entries in the route table that have the virtual private gateway you created earlier as a target. Confirm that you do not have any firewall rules that could be causing an interruption. Check if you are using a policy-based VPN as this can cause complications depending on how it is configured. Further troubleshooting steps can be found at the AWS Knowledge Center . Tunnel does not stay connected If the tunnel connection has trouble staying Up consistently, know that all AWS tunnel connections must be initiated from your gateway. AWS tunnels do not initiate tunneling . Red Hat recommends setting up an SLA Monitor (Cisco ASA) or some device on your side of the tunnel that constantly sends "interesting" traffic, for example ping , nc , or telnet , at any IP address configured within the VPC CIDR range. It does not matter whether the connection is successful, just that the traffic is being directed at the tunnel. Secondary tunnel in Down state When a VPN tunnel is created, AWS creates an additional failover tunnel. Depending upon the gateway device, sometimes the secondary tunnel will be seen as in the Down state. The AWS Notification is as follows: 2.4. Configuring AWS Direct Connect This process describes accepting an AWS Direct Connect virtual interface with Red Hat OpenShift Service on AWS. For more information about AWS Direct Connect types and configuration, see the AWS Direct Connect components documentation. 2.4.1. AWS Direct Connect methods A Direct Connect connection requires a hosted Virtual Interface (VIF) connected to a Direct Connect Gateway (DXGateway), which is in turn associated to a Virtual Gateway (VGW) or a Transit Gateway in order to access a remote VPC in the same or another account. If you do not have an existing DXGateway, the typical process involves creating the hosted VIF, with the DXGateway and VGW being created in the Red Hat OpenShift Service on AWS AWS Account. If you have an existing DXGateway connected to one or more existing VGWs, the process involves the Red Hat OpenShift Service on AWS AWS Account sending an Association Proposal to the DXGateway owner. The DXGateway owner must ensure that the proposed CIDR will not conflict with any other VGWs they have associated. See the following AWS documentation for more details: Virtual Interfaces Direct Connect Gateways Associating a VGW across accounts Important When connecting to an existing DXGateway, you are responsible for the costs . There are two configuration options available: Method 1 Create the hosted VIF and then the DXGateway and VGW. Method 2 Request a connection via an existing Direct Connect Gateway that you own. 2.4.2. Creating the hosted Virtual Interface Prerequisites Gather Red Hat OpenShift Service on AWS AWS Account ID. 2.4.2.1. Determining the type of Direct Connect connection View the Direct Connect Virtual Interface details to determine the type of connection. Procedure Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region. Select Direct Connect from the Services menu. There will be one or more Virtual Interfaces waiting to be accepted, select one of them to view the Summary . View the Virtual Interface type: private or public. Record the Amazon side ASN value. If the Direct Connect Virtual Interface type is Private, a Virtual Private Gateway is created. If the Direct Connect Virtual Interface is Public, a Direct Connect Gateway is created. 2.4.2.2. Creating a Private Direct Connect A Private Direct Connect is created if the Direct Connect Virtual Interface type is Private. Procedure Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region. From the AWS region, select VPC from the Services menu. Select Virtual Private Gateways from VPN Connections . Click Create Virtual Private Gateway . Give the Virtual Private Gateway a suitable name. Select Custom ASN and enter the Amazon side ASN value gathered previously. Create the Virtual Private Gateway. Click the newly created Virtual Private Gateway and choose Attach to VPC from the Actions tab. Select the Red Hat OpenShift Service on AWS Cluster VPC from the list, and attach the Virtual Private Gateway to the VPC. From the Services menu, click Direct Connect . Choose one of the Direct Connect Virtual Interfaces from the list. Acknowledge the I understand that Direct Connect port charges apply once I click Accept Connection message, then choose Accept Connection . Choose to Accept the Virtual Private Gateway Connection and select the Virtual Private Gateway that was created in the steps. Select Accept to accept the connection. Repeat the steps if there is more than one Virtual Interface. 2.4.2.3. Creating a Public Direct Connect A Public Direct Connect is created if the Direct Connect Virtual Interface type is Public. Procedure Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region. From the Red Hat OpenShift Service on AWS AWS Account region, select Direct Connect from the Services menu. Select Direct Connect Gateways and Create Direct Connect Gateway . Give the Direct Connect Gateway a suitable name. In the Amazon side ASN , enter the Amazon side ASN value gathered previously. Create the Direct Connect Gateway. Select Direct Connect from the Services menu. Select one of the Direct Connect Virtual Interfaces from the list. Acknowledge the I understand that Direct Connect port charges apply once I click Accept Connection message, then choose Accept Connection . Choose to Accept the Direct Connect Gateway Connection and select the Direct Connect Gateway that was created in the steps. Click Accept to accept the connection. Repeat the steps if there is more than one Virtual Interface. 2.4.2.4. Verifying the Virtual Interfaces After the Direct Connect Virtual Interfaces have been accepted, wait a short period and view the status of the Interfaces. Procedure Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region. From the Red Hat OpenShift Service on AWS AWS Account region, select Direct Connect from the Services menu. Select one of the Direct Connect Virtual Interfaces from the list. Check the Interface State has become Available Check the Interface BGP Status has become Up . Repeat this verification for any remaining Direct Connect Interfaces. After the Direct Connect Virtual Interfaces are available, you can log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and download the Direct Connect configuration file for configuration on your side. 2.4.3. Connecting to an existing Direct Connect Gateway Prerequisites Confirm the CIDR range of the Red Hat OpenShift Service on AWS VPC will not conflict with any other VGWs you have associated. Gather the following information: The Direct Connect Gateway ID. The AWS Account ID associated with the virtual interface. The BGP ASN assigned for the DXGateway. Optional: the Amazon default ASN may also be used. Procedure Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region. From the Red Hat OpenShift Service on AWS AWS Account region, select VPC from the Services menu. From VPN Connections , select Virtual Private Gateways . Select Create Virtual Private Gateway . Give the Virtual Private Gateway a suitable name. Click Custom ASN and enter the Amazon side ASN value gathered previously or use the Amazon Provided ASN. Create the Virtual Private Gateway. In the Navigation pane of the Red Hat OpenShift Service on AWS AWS Account Dashboard, choose Virtual private gateways and select the virtual private gateway. Choose View details . Choose Direct Connect gateway associations and click Associate Direct Connect gateway . Under Association account type , for Account owner, choose Another account . For Direct Connect gateway owner , enter the ID of the AWS account that owns the Direct Connect gateway. Under Association settings , for Direct Connect gateway ID, enter the ID of the Direct Connect gateway. Under Association settings , for Virtual interface owner, enter the ID of the AWS account that owns the virtual interface for the association. Optional: Add prefixes to Allowed prefixes, separating them using commas. Choose Associate Direct Connect gateway . After the Association Proposal has been sent, it will be waiting for your acceptance. The final steps you must perform are available in the AWS Documentation . 2.4.4. Troubleshooting Direct Connect Further troubleshooting can be found in the Troubleshooting AWS Direct Connect documentation.

Commands:
oc run netcat-test --image=busybox -i -t --restart=Never --rm -- /bin/sh
/ nc -zvv 192.168.1.1 8080 10.181.3.180 (10.181.3.180:8080) open sent 0, rcvd 0
/ nc -zvv 192.168.1.2 8080 nc: 10.181.3.180 (10.181.3.180:8081): Connection refused sent 0, rcvd 0
/ exit
oc run netcat-test --image=busybox -i -t --restart=Never --rm -- /bin/sh
/ nc -zvv 192.168.1.1 8080 10.181.3.180 (10.181.3.180:8080) open sent 0, rcvd 0
/ nc -zvv 192.168.1.2 8080 nc: 10.181.3.180 (10.181.3.180:8081): Connection refused sent 0, rcvd 0
/ exit
You have new non-redundant VPN connections One or more of your vpn connections are not using both tunnels. This mode of operation is not highly available and we strongly recommend you configure your second tunnel. View your non-redundant VPN connections.