Chapter 9. Tutorial: AWS Load Balancer Operator on ROSA Important This content is authored by Red Hat experts, but has not yet been tested on every supported configuration. Tip Load Balancers created by the AWS Load Balancer Operator cannot be used for OpenShift Routes , and should only be used for individual services or ingress resources that do not need the full layer 7 capabilities of an OpenShift Route. The AWS Load Balancer Controller manages AWS Elastic Load Balancers for a Red Hat OpenShift Service on AWS (ROSA) cluster. The controller provisions AWS Application Load Balancers (ALB) when you create Kubernetes Ingress resources and AWS Network Load Balancers (NLB) when implementing Kubernetes Service resources with a type of LoadBalancer. Compared with the default AWS in-tree load balancer provider, this controller is developed with advanced annotations for both ALBs and NLBs. Some advanced use cases are: Using native Kubernetes Ingress objects with ALBs Integrate ALBs with the AWS Web Application Firewall (WAF) service Specify custom NLB source IP ranges Specify custom NLB internal IP addresses The AWS Load Balancer Operator is used to used to install, manage and configure an instance of aws-load-balancer-controller in a ROSA cluster. 9.1. Prerequisites Note AWS ALBs require a multi-AZ cluster, as well as three public subnets split across three AZs in the same VPC as the cluster. This makes ALBs unsuitable for many PrivateLink clusters. AWS NLBs do not have this restriction. A multi-AZ ROSA classic cluster BYO VPC cluster AWS CLI OC CLI 9.1.1. Environment Prepare the environment variables: USD export AWS_PAGER="" USD export ROSA_CLUSTER_NAME=USD(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}USD//') USD export REGION=USD(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") USD export OIDC_ENDPOINT=USD(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') USD export AWS_ACCOUNT_ID=USD(aws sts get-caller-identity --query Account --output text) USD export SCRATCH="/tmp/USD{ROSA_CLUSTER_NAME}/alb-operator" USD mkdir -p USD{SCRATCH} USD echo "Cluster: USD{ROSA_CLUSTER_NAME}, Region: USD{REGION}, OIDC Endpoint: USD{OIDC_ENDPOINT}, AWS Account ID: USD{AWS_ACCOUNT_ID}" 9.1.2. AWS VPC and subnets Note This section only applies to clusters that were deployed into existing VPCs. If you did not deploy your cluster into an existing VPC, skip this section and proceed to the installation section below. Set the below variables to the proper values for your ROSA deployment: USD export VPC_ID=<vpc-id> USD export PUBLIC_SUBNET_IDS=<public-subnets> USD export PRIVATE_SUBNET_IDS=<private-subnets> USD export CLUSTER_NAME=USD(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}") Add a tag to your cluster's VPC with the cluster name: USD aws ec2 create-tags --resources USD{VPC_ID} --tags Key=kubernetes.io/cluster/USD{CLUSTER_NAME},Value=owned --region USD{REGION} Add a tag to your public subnets: USD aws ec2 create-tags \ --resources USD{PUBLIC_SUBNET_IDS} \ --tags Key=kubernetes.io/role/elb,Value='' \ --region USD{REGION} Add a tag to your private subnets: USD aws ec2 create-tags \ --resources "USD{PRIVATE_SUBNET_IDS}" \ --tags Key=kubernetes.io/role/internal-elb,Value='' \ --region USD{REGION} 9.2. Installation Create an AWS IAM policy for the AWS Load Balancer Controller: Note The policy is sourced from the upstream AWS Load Balancer Controller policy plus permission to create tags on subnets. This is required by the operator to function. USD oc new-project aws-load-balancer-operator USD POLICY_ARN=USD(aws iam list-policies --query \ "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \ --output text) USD if [[ -z "USD{POLICY_ARN}" ]]; then wget -O "USD{SCRATCH}/load-balancer-operator-policy.json" \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json POLICY_ARN=USD(aws --region "USDREGION" --query Policy.Arn \ --output text iam create-policy \ --policy-name aws-load-balancer-operator-policy \ --policy-document "file://USD{SCRATCH}/load-balancer-operator-policy.json") fi USD echo USDPOLICY_ARN Create an AWS IAM trust policy for AWS Load Balancer Operator: USD cat <<EOF > "USD{SCRATCH}/trust-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "USD{OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"] } }, "Principal": { "Federated": "arn:aws:iam::USDAWS_ACCOUNT_ID:oidc-provider/USD{OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF Create an AWS IAM role for the AWS Load Balancer Operator: USD ROLE_ARN=USD(aws iam create-role --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" \ --assume-role-policy-document "file://USD{SCRATCH}/trust-policy.json" \ --query Role.Arn --output text) USD echo USDROLE_ARN USD aws iam attach-role-policy --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" \ --policy-arn USDPOLICY_ARN Create a secret for the AWS Load Balancer Operator to assume our newly created AWS IAM role: USD cat << EOF | oc apply -f - apiVersion: v1 kind: Secret metadata: name: aws-load-balancer-operator namespace: aws-load-balancer-operator stringData: credentials: | [default] role_arn = USDROLE_ARN web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token EOF Install the AWS Load Balancer Operator: USD cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: aws-load-balancer-operator namespace: aws-load-balancer-operator spec: upgradeStrategy: Default --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: aws-load-balancer-operator namespace: aws-load-balancer-operator spec: channel: stable-v1.0 installPlanApproval: Automatic name: aws-load-balancer-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: aws-load-balancer-operator.v1.0.0 EOF Deploy an instance of the AWS Load Balancer Controller using the operator: Note If you get an error here wait a minute and try again, it means the Operator has not completed installing yet. USD cat << EOF | oc apply -f - apiVersion: networking.olm.openshift.io/v1 kind: AWSLoadBalancerController metadata: name: cluster spec: credentials: name: aws-load-balancer-operator EOF Check the that the operator and controller pods are both running: USD oc -n aws-load-balancer-operator get pods You should see the following, if not wait a moment and retry: NAME READY STATUS RESTARTS AGE aws-load-balancer-controller-cluster-6ddf658785-pdp5d 1/1 Running 0 99s aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn 2/2 Running 0 2m4s 9.3. Validating the deployment Create a new project: USD oc new-project hello-world Deploy a hello world application: USD oc new-app -n hello-world --image=docker.io/openshift/hello-openshift Configure a NodePort service for the AWS ALB to connect to: USD cat << EOF | oc apply -f - apiVersion: v1 kind: Service metadata: name: hello-openshift-nodeport namespace: hello-world spec: ports: - port: 80 targetPort: 8080 protocol: TCP type: NodePort selector: deployment: hello-openshift EOF Deploy an AWS ALB using the AWS Load Balancer Operator: USD cat << EOF | oc apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello-openshift-alb namespace: hello-world annotations: alb.ingress.kubernetes.io/scheme: internet-facing spec: ingressClassName: alb rules: - http: paths: - path: / pathType: Exact backend: service: name: hello-openshift-nodeport port: number: 80 EOF Curl the AWS ALB Ingress endpoint to verify the hello world application is accessible: Note AWS ALB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host , please wait and try again. USD INGRESS=USD(oc -n hello-world get ingress hello-openshift-alb \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') USD curl "http://USD{INGRESS}" Example output Hello OpenShift! Deploy an AWS NLB for your hello world application: USD cat << EOF | oc apply -f - apiVersion: v1 kind: Service metadata: name: hello-openshift-nlb namespace: hello-world annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing spec: ports: - port: 80 targetPort: 8080 protocol: TCP type: LoadBalancer selector: deployment: hello-openshift EOF Test the AWS NLB endpoint: Note NLB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host , please wait and try again. USD NLB=USD(oc -n hello-world get service hello-openshift-nlb \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') USD curl "http://USD{NLB}" Example output Hello OpenShift! 9.4. Cleaning up Delete the hello world application namespace (and all the resources in the namespace): USD oc delete project hello-world Delete the AWS Load Balancer Operator and the AWS IAM roles: USD oc delete subscription aws-load-balancer-operator -n aws-load-balancer-operator USD aws iam detach-role-policy \ --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" \ --policy-arn USDPOLICY_ARN USD aws iam delete-role \ --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" Delete the AWS IAM policy: USD aws iam delete-policy --policy-arn USDPOLICY_ARN

Commands:
export AWS_PAGER="" export ROSA_CLUSTER_NAME=USD(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}USD//') export REGION=USD(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") export OIDC_ENDPOINT=USD(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') export AWS_ACCOUNT_ID=USD(aws sts get-caller-identity --query Account --output text) export SCRATCH="/tmp/USD{ROSA_CLUSTER_NAME}/alb-operator" mkdir -p USD{SCRATCH} echo "Cluster: USD{ROSA_CLUSTER_NAME}, Region: USD{REGION}, OIDC Endpoint: USD{OIDC_ENDPOINT}, AWS Account ID: USD{AWS_ACCOUNT_ID}"
export VPC_ID=<vpc-id> export PUBLIC_SUBNET_IDS=<public-subnets> export PRIVATE_SUBNET_IDS=<private-subnets> export CLUSTER_NAME=USD(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}")
aws ec2 create-tags --resources USD{VPC_ID} --tags Key=kubernetes.io/cluster/USD{CLUSTER_NAME},Value=owned --region USD{REGION}
aws ec2 create-tags --resources USD{PUBLIC_SUBNET_IDS} --tags Key=kubernetes.io/role/elb,Value='' --region USD{REGION}
aws ec2 create-tags --resources "USD{PRIVATE_SUBNET_IDS}" --tags Key=kubernetes.io/role/internal-elb,Value='' --region USD{REGION}
oc new-project aws-load-balancer-operator POLICY_ARN=USD(aws iam list-policies --query "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" --output text) if [[ -z "USD{POLICY_ARN}" ]]; then wget -O "USD{SCRATCH}/load-balancer-operator-policy.json" https://raw.githubusercontent.com/rh-mobb/documentation/main/content/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json POLICY_ARN=USD(aws --region "USDREGION" --query Policy.Arn --output text iam create-policy --policy-name aws-load-balancer-operator-policy --policy-document "file://USD{SCRATCH}/load-balancer-operator-policy.json") fi echo USDPOLICY_ARN
cat <<EOF > "USD{SCRATCH}/trust-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "USD{OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"] } }, "Principal": { "Federated": "arn:aws:iam::USDAWS_ACCOUNT_ID:oidc-provider/USD{OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF
ROLE_ARN=USD(aws iam create-role --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" --assume-role-policy-document "file://USD{SCRATCH}/trust-policy.json" --query Role.Arn --output text) echo USDROLE_ARN aws iam attach-role-policy --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" --policy-arn USDPOLICY_ARN
cat << EOF | oc apply -f - apiVersion: v1 kind: Secret metadata: name: aws-load-balancer-operator namespace: aws-load-balancer-operator stringData: credentials: | [default] role_arn = USDROLE_ARN web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token EOF
cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: aws-load-balancer-operator namespace: aws-load-balancer-operator spec: upgradeStrategy: Default --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: aws-load-balancer-operator namespace: aws-load-balancer-operator spec: channel: stable-v1.0 installPlanApproval: Automatic name: aws-load-balancer-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: aws-load-balancer-operator.v1.0.0 EOF
cat << EOF | oc apply -f - apiVersion: networking.olm.openshift.io/v1 kind: AWSLoadBalancerController metadata: name: cluster spec: credentials: name: aws-load-balancer-operator EOF
oc -n aws-load-balancer-operator get pods
NAME READY STATUS RESTARTS AGE aws-load-balancer-controller-cluster-6ddf658785-pdp5d 1/1 Running 0 99s aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn 2/2 Running 0 2m4s
oc new-project hello-world
oc new-app -n hello-world --image=docker.io/openshift/hello-openshift
cat << EOF | oc apply -f - apiVersion: v1 kind: Service metadata: name: hello-openshift-nodeport namespace: hello-world spec: ports: - port: 80 targetPort: 8080 protocol: TCP type: NodePort selector: deployment: hello-openshift EOF
cat << EOF | oc apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello-openshift-alb namespace: hello-world annotations: alb.ingress.kubernetes.io/scheme: internet-facing spec: ingressClassName: alb rules: - http: paths: - path: / pathType: Exact backend: service: name: hello-openshift-nodeport port: number: 80 EOF
INGRESS=USD(oc -n hello-world get ingress hello-openshift-alb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') curl "http://USD{INGRESS}"
Hello OpenShift!
cat << EOF | oc apply -f - apiVersion: v1 kind: Service metadata: name: hello-openshift-nlb namespace: hello-world annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing spec: ports: - port: 80 targetPort: 8080 protocol: TCP type: LoadBalancer selector: deployment: hello-openshift EOF
NLB=USD(oc -n hello-world get service hello-openshift-nlb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') curl "http://USD{NLB}"
Hello OpenShift!
oc delete project hello-world
oc delete subscription aws-load-balancer-operator -n aws-load-balancer-operator aws iam detach-role-policy --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator" --policy-arn USDPOLICY_ARN aws iam delete-role --role-name "USD{ROSA_CLUSTER_NAME}-alb-operator"
aws iam delete-policy --policy-arn USDPOLICY_ARN