Experimenting with Ingress Controllers on Oracle Container Engine (OKE) — Part 2
By: Ali Mukadam
Tags: Kubernetes, Oracle Cloud, Nginx Ingress, Haproxy, Traefik

Experimenting with Ingress Controllers on Oracle Container Engine (OKE) — Part 2

Using multiple controllers and load balancers with OKE

In Part 1, I briefly described what Ingress and Ingress Controllers are. We also took on a spin on OKE some of the most popular Ingress Controllers for Kubernetes, namely:

In this post, we’ll look at Ingress class and how they can be used to deploy multiple instances of Ingress Controllers concurrently. We’ll also briefly look at a different type of Kubernetes service (ExternalName) and one of the ways you would use it. Finally, we look at how they can be used with public and internal load balancers.

All the yamls in the exercises below can be found on github.

Using multiple Ingress Controllers

Which Ingress Controller to use is a difficult question to answer. As usual, it depends on your needs, your team skillset (e.g. your team maybe more familiar with NGINX or HAProxy than Traefik or Contour), the type of applications you are deploying, the technical and operational features you need, protocols supported, the maturity of the controller or the helm package (if that’s how you are deploying), your appetite for experimenting and risk, internal approvals and so on. This excellent Kubedex guide with its summary of features can help you narrow the list to evaluate.

Regardless, it is possible to deploy multiple controllers in a single Kubernetes cluster. It is also possible to deploy multiple instances of the same controller in the same cluster by specifying a different Ingress class. So, let’s look at how the Ingress class works.

Recall that an Ingress Controller listens for changes and update its routing rules. However, when you deploy multiple Ingress Controllers and Ingresses, there has to be a way to ensure that a specific Ingress is picked up by the right controller. That’s the purpose of the Ingress class. Think of the Ingress class as an instruction to the all the deployed Ingress Controllers:

you are responsible for implementing my routing rules

you are not responsible for implementing my routing rules

The diagram below illustrates this. When an Ingress class value matches that of a controller instance, that controller instance will handle the routing rules for your Ingress. If you have other controller instances, whether of the same or different type but with different Ingress class values from your Ingress, they will ignore that Ingress.

Ingress classes and Ingress mapping

Specifying the Ingress class through annotation

The Ingress API has an annotation which allows you to specify this Ingress class:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-blog-ingress

annotations:

kubernetes.io/ingress.class: "ingressclass"

Except for traefik (see below), each controller has a default Ingress class value but also allows you to specify your own custom class. This is useful, especially when you have to deploy multiple instances of them in a single cluster. Below are the default Ingress class values for each when you use the respective helm charts:

nginx: nginx

haproxy: haproxy

traefik:

contour: contour

If you use traefik’s helm chart as described in Part 1, no ingress class is set by default. You need to explicitly specify it during your deployment as follows:

helm install stable/traefik --name traefikcontroller --set kubernertes.ingressClass=traefik

Finally, note that if you use Contour, you do not set the annotation kubernetes.io/ingress.class on the IngressRoute. More on Contour and its Ingress class later.

Using the default Ingress class annotations in your Ingresses

The examples below show the default annotation you need to provide for each of the four controllers.

NGINX Ingress Controller:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-website

annotations:

kubernetes.io/ingress.class: "nginx"

...

HAProxy:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-website

annotations:

kubernetes.io/ingress.class: "haproxy"

...

Traefik:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-website

annotations:

kubernetes.io/ingress.class: "traefik"

...

Contour:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-website

annotations:

kubernetes.io/ingress.class: "contour"

...

Customizing the Ingress class

Let’s now look at how to customize the Ingress class for each controller during their installation with the helm charts.

NGINX:

helm install --name acmecontroller stable/nginx-ingress --set controller.ingressClass=mycustomnginx

You can set any value for the NGINX Ingress class.

HAProxy:

helm install --name acmecontroller incubator/haproxy-ingress --set controller.ingressClass=mycustomhaproxy

Similarly, you can set any value for HAProxy’s Ingress class.

Traefik:

helm install stable/traefik --name traefikcontroller --set kubernertes.ingressClass=traefikcustom

With Traefik, the Ingress class value has to start with ‘traefik’. I’m not sure whether it’s the helm package or Traefik itself. This is based on Traefik’s helm chart documentation.

Contour:

kubectl apply -f examples/deployment-grpc-v2/ --ingress-class-name=newcontour

Similar to NGINX and HAProxy, you can set any value for Contour’s Ingress class. However, note the following applies according to Contour’s documentation:

contour.heptio.com/ingress.class : The Ingress class that should interpret and serve the IngressRoute. If not set, then all Contour instances serve the IngressRoute. If specified as contour.heptio.com/ingress.class: contour , then Contour serves the IngressRoute. If any other value, Contour ignores the IngressRoute definition. You can override the default class contour with the --ingress-class-name flag at runtime

So, if you’re planning to use Contour’s IngressRoute API, you must set this value to ‘contour’ or not set it at all.

Using multiple controllers in action

I said that it was possible to have multiple Ingress Controllers deployed simultaneously on a single cluster. Let’s look at how we can use them concurrently.

Pick any 2 controllers you like and deploy them. In this example, we will use NGINX and Traefik for illustration but you can use any combination as long as you specify the correct Ingress class e.g. you can also have 2 NGINX controllers if you want to as long as you set a different class name for the 2nd controller.

This is how we want to deploy our artifacts:

acme services (website, blog) in the acme namespace

nginx controller and website ingress in the web namespace

traefik controller and blog ingress in the blog namespace

The following diagram illustrates our deployment.

Using multiple controllers and public load balancers

Let’s first create the namespaces:

kubectl create ns acme

kubectl create ns web

kubectl create ns blog

We can now deploy the website and blog.

kubectl create -f ingresscontrollers/multiplecontrollers/acme-website.yaml

kubectl create -f ingresscontrollers/multiplecontrollers/acme-blog.yaml

Let’s now deploy our 2 Ingress Controllers (NGINX and Traefik).

NGINX:

helm install --name webcontroller stable/nginx-ingress \

--namespace web \

--set defaultBackend.enabled=true \

--set defaultBackend.name=acmedefaultbackend \

--set rbac.create=true

Verify the NGINX controller has been deployed properly:

kubectl -n web get pods

NAME READY STATUS RESTARTS AGE

webcontroller-nginx-ingress-acmedefaultbackend-86689bd54b-dzpcg 1/1 Running 0 5m45s

webcontroller-nginx-ingress-controller-8bdc8c665-f6xqm 1/1 Running 0 5m45s

And retrieve the LoadBalancer’s public IP Address:

kubectl -n web get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

webcontroller-nginx-ingress-acmedefaultbackend ClusterIP 10.96.122.109 80/TCP 46s app=nginx-ingress,component=acmedefaultbackend,release=we

bcontroller

webcontroller-nginx-ingress-controller LoadBalancer 10.96.45.231 129.146.208.247 80:30698/TCP,443:32651/TCP 46s app=nginx-ingress,component=controller,release=webcontrol

ler

Traefik:

helm install --name blogcontroller stable/traefik \

--namespace blog \

--set rbac.enabled=true \

--set kubernetes.ingressClass=traefik

Verify the Traefik controller has been deployed properly:

kubectl -n blog get pods

NAME READY STATUS RESTARTS AGE

blogcontroller-traefik-65579ddcb4-7hqdz 1/1 Running 0 6m30s

And retrieve the LoadBalancer’s public IP Address:

kubectl -n blog get svc -o wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

blogcontroller-traefik LoadBalancer 10.96.127.166 129.146.89.209 80:30293/TCP,443:30455/TCP 69s app=traefik,release=blogcontroller

Update your DNS ‘A’ records so that they match the following:

129.146.208.247 (used by the Load Balancer created by NGINX Ingress Controller) maps to www.acme.com

129.146.89.209 (used by the Load Balancer created by Traefik’s Ingress Controller) maps to blog.acme.com

Note:

Your Load Balancer public IP addresses will be different, remember to use yours

Replace acme.com with your domain

Let’s now create the Ingresses, starting with the website Ingress using the same Ingress code as in the examples in Part 1.

Open a new terminal to watch the NGINX controller logs (replace with the name of your controller pod):

kubectl -n web log webcontroller-nginx-ingress-controller-8bdc8c665-f6xqm -f ... I0712 00:36:36.398883 8 status.go:86] new leader elected: webcontroller-nginx-ingress-controller-8bdc8c665-f6xqm

We’ll now create the following Ingress (replace acme.com with that of your domain):

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-website

namespace: web

annotations:

kubernetes.io/ingress.class: "nginx"

spec:

rules:

- host: www.acme.com

http:

paths:

- path: /

backend:

serviceName: acme-website

servicePort: 80

On your first terminal, create the Ingress:

kubectl create -f multiplecontrollers/acme-website-ingress.yaml

In the watch terminal, you’ll notice the following:

I0712 00:50:13.540522 8 controller.go:133] Configuration changes detected, backend reload required.

I0712 00:50:13.540695 8 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"acme", Name:"acme-website", UID:"02051a8b-a43f-11e9-a88d-0a580aed1b23", APIVersion:"extensions

/v1beta1", ResourceVersion:"192541", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress acme/acme-website

I0712 00:50:13.624379 8 controller.go:149] Backend successfully reloaded.

[12/Jul/2019:00:50:13 +0000]TCP200000.000

I0712 00:50:36.427547 8 status.go:309] updating Ingress acme/acme-website status from [] to [{ }]

I0712 00:50:36.434886 8 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"acme", Name:"acme-website", UID:"02051a8b-a43f-11e9-a88d-0a580aed1b23", APIVersion:"extensions

/v1beta1", ResourceVersion:"192593", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress acme/acme-website

And now if you access the website URL, you’ll be able to access the website page. If you are still watching the controller pod logs, you’ll also notice the incoming traffic:

10.244.0.0 - [10.244.0.0] - - [12/Jul/2019:00:51:26 +0000] "GET / HTTP/1.1" 200 1855 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" 388 0.002 [acme-acme

-website-80] [] 10.244.2.13:80 4446 0.002 200 0bed30fdf9d16bde887ed8e488b80735

10.244.0.0 - [10.244.0.0] - - [12/Jul/2019:00:51:27 +0000] "GET /favicon.ico HTTP/1.1" 404 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" 320 0.001

[acme-acme-website-80] [] 10.244.2.13:80 178 0.001 404 d91af2a400fd4230ec3dec8b0eaf713f

One important thing to note here is that the Ingress is defined in the same namespace as the service. If you create the Ingress in a namespace different from the targeted service, the Ingress Controller will still be able to detect the Ingress change. However, it won’t be able to obtain the endpoints and therefore won’t be able to update its configuration.

To illustrate this, we’ll deliberately create the blog Ingress in the blog namespace i.e. that of traefik.

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-blog-ingress

namespace: blog

annotations:

kubernetes.io/ingress.class: "traefik"

spec:

rules:

- host: blog.acme.com

http:

paths:

- path: /

backend:

serviceName: acme-blog

servicePort: 80

On your 2nd terminal, stop watching the NGINX controller and start watching the traefik controller (replace with the name of your controller pod) instead:

kubectl -n blog log blogcontroller-traefik-65579ddcb4-7hqdz -f

Let’s now create the blog Ingress (remember to change the hostname in the Ingress rule):

kubectl create -f acme-blog-ingress.yaml

You’ll see the following in the traefikcontroller logs:

{"level":"error","msg":"Service not found for blog/acme-blog","time":"2019-07-12T01:02:20Z"}

Delete the blog Ingress:

kubectl delete -f acme-blog-ingress.yaml

Edit its namespace to acme:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-blog-ingress

namespace: acme

annotations:

kubernetes.io/ingress.class: "traefik"

spec:

rules:

- host: blog.acme.com

http:

paths:

- path: /

backend:

serviceName: acme-blog

servicePort: 80

And recreate the Ingress:

kubectl create -f acme-blog-ingress.yaml

Inspect the controller log again:

kubectl -n blog log blogcontroller-traefik-65579ddcb4-7hqdz -f {"level":"info","msg":"Server configuration reloaded on :8880","time":"2019-07-12T02:01:36Z"}

And now if we access the blog URL, we can also access the blog.

Let’s clean up before our final experiment:

kubectl delete -f acme-blog-ingress.yaml

kubectl delete -f acme-website-ingress.yaml

helm delete --purge blogcontroller

helm delete --purge webcontroller

Using a single public load balancer, private load balancers with multiple controllers

As you might have noticed in the previous experiments, whenever you deploy an Ingress Controller, a public Load Balancer is created by default. Deploying multiple controllers means you will have multiple public load balancers. This can have an obvious impact on costs and although OCI is quite generous in terms of data charges, you as the superadmin, want to keep a close eye on it.

Now, let’s say you have 2 teams (Team Web and Team Blog), each with their favourite Ingress Controller (NGINX and Traefik respectively) as above and for whatever technical reasons, they both insist on using their favourites. 2 cannot become 1 despite what the Spice Girls sing.

Well, it turns out you can use internal Load Balancers by setting the right annotations. We used Load Balancer annotations before in an earlier post but for changing the shape of the Load Balancer. But how to do this with Ingress Controllers and then expose the applications publicly?

Below is a slight modification of the previous deployment architecture:

Using a public load balancer and multiple private load balancers

In the above diagram, when the user accesses either the website or the blog URLs, they will hit the public load balancer. We also want the web and blog traffic to go through the internal load balancers.

We’ll use the following 3 controllers this time:

nginx-ingress: private, website

traefik: private, blog

haproxy: frontend load balancer

Let us first deploy nginx-ingress and traefik in internal mode:

helm install --name webcontroller stable/nginx-ingress \

--namespace web \

--set controller.name=web \

--set rbac.create=true \

--set controller.ingressClass=nginx \

--set controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"=true

By adding the annotation above, the load balancer created by NGINX Ingress Controller will be deployed with a private IP address only. Only, requests originating from within the VCN can reach it.

Let’s do the same for Traefik:

helm install --name blogcontroller stable/traefik \

--namespace blog \

--set rbac.enabled=true \

--set kubernetes.ingressClass=traefik \

--set service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal=true"

And finally, let’s create a frontend namespace and deploy HAProxy:

kubectl create ns frontend helm install --name frontendcontroller incubator/haproxy-ingress \

--namespace frontend \

--set defaultBackend.enabled=true \

--set defaultBackend.name=defaultbackend \

--set rbac.create=true \

--set controller.ingressClass=haproxy

For HAProxy, we don’t set the oci-load-balancer-internal=true annotation because we want it to be publicly accessible. When you login to OCI Console, you can verify this:

Public and Private Load Balancers

Now, we need to deploy the services and ingresses. Recall that we want the traffic to route through the private load balancers. I also mentioned before that your Ingresses should be in the same namespace as your service. Since we want to force the traffic through a particular path i.e. the internal load balancers, we’ll use an alternative method to the above. In this case, we’ll use ExternalName. As the documentation explains, services of type ExternalName map a service to a DNS name.

Here’s the explanation from the documentation:

When looking up the host my-service.prod.svc.cluster.local , the cluster DNS Service returns a CNAME record with the value my.database.example.com . Accessing my-service works in the same way as other Services but with the crucial difference that redirection happens at the DNS level rather than via proxying or forwarding.

apiVersion: v1

kind: Service

metadata:

name: my-service

namespace: prod

spec:

type: ExternalName

externalName: my.database.example.com

However, there’s also this little note in the Kubernetes documentation:

ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name.

Since ExternalNames that resemble IPv4 addresses are not resolved, we can use IPv4 addresses instead to force the traffic through the internal load balancers.

The website and blog services in the acme namespace are still there, so we only need to create the following:

an ingress for the website with class “nginx”. This will allow the internal nginx controller to do the necessary routing to the website service. an ingress for the blog with class “traefik”. This will allow the internal traefik controller to do the necessary routing to the blog service.

The above 2 ingresses are the same as in the previous exercise when using 2 public load balancers. We could have left the 2 ingresses as is and not delete them during clean up but I want to make this clearer here what is happening in the current scenario.

We’ll also create:

an acme-website service of type ExternalName whose externalName value is the private IP address of the internal load balancer created by the NGINX controller an acme-blog service of type ExternalName whose externalName value is the private IP address of the internal load balancer created by the Traefik controller. And finally 2 ingresses of class “haproxy” whose target services are the 2 ExternalName services for the website and the blog.

All the yaml files can be found under the ingresscontrollers/private directory.

First, edit all the ingresses and change ‘acme.com’ to your domain as before.

Next obtain the private IP addresses of the load balancers:

Once the nginx-ingress and traefik controllers are deployed, obtain their private IP Addresses.

Nginx:

kubectl -n web get svc -o wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

webcontroller-nginx-ingress-default-backend ClusterIP 10.96.44.162 80/TCP 29s app=nginx-ingress,component=default-backend,release=webcontroller

webcontroller-nginx-ingress-web LoadBalancer 10.96.54.89 10.0.12.23 80:30626/TCP,443:30352/TCP 29s app=nginx-ingress,component=web,release=webcontroller

Traefik:

kubectl -n blog get svc -o wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

blogcontroller-traefik LoadBalancer 10.96.236.78 10.0.12.26 80:30668/TCP,443:32269/TCP 86s app=traefik,release=blogcontroller

Edit the frontend-website-ingress.yaml and change the externalName value:

kind: Service

apiVersion: v1

metadata:

name: acme-website

namespace: frontend

spec:

type: ExternalName

externalName: 10.0.12.23

ports:

- port: 80

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-website

namespace: frontend

annotations:

kubernetes.io/ingress.class: "haproxy"

spec:

rules:

- host: www.acme.com

http:

paths:

- path: /

backend:

serviceName: acme-website

servicePort: 80

Repeat for frontend-website-ingress.yaml and change the externalName value.

kind: Service

apiVersion: v1

metadata:

name: acme-blog

namespace: public

spec:

type: ExternalName

externalName: 10.0.12.26

ports:

- port: 80

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: acme-blog

namespace: public

annotations:

kubernetes.io/ingress.class: "haproxy"

spec:

rules:

- host: blog.acme.com

http:

paths:

- path: /

backend:

serviceName: acme-website

servicePort: 80

Apply the manifests:

kubectl create -f ingresscontrollers/private/ ingress.extensions/acme-blog-ingress created

deployment.apps/acme-blog created

service/acme-blog created

ingress.extensions/acme-website created

deployment.apps/acme-website created

service/acme-website created

service/acme-blog created

ingress.extensions/acme-blog created

service/acme-website created

ingress.extensions/acme-website created

Finally, get the public IP address of your loadbalancer either from the OCI Console or with kubectl:

kubectl -n frontend get svc -o wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

frontendcontroller-haproxy-ingress-controller LoadBalancer 10.96.41.118 129.146.95.188 80:30746/TCP,443:30777/TCP 83s app=haproxy-ingress,component=controller,release=fronten

dcontroller

frontendcontroller-haproxy-ingress-defaultbackend ClusterIP 10.96.158.93 8080/TCP 83s app=haproxy-ingress,component=defaultbackend,release=fro

ntendcontroller

Update the 2 entries in your OCI DNS for the website and the blog (replacing ‘acme.com’ with your domain) and set the IP Address to that of the public Load Balancer:

www.acme.com

blog.acme.com

Remember to click on ‘Publish Changes’ to make it effective and test access to your website and the blog.

Summary

We have experimented with 4 different Ingress Controllers on OKE. We then looked at Ingress class and its purpose and deployed multiple controllers simultaneously on a cluster and used the Ingress class to ensure the requests are routed accordingly. Finally, we looked at how private Load Balancers can also be used with multiple Ingress Controllers and then subsequently exposing the services publicly.

The examples and reasons we used are rather trivial and contrived but I hope they serve to illustrate the flexibility you have when you deploy your applications on Kubernetes and Oracle Cloud Infrastructure.

Additional References:

NGINX and multiple Ingress Controllers

Deploying Traefik using helm with service annotation

OCI Documentation: Creating Internal Load Balancers

Ingress Controller Comparison

Kubernetes Cross Namespace Ingress