How to Expose Your Services With Kubernetes Ingress
By: Randal Kamradt Sr
Tags: Kubernetes, Cert Manager, DevOps, Programming, Nginx

How to Expose Your Services With Kubernetes Ingress

Allow your applications to talk to outsiders

Photo by Dima Pechurin on Unsplash

In my previous posts about Kubernetes, in order to expose my services to my home network, I used a load balancer service called MetalLB to expose the service to the VM bridge and an instance of Nginx on the main host as a reverse proxy to my home network. But there are some limitations to that. Kubernetes is not aware of Nginx, and can’t control its configuration, so you have to manually configure it. Also, MetalLB steals away some IP addresses from the VM bridge, and the number of IPs you configure it with, you have to guess at. If you need more than allocated, you’d have to go back in and configure MetalLB again.

It would be nice to have services that are defined as ingress routes directly exposed to your home network. In order to do that, you need an ingress controller and a route from the main network to the Kubernetes bridge network. We’ll start with the ingress controller.

To follow along with this article, you’ll need a setup similar to the one I created with my article Kubernetes from Scratch. In it, I created a bare-bones Kubernetes system on VMs running on bare-metal. My intention for this article is to help understand what’s going on behind the scenes by not using a pre-packaged system or one offered by a cloud provider. I’m also trying to be cloud provider agnostic, so knowledge gleaned from this article will help you better understand Kubernetes no matter what cloud providers you are using. You will also need an understanding of Linux systems, especially networking.

First, we need something to ingress to. I happen to have a simple service that listens for HTTP get requests and responds with “hello world”. Create a file called simple-service.yaml and add the following:

apiVersion: apps/v1

kind: Deployment

metadata:

name: hellok8s-deployment

labels:

app: hellok8s

spec:

selector:

matchLabels:

app: hellok8s

template:

metadata:

labels:

app: hellok8s

spec:

containers:

- name: hellok8s

image: docker.io/rlkamradt/hellok8s:latest

ports:

- containerPort: 8080

---

apiVersion: v1

kind: Service

metadata:

name: hellok8s-service

spec:

type: ClusterIP

selector:

app: hellok8s

ports:

- port: 8080

targetPort: 8080

That’s about as simple as you get in Kubernetes. Now, run kubectl apply -f simple-service.yaml That will start up the service, and expose port 8080. But expose where? If you run curl http://localhost:8080/ on the main host, you’ll get nothing. You can look at the service:

rkamradt@beast:~/scratch$ kubectl get service hellok8s-service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

hellok8s-service ClusterIP 10.111.98.155 <none> 8080/TCP 80s

Notice that it has a cluster IP but not an external IP. You can try curling with the cluster IP, but you won’t get anything.

Let’s try one other option. If you have access to your nodes (my nodes are called kube1–4) you can ssh to them. Now run the curl with your cluster IP:



Hello World rkamradt@kube1:~$ curl http://10.111.98.155:8080/ Hello World

You can try that on each node, and you’ll get the same answer. So one of the functions of the ingress controller is to expose the service back out to the main host.

Now we must pick an ingress controller, of which there are several. If you are running with a cloud provider, they will normally have a preferred one (or only one). But since I’m my own cloud provider, I have to pick one. I’m picking the semi-official Nginx ingress controller. You can install it straight from the Kubernetes GitHub page:

This creates a namespace called ingress-nginx . You can see all the objects it creates with kubectl get all -n ingress-nginx .

rkamradt@beast:~/scratch$ kubectl get all -n ingress-nginx

NAME READY STATUS RESTARTS AGE

pod/ingress-nginx-admission-create-lfc2b 0/1 Completed 0 89s

pod/ingress-nginx-admission-patch-rmcvq 0/1 Completed 0 89s

pod/ingress-nginx-controller-5f98fb55b8-7pnjt 1/1 Running 0 99s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/ingress-nginx-controller NodePort 10.96.223.195 <none> 80:32688/TCP,443:32569/TCP 99s

service/ingress-nginx-controller-admission ClusterIP 10.100.112.59 <none> 443/TCP 99s NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/ingress-nginx-controller 1/1 1 1 99s NAME DESIRED CURRENT READY AGE

replicaset.apps/ingress-nginx-controller-5f98fb55b8 1 1 1 99s NAME COMPLETIONS DURATION AGE

job.batch/ingress-nginx-admission-create 1/1 8s 99s

job.batch/ingress-nginx-admission-patch 1/1 8s 99s

A couple of things to note: first, it has two jobs that run once and then leave their pods around, perhaps for diagnostic purposes. They may have a TTL, or we may just need to get rid of them after a while. I’ll have to look into that when I get sufficiently annoyed seeing them hanging around.

The other thing to note is that it created a NodePort service, which would require you to forward with kubectl port-forward . I suppose it does this because it can’t expect that a load balancer would be configured, but in our case, we have MetalLB. We need to rectify that by editing the service/ingress-nginx-controller service. Run kubectl edit service ingress-nginx-controller -n ingress-nginx . Find the spec.type field and change it from NodePort to LoadBalancer . Now when we run kubectl get all -n ingress-nginx we can see the type has changed to LoadBalancer and that it has an external IP address.

service/ingress-nginx-controller LoadBalancer 10.96.223.195 192.168.122.240 80:32688/TCP,443:32569/TCP 4m36s

The IP address was allocated from MetalLB, yours will probably be different. If we curl that address, we’ll get:



<html>

<head><title>404 Not Found</title></head>

<body>

<center><h1>404 Not Found</h1></center>

<hr><center>nginx/1.17.8</center>

</body>

</html> rkamradt@beast:~/scratch$ curl http://192.168.122.240 404 Not Found 404 Not Found nginx/1.17.8

Now we can configure our ingress point. There are a lot of different options including virtual hosts, path redirect, and https termination. But we’ll start very simple indeed. Create a file called simple-ingress.yaml and add the following to it:

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

name: test-ingress

annotations:

nginx.ingress.kubernetes.io/rewrite-target: /

spec:

rules:

- http:

paths:

- path: /testpath

pathType: Prefix

backend:

serviceName: hellok8s-service

servicePort: 8080

Start it up with kubectl apply -f simple-ingress.yaml . This will configure the ingestion service to forward all requests that have /testpath as the path to the hellok8s-service . Try it out:



Hello World rkamradt@beast:~/scratch$ curl http://192.168.122.240/testpath Hello World

But curling again without the path /testpath it’s still the 404 page:



<html>

<head><title>404 Not Found</title></head>

<body>

<center><h1>404 Not Found</h1></center>

<hr><center>nginx/1.17.8</center>

</body>

</html> rkamradt@beast:~/scratch$ curl http://192.168.122.240 404 Not Found 404 Not Found nginx/1.17.8

We could change the ingress rules and have different paths go to different services, or use virtual hosts that go to different services based on the Host header. You can also define a default ingress in case you don’t like the standard Nginx 404 page.

But there’s still a problem: if I use the browser on my laptop to go to http://192.168.122.240/testpath , I’ll get an error. We need to somehow route to the 192.168.122.0/24 network from our home network. That’s going to require a bit of skill in Linux networking.

Where does the 192.168.122.0/24 network come from? That’s KVM’s default bridge network. All the VMs we create get IP addresses assigned to that range. MetalLB also uses that network to assign addresses within a certain pool (my configuration is 192.168.122.240-192.168.122.250 ) You can find the addresses that KVM assigns from by running sudo virsh net-edit default . You can see the output and ensure that it doesn’t interfere with the range used by MetalLB:

<network>

<name>default</name>

<uuid>dc658641-aded-465d-b472-1cc427c76626</uuid>

<forward mode='nat'/>

<bridge name='virbr0' stp='on' delay='0'/>

<mac address='52:54:00:1d:5b:25'/>

<ip address='192.168.122.1' netmask='255.255.255.0'>

<dhcp>

<range start='192.168.122.2' end='192.168.122.232'/>

</dhcp>

</ip>

</network>

I set my dhcp range to end at 192.168.122.232 so it wouldn’t interfere with the range from MetalLB. You’ll also see that the forward mode is nat so it acts similar to your home router, traffic can go out, but the outside world can’t initiate a connection.

But the main host has an ‘in’ to this network, it can access the bridge without going through NAT. If the main host can see it, I should be able to play around with iptables and port forward. Be sure to shut off any service on your main host that might be listening on port 80 or 443. Here are the commands that I use to accomplish that:

You’ll have to use sudo /bin/bash to run all these commands as root (or prefix each with sudo ). Test it out by opening http://<mainhost>/testpath in your browser. It worked the first time for me! Then you can make your changes permanent. I was able to use apt-get install iptables-persistent which saved the rules to /etc/iptables/rules.v4 which are automatically loaded on reboot. Your particular distribution of Linux may require different ways to save.

You might be tempted to open up additional ports as well, but by opening up just the two, you can keep your ingress security simple. I might even remove the ingress to port 80 and force people to use 443. That is, as soon as I get HTTPS set up.

Which brings me to my next section: setting up HTTPS. Now, HTTPS, SSL, and TLS have given great security to the internet. But they have also caused many programmers to pull out their hair. Even if the applications you write don’t terminate HTTPS, eventually you’ll be called upon to make calls to an HTTPS service with a funky certificate. Why are certificates so hard to deal with? First, they need to come from official certificate authorities (CA). Second, the common name (CN) value in the certificate must match the hostname to ensure that the hostname you typed in your browser matches the certificate offered by the server. This is true not just for browser access, but for application access.

One way to alleviate this situation is to have one server provide all ingress and terminate HTTPS. When I say “terminate HTTPS”, that means that the ingress server provides the certificate and then forwards requests to the application in plain HTTP. This is what I like to call “exoskeleton security,” because your security is provided by the fact that you have only one ingress point, and it can be locked down pretty tightly.

The fact that we can’t route to the KVM bridge network — except by making specific rules in the IP tables of the main host — gives us that security. It’s a little harder when you have multiple hosts or cloud hosts where you don’t know exactly where they are, but the subnet isolation is still there and should provide the exoskeleton you need. It also alleviates the need to have HTTPS handling in your application, and that’s a security risk in and of itself.

If we browse to our new service using HTTPS on Chrome, we get this screen:

Other browsers may present different screens. It’s meant to look scary; if this was a real website on the internet, I would definitely be hitting the “back to safety” button. When Nginx is installed, it will create a self-signed certificate. Clicking the “Advanced” button gives us more information:

Now you can set up Chrome to trust your certificate, but because there’s no standard place for trusted certificates to be stored, each browser you might use (and each programming language that might read from this URL) needs to be configured to trust this certificate.

By clicking on the “Proceed to beast (unsafe)” link, you’ll get your “Hello World”. If you click on the “Not Secure” next to the URL, it will show you why it thinks it’s not secure.

Click on the “Certificate (Invalid)” link and you’ll see the certificate Nginx created for you.

So, not only is this certificate not verified by a third party (a CA), the Common Name, which should be the hostname, is “Kubernetes Ingress Controller Fake Certificate.”

As stated above, this isn’t a tutorial for creating a production system. It’s for learning the nuts-and-bolts of Kubernetes, so you’ll be better prepared to create a production system. We could just leave it as is and make you click through the warnings on every access. (I’ve been in many workplaces that made you do that). But let’s go one step further and install a certificate manager into Kubernetes that will allow us to create certificates from a CA for each ingress.

We’re going to use the originally named cert-manager for our certificate manager. To install all the artifacts needed for the basic manager system, use the following:

That will create the namespace call cert-manager . You can see all the artifacts with kubectl get all -n cert-manager .

rkamradt@beast:~/scratch$ kubectl get all -n cert-manager

NAME READY STATUS RESTARTS AGE

pod/cert-manager-74d6c4d49b-md4lf 1/1 Running 1 22h

pod/cert-manager-cainjector-77bc84779-7vhgj 1/1 Running 1 22h

pod/cert-manager-webhook-5b5485577f-6csgf 1/1 Running 1 22h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/cert-manager ClusterIP 10.100.138.27 <none> 9402/TCP 22h

service/cert-manager-webhook ClusterIP 10.97.218.179 <none> 443/TCP 22h NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/cert-manager 1/1 1 1 22h

deployment.apps/cert-manager-cainjector 1/1 1 1 22h

deployment.apps/cert-manager-webhook 1/1 1 1 22h NAME DESIRED CURRENT READY AGE

replicaset.apps/cert-manager-74d6c4d49b 1 1 1 22h

replicaset.apps/cert-manager-cainjector-77bc84779 1 1 1 22h

replicaset.apps/cert-manager-webhook-5b5485577f 1 1 1 22h

Now we should have a certificate from a CA. We’re going to create our own CA and produce a root certificate that the certificate manager can use to sign individual certificates. They still won’t be trusted by your browser, but you only have to install the one root certificate on the browser to get any certificate created from it trusted and it’s only one step away from using a real CA.

This will create two files, ca.key and ca.crt. The key file is your private key which will need to be secret. The crt file is your public root certificate, which you can install in your browser. The key file is needed to create certificates on behalf of your new CA, hence the need to keep it absolutely secret. You have to let anyone that you want to trust you have the crt file. The crt file is base-64 encoded, so it’s just text, you can print it out if you want.

The last two lines of the above snippet will allow it to be used as a secret. The third line removes the passphrase, and the fourth line creates the secrets in Kubernetes. This secret will be used by cert-manager to create an Issuer . Create a file called issuer.yaml and insert the following:

apiVersion: cert-manager.io/v1alpha2

kind: Issuer

metadata:

name: ca-issuer

spec:

ca:

secretName: ca-key-pair

This creates an Issuer named ca-issuer and associates the CA certificate secret we just created. Now the Issuer can be called upon to issue certificates as needed. Let’s create an ingress that will use a virtual hostname and issue a certificate for it. Create a file called vhost-ingress.yaml and add the following:

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

name: test-ingress

annotations:

cert-manager.io/issuer: ca-issuer

spec:

rules:

- host: helloworld.local

http:

paths:

- backend:

serviceName: hellok8s-service

servicePort: 8080

path: /

tls:

- hosts:

- helloworld.local

secretName: myingress-cert

In this file, the cert-manager.io/issuer annotation tells it which issuer to use to issue certificates, in this case, the one we just created. Under spec.rules we set the host to helloworld.local . Finally, we add the tls section that says to create a certificate for helloworld.local and store it in the secret myingress-cert for Nginx ingress to use.

Now we can add them to the system in the normal fashion:

kubectl apply -f issuer.yaml

kubectl apply -f vhost-ingress.yaml

Run kubectl get secrets myingress-cert to see the generated certificate.

Back on your development machine, you can add helloworld.local as an alias for the main host in the /etc/hosts file. Then browse to https://helloworld.local. You should get the same warning because our system doesn’t trust the new root certificate yet. But click through the warning, and have another look at the certificate. It should be a new certificate, not the dummy Nginx certificate.

If you’re running Chrome on macOS, it turns out you can add it to the trusted certificates on the Mac Keychain. Open up the Keychain tool and select the system tab. Then, under the File menu, choose Import Item. Select the certificate (ca.crt) you copied over from your main host. Once imported, it should appear in your list of certificates, but it should have a red X indicating it’s not yet trusted. Double click on it, and set trusted to all. Now if you refresh https://helloworld.local, it should come up as trusted. If you click the lock icon next to the URL, you’ll see the details of the certificate and it’s trusted root.

It’s set to expire in three months, but if I understand cert-manager, it will create a new one as needed. Pretty nifty! If you’re running another browser or another OS, I’m sure you’ll be able to find instructions on installing root certificates.

It’s probably overkill to create a CA and install a certificate manager just for one virtual host. But now I can add services to my system with virtual hosting and https just by adding another ingress and updating my /etc/hosts file. While we’re still very far from a production system, we’ve come a long way towards it and solved a lot of problems along the way. If you had a domain name, you could easily use a service like LetsEncrypt. The cert-manager component has a configuration for LetsEncrypt built-in.

This has been a long and complicated article, and I hope you stuck with it to the end. We solved a lot of problems and I know I’ve learned a lot. I hope you have too and thank you for sharing this journey with me. All of the scripts used can be found on my GitHub page.