Using SSL certificates from Let’s Encrypt in your Kubernetes Ingress via cert-manager
By: Flant Staff
Tags: Kubernetes, Cert Manager, Ssl, Lets Encrypt

In this article, I will walk you through the process of automating the issuance and renewal of certificates provided by Let’s Encrypt (and other services) for Kubernetes Ingress using the cert-manager add-on. But first, let me start with a brief introduction to the problem.

HTTPS in a nutshell

The HTTP protocol developed in the early 1990s has become an integral part of our daily life. Today, we cannot live a single day without it. However, it does not even provide a basic level of security when exchanging information between the user and the web server. That is when HTTPS (“S” means “secure” here) comes to the rescue. In HTTPS, the exchanged data is encrypted using SSL/TLS — that family of protocols has proven itself well in protecting privacy and data integrity and is being actively promoted by the industry.

For example, Google has been calling for “HTTPS everywhere” since 2014. When prioritizing search results, it takes into account whether sites use secure, encrypted connections. All this propaganda affects ordinary users as well: modern browsers warn their users about insecure connections and invalid SSL certificates.

A certificate for a personal website might cost tens of dollars. However, buying it is not always justified. Fortunately, since late 2015, there is a free alternative in the form of Let’s Encrypt (LE) certificates. This nonprofit authority was created by Mozilla enthusiasts to make internet-wide hassle-free encryption a reality.

The certificate authority issues domain-validated certificates (the most basic ones available on the market) valid for 90 days, and it is also possible to obtain a so-called wildcard certificate for several subdomains.

The algorithms described in the Automated Certification Management Environment (ACME) protocol (designed specifically for Let’s Encrypt) are used to obtain a certificate. With it, the agent can prove control of the domain either by provisioning an HTTP resource (the so-called “HTTP-01 challenge”) or DNS records (“DNS-01 challenge”) — more information about them you may find below.

Cert-manager

Cert-manager is a Kubernetes-native certificate management controller consisting of a set of CustomResourceDefinitions (hence the restriction on the minimum supported version of K8s, v1.12) for configuring CA (certificate authorities) and obtaining certificates. The installation of CRDs in a cluster is straightforward and boils down to applying a single YAML file:

kubectl create ns cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml

(You can also install it with Helm).

Before being able to request certificates, you must create CA resources: Issuer or ClusterIssuer . They are used for signing CSRs (certificate requests). The difference between these resources is that ClusterIssuer is non-namespaced and can be used in multiple namespaces:

Issuer is used within a single namespace only;

is used within a single namespace only; ClusterIssuer is a global cluster object.

Using cert-manager

Method #1. Self-signed certificate

Let’s start with the simplest case — requesting a self-signed certificate. It is quite common. For example, you can use it in your K8s cluster for testing environments dynamically created for developers needs. It can be also useful in the case of the external load balancer that terminates SSL traffic.

The Issuer resource would look like this:

apiVersion: cert-manager.io/v1alpha2

kind: Issuer

metadata:

name: selfsigned

spec:

selfSigned: {}

To issue a certificate, you have to define the Certificate resource that determines the issuance (see the issuerRef section below) and the location of the private key (the secretName field). Then you need to invoke that key in the Ingress (note the tls section in the spec field):

---

apiVersion: cert-manager.io/v1alpha2

kind: Certificate

metadata:

name: selfsigned-crt

spec:

secretName: tls-secret

issuerRef:

kind: Issuer

name: selfsigned

commonName: "yet-another.website"

dnsNames:

- "yet-another.website"

---

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

name: app

spec:

tls:

- hosts:

- "yet-another.website"

secretName: tls-secret

rules:

- host: "yet-another.website"

http:

paths:

- path: /

backend:

serviceName: app

servicePort: 8080

The certificate will be issued in a few seconds after these resources are added to the cluster. You can see the confirmation in the output of the command:

kubectl -n app describe certificate selfsigned-crt

...

Normal GeneratedKey 5s cert-manager Generated a new private key

Normal Requested 5s cert-manager Created new CertificateRequest resource "selfsigned-crt-4198958557"

Normal Issued 5s cert-manager Certificate issued successfully

Looking at the secret resource itself, you will see:

the tls.key private key,

private key, the ca.crt root certificate,

root certificate, and the self-signed tls.crt certificate.

You can browse the contents of these files using the openssl utility:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer

notBefore=Feb 10 21:01:59 2020 GMT

notAfter=May 10 21:01:59 2020 GMT

issuer=O = cert-manager, CN = yet-another.website

It is worth noting that clients consuming certificates issued in such a way (i.e., using the self-signed issuer) will not trust them. The reason is simple: this issuer type does not have a CA (see the note on the project website).

To avoid this, specify the path to the secret file containing ca.crt in the Certificate resource. For example, you can use the corporate CA. This way, you will be able to sign certificates issued for your Ingress with a key that is already in use by other server services/information systems.

Method #2. Let’s Encrypt certificate with the HTTP validation

As I mentioned before, there are two types of “challenges” available to prove the control of the domain, HTTP-01 and DNS-01.

The first approach (HTTP-01) involves deploying a tiny web server as a separate deployment. It will be serving some information at the http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> URL per request of the certification server. Therefore, this method implies the accessibility of Ingress from the outer world via port 80 and the publicity of the domain’s DNS record.

The second challenge (DNS-01) makes sense if there is an API you can use to change the DNS records of your domain. The Issuer uses these tokens to create TXT records for your domain. Then, ACME server gets these records during confirmation. Let’s Encrypt easily integrates with various DNS providers, including CloudFlare, AWS Route53, Google CloudDNS, and others (as well as with the LE’s own DNS implementation, acme-dns).

Note: Let’s Encrypt imposes fairly strict limits on requests to ACME servers. To avoid unnecessary load on LE’s production environment, we recommend using the letsencrypt-staging certificate for testing (the difference is in the ACME server only).

So, let’s describe the resources:

apiVersion: cert-manager.io/v1alpha2

kind: Issuer

metadata:

name: letsencrypt

spec:

acme:

server: https://acme-staging-v02.api.letsencrypt.org/directory

privateKeySecretRef:

name: letsencrypt

solvers:

- http01:

ingress:

class: nginx

---

apiVersion: cert-manager.io/v1alpha2

kind: Certificate

metadata:

name: le-crt

spec:

secretName: tls-secret

issuerRef:

kind: Issuer

name: letsencrypt

commonName: yet-another.website

dnsNames:

- yet-another.website

Note that we use the staging server in acme ’s server field for our Issuer . You can replace it with the production one later.

Let’s apply this configuration and trace the entire process of obtaining the certificate:

1. The creation of a Certificate leads to the emergence of a new CertificateRequest resource:

kubectl -n app describe certificate le-crt

...

Created new CertificateRequest resource "le-crt-1127528680"

2. In its description there is a notification about an Order creation:

kubectl -n app describe certificaterequests le-crt-1127528680

…

Created Order resource app/le-crt-1127528680-1805948596

3. The Order contains the description of parameters of the validation and its current status. The validation is performed by the Challenge resource:

kubectl -n app describe order le-crt-1127528680-1805948596

…

Created Challenge resource "le-crt-1127528680-1805948596-1231544594" for domain "yet-another.website"

4. And finally, there is information about the status of the validation itself in resource details:

kubectl -n app describe challenges le-crt-1127528680-1805948596-1231544594

...

Reason: Successfully authorized domain

...

Normal Started 2m45s cert-manager Challenge scheduled for processing

Normal Presented 2m45s cert-manager Presented challenge using http-01 challenge mechanism

Normal DomainVerified 2m22s cert-manager Domain "yet-another.website" verified with "http-01" validation

The certificate will be issued in less than a minute if all the prerequisites are met: the domain is accessible from the outer world, the rate limits on the LE side are respected, and so on. If the issuance was successful, you should see the message “Certificate issued successfully” in the output of the describe certificate le-tls command.

Now you can safely change the ACME server address to the production one ( https://acme-v02.api.letsencrypt.org/directory ) and re-issue valid certificates signed by Let's Encrypt Authority X3 instead of Fake LE Intermediate X1 .

But first, you have to delete the Certificate resource. Otherwise, the issuance procedure will not start because the certificate already exists, and it is valid. Deleting a secret would immediately result in invalidation of the certificate with the following message in the output of the describe certificate command:

Normal PrivateKeyLost 44s cert-manager Lost private key for CertificateRequest "le-crt-613810377", deleting old resource

Now it is time to apply the production manifest for the Issuer with the Certificate described above (it has not changed):



kind: Issuer

metadata:

name: letsencrypt

spec:

acme:

server:

privateKeySecretRef:

name: letsencrypt

solvers:

- http01:

ingress:

class: nginx apiVersion: cert-manager.io/v1alpha2kind: Issuermetadata:name: letsencryptspec:acme:server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef:name: letsencryptsolvers:- http01:ingress:class: nginx

After receiving the Certificate issued successfully confirmation, let us check it out:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer

notBefore=Feb 10 21:11:48 2020 GMT

notAfter=May 10 21:11:48 2020 GMT

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

Method #3. Validating Wildcard LE over DNS

To make one step further we will issue a certificate for all subdomains of the site using another method of validation — the DNS one. We will use CloudFlare as our DNS provider to change the domain records we need.

First, let’s create a token to use the CloudFlare API:

1. Profile → API Tokens → Create Token.

2. Set access rights as follows:

Permissions:

→ Zone — DNS — Edit

→ Zone — Zone — Read

Zone Resources:

→ Include — All Zones

3. Copy the token generated (for example, y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M ).

Create a Secret resource containing the token and describe it in your Issuer :

apiVersion: v1

kind: Secret

metadata:

name: cloudflare-api-token

type: Opaque

stringData:

api-token: y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M

---

apiVersion: cert-manager.io/v1alpha2

kind: Issuer

metadata:

name: letsencrypt

spec:

acme:

server: https://acme-v02.api.letsencrypt.org/directory

privateKeySecretRef:

name: letsencrypt

solvers:

- dns01:

cloudflare:

email: my-cloudflare-acc@example.com

apiTokenSecretRef:

name: cloudflare-api-token

key: api-token ---

apiVersion: cert-manager.io/v1alpha2

kind: Certificate

metadata:

name: le-crt

spec:

secretName: tls-secret

issuerRef:

kind: Issuer

name: letsencrypt

commonName: yet-another.website

dnsNames:

- "yet-another.website"

- "*.yet-another.website"

(Do not forget to use a staging environment for testing!)

It is time to go through the domain ownership confirmation procedure:

kubectl -n app describe challenges.acme.cert-manager.io le-crt-613810377-1285319347-3806582233

...

Status:

Presented: true

Processing: true

Reason: Waiting for dns-01 challenge propagation: DNS record for "yet-another.website" not yet propagated

State: pending

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Started 54s cert-manager Challenge scheduled for processing

Normal Presented 53s cert-manager Presented challenge using dns-01 challenge mechanism

The TXT record will appear at your DNS dashboard:

… and after a while, the status will change to:

Domain "yet-another.website" verified with "dns-01" validation

Let’s make sure that the certificate is valid for all subdomains:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -text |grep DNS:

DNS:*.yet-another.website, DNS:yet-another.website

The validation over DNS is usually slow since most DNS providers have a so-called “propagation time” — a period showing how long it takes for an updated DNS record to become available on all DNS servers of the provider.

The ACME standard also supports a combination of both types of validation. You can use it to speed up obtaining a certificate for the main domain. In this case, the description of the Issuer will have the following form:

apiVersion: cert-manager.io/v1alpha2

kind: Issuer

metadata:

name: letsencrypt

spec:

acme:

server: https://acme-v02.api.letsencrypt.org/directory

privateKeySecretRef:

name: letsencrypt

solvers:

- selector:

dnsNames:

- "*.yet-another.website"

dns01:

cloudflare:

email: my-cloudflare-acc@example.com

apiTokenSecretRef:

name: cloudflare-api-token

key: api-token

- selector:

dnsNames:

- "yet-another.website"

http01:

ingress:

class: nginx

If you apply this configuration, two Challenge resources will be created:

kubectl -n app describe orders le-crt-613810377-1285319347

…

Normal Created 3m29s cert-manager Created Challenge resource "le-crt-613810377-1285319347-3996324737" for domain "yet-another.website"

Normal Created 3m29s cert-manager Created Challenge resource "le-crt-613810377-1285319347-1443470517" for domain "yet-another.website"

Method #4. Using special Ingress annotations

Besides creating certificates directly, you can use cert-manager’s ingress-shim component. It relieves you of the need to create Certificate resources explicitly. The idea is to obtain a certificate automatically using the Issuer specified in the special annotations of Ingress. Here is an example of the respective Ingress resource:

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

annotations:

cert-manager.io/cluster-issuer: letsencrypt

spec:

tls:

- hosts:

- "yet-another.website"

secretName: tls-secret

rules:

- host: "yet-another.website"

http:

paths:

- path: /

backend:

serviceName: app

servicePort: 8080

In this case, the availability of Issuer is enough to get the work done, meaning that we have to create a lesser number of entities.

Also, there is an obsolete kube-lego annotation — kubernetes.io/tls-acme: "true" . The peculiarity about it is that when deploying cert-manager, the default Issuer must be specified via Helm arguments (or by appending arguments of cert-manager’s deployment container).

We at Flant do not use these approaches (mentioned in Method #4) and cannot recommend them due to their opacity (and various associated problems). However, it’s good to mention them in the article to provide a fuller picture.

Key takeaways

We have learned how to obtain auto-renewable, self-signed, and free SSL certificates from Let’s Encrypt for website domains that are managed by Ingresses in the Kubernetes clusters.

The article provides example solutions for the most common problems we face. However, cert-manager features are not limited to those described above. On the project’s website, you can find examples of using it with other services. For example, you can use Vault as a certificate authority or set up external Issuers.

This article has been written by our engineer Oleg Saprykin. Follow our blog to get new excellent content from Flant!