Browser tab content: # Kubernetes on Hetzner

Alternative title: _How I cut infrastructure costs by 75% and increased compute by 2x._

I recently spent a few weeks moving a Kubernetes cluster from DigitalOcean to Hetzner. I made a few changes to our infrastructure to support Hetzner that I thought worth writing up. Hetzner also has done a lot of work to support Kubernetes users so I will touch on that too.

The motivation for moving to Hetzner is entirely financial. Hetzner's prices (despite [minor changes](https://news.ycombinator.com/item?id=42264427) to US-based prices) are world-beating.

Hetzner provide both virtual machines and bare metal machines. Hetzner's offering is limited versus AWS but they entirely make up for it in price.

By moving to Hetzner we decreased our infrastructure bill by over 75%. In the process of migrating I also made improvements to our infrastructure.

## Kubernetes on Hetzner

### Self-managing Kubernetes

I previously ran Kubernetes on DigitalOcean. Both clusters were self managed. While DigitalOcean, like other providers, offers a free managed control plane, there is typically a 100% markup on the nodes that belong to these managed clusters. Since we have known minimum capacity requirements it's economical to run the control plane ourselves.

It may cost $20/month to run the control plane, and additional time to upgrade the cluster bi-annually, but you would make back this $240/year in server costs if you have just six worker nodes. If you have 100 worker nodes, your virtual machine cost is effectively doubled by using a managed control plane.

Regardless, Hetzner does not offer a managed Kubernetes control plane, so I had to self-manage Kubernetes once again.

I manage the nodes with Terraform and configure them with Puppet. I will be upstreaming some fixes to the puppetlabs-kubernetes module, as it is a bit stale. If you wish to use this module to manage the latest version of Kubernetes [my changes are here](https://github.com/puppetlabs/puppetlabs-kubernetes/compare/main...bilbof:puppetlabs-kubernetes:main#diff-50ae7fb3724b662b58dbc1c71663cb16a484ab36aecd5a11317fb14465f847fa).

### Node roles

I used the node naming convention `<region><zone><environment><role><number>` that smarter minds than mine came up with. I try to keep it simple and minimize the number of roles in use in the cluster: `control-plane`, `worker`, and `database` are sufficiently flexible. It's trivial to add more roles, but this makes scheduling and efficient use of resources harder.

```
$ k get nodes
NAME        STATUS   ROLES           AGE   VERSION
euc1pmgr1   Ready    control-plane   54d   v1.31.1
euc1pwkr1   Ready    worker          37d   v1.31.1
```

### Building nodes

The cluster is bootstrapped with Terraform. Hetzner provide a managed firewall and networking, which are trivial to configure with their Terraform provider.

```
resource "hcloud_firewall" "controlplane" {
  name = "control-plane"
  rule { ... }
  apply_to {
    label_selector = "role=control-plane"
  }
}

resource "hcloud_placement_group" "controlplane" {
  name = "controlplane"
  type = "spread"
  labels = {
    node = "controlplane"
  }
}
```

Servers are entirely managed in Terraform. I wrote a module per role that handles turning up a server with templated userdata etc., such that it's a one line change to add a server to the cluster. I boot from a golden image snapshot that lives in Hetzner, that you can reference from Terraform with snapshot labels.

Hetzner allows setting the private IP address of a server (`ip`); a public IP is either requested or can be assigned from a pool of floating IPs. Floating IPs, in contrast to the single public IPV4s that can be attached to nodes, are not included in the price of servers.

```
resource "hcloud_server" "server" {
  name        = var.name
  image       = var.image
  server_type = var.server_type
  location    = var.location
  backups     = true
  ...
}
```

The userdata for Kubernetes nodes (Puppet agents) installs, updates and configures various essential software, including Puppet, and then initiates the first Puppet run.

When a node comes up, it runs Puppet which configures the Kubernetes worker and then enrols the server as a Kubernetes worker, based on its hostname. I am glossing over some details that ensures this process is secure - since you don't want to blindly enrol new nodes into your cluster.

Nodes are configured with Puppet. I use the region, zone, and role to configure nodes, below is the partial configuration for a `worker`.

```
classes:
- apt
- worker
- lvm
- kubernetes
- ...

kubernetes::worker: true
kubernetes::kubelet_extra_arguments:
- name: cloud-provider
  value: external
- name: "node-labels"
  value: "failure-domain.beta.kubernetes.io/region=%{::region},failure-domain.beta.kubernetes.io/zone=%{::region}%{::zone},topology.kubernetes.io/region=%{::region},topology.kubernetes.io/zone=%{::region}%{::zone}"
- ...
```

### Networking

While Hetzner provides break-glass access to nodes. I use [Tailscale VPN](https://tailscale.com/) for administrative connection to nodes. Tailscale makes it very easy to use Wireguard to configure nodes. While Tailscale is [open source](https://github.com/tailscale/tailscale/) they provide a managed admin dashboard for setting policies across nodes and they provide [relays](https://tailscale.com/kb/1411/device-connectivity) for when a direct connection is not possible. Since Tailscale provides NAT hole punching this means that I can close all ports on my worker nodes. I cannot recommend Tailscale highly enough.

> Tailscale takes it one step further: we recommend not opening up any firewall ports at all. Instead, both ends of a Tailscale link make outgoing connections to each other (“NAT hole punching”) that only each other can see. An attacker doing a port scan will see no openings at all, because the firewall is not open to them at all. [source](https://tailscale.com/compare/reviewer-guide)

I use Hetzner's managed firewall to block ports, in addition to Ubuntu's UFW for good measure. The internal network is flat, managed by Hetzner, and defined in Terraform

```
resource "hcloud_network_subnet" "euc" {
  network_id   = hcloud_network.euc.id
  type         = "cloud"
  network_zone = "euc"
  ip_range     = ...
  ...
```

I use Calico for the container networking interface. I hit a small snag when configuring Calico. Since Tailscale and Calico use the same [netfilter mark mask](https://unix.stackexchange.com/questions/467076/how-set-mark-option-works-on-netfilter-iptables), routing between pods was effectively broken: Tailscale rules could be applied to inter-pod packets. Luckily Calico enables overriding the mask it uses with an environment variable, and this can be fixed with a one-liner:

```
- name: FELIX_IPTABLESMARKMASK
  value: "0xff00ff00"
```

For ingress (serving requests) Hetzner provide a managed Load Balancer (akin to an AWS ALB) for ~$5/month. I use [Nginx Ingress controller](https://github.com/kubernetes/ingress-nginx) for routing requests to services in the cluster.

Hetzner have released a [cloud-controller-manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager), which I use to automatically provision Hetzner Load Balancers (recall the `--cloud-provider=external` flag I am setting above). Among other features the controller manager sets node `EXTERNAL-IP` to the primary IP of each node.

Note that Load Balancers are not global or regional – they belong to a single zone. This presents a single point of failure (SPOF). You can avoid this SPOF with various methods (DNS round robin, using CloudFlare load balancers, etc.) but this is out of scope.

For TLS, I provision certificates from CloudFlare such that requests from CloudFlare to our origin servers are encrypted. You can use a service mesh like Istio/Linkerd to encrypt requests between the nginx instances and your applications (mTLS) within your internal network.

### Storage

Hetzner's cloud virtual machines come with decent sized nVME SSDs, and they also provide [Cloud Volumes](https://docs.hetzner.com/cloud/volumes/overview/): SSD-based block storage. This is another area where Hetzner has gone out of its way (for a purportedly budget provider) to support Kubernetes users. Hetzner provides a [Container Storage Provider](https://github.com/hetznercloud/csi-driver) for provisioning and attaching persistent volumes to their nodes.

Volumes can optionally be encrypted with LUKS. Volumes cannot move zone but can be automatically attached to any node within a zone.

Hetzner also provide a Helm chart (sample of values below) for configuring the provider. Though I would recommend configuring things like storage classes separately to Helm Charts. In general I try to keep the blast radius of a Helm reconciliation as small as possible.

```
storageClasses:
  - name: snapshotted-volumes
    defaultStorageClass: true
    reclaimPolicy: Retain
  - name: snapshotted-luks
    reclaimPolicy: Retain
    parameters:
      csi.storage.k8s.io/node-publish-secret-name: encryption-secret
      csi.storage.k8s.io/node-publish-secret-namespace: kube-system
```

One missing feature with Hetzner volumes is a snapshotting feature. While one can back up data within volumes and PVCs yourself, _point-in-time_ snapshots (like those [provided by AWS EBS](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html)) are a very valuable feature and I am hoping Hetzner will add support for this soon. I discuss backups later in the article.

Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner's volumes we were seeing >50ms of IOWAIT with very low IOPS.

To address this problem on our database nodes I used the [Local Persistence Volume Static Provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner). This requires pre-provisioning local volumes (or even directories) on each machine. The provisioner creates a persistent volume for each local volume.

This makes workloads non-portable. A pod scheduled to a node with a PVC bound to a local static volume cannot be rescheduled on another node. Intervention is required: PVs assigned to deleted nodes must be deleted when a node dies. To enable rescheduling you can delete the PVs associated with dead nodes automatically or, as in our case since I am a little scared of this right now, manually.

I could have added Rook/Ceph to address this issue. That would have provided in-cluster block storage with potentially better performance than Hetzner had provided. However, all of our databases are self-replicating. For instance we use Redis with Sentinels, and Postgres with [synchronous replication](https://patroni.readthedocs.io/en/latest/replication_modes.html#synchronous-mode), therefore they are resilient to a single-node failure.

To provision the local volumes, I use LVM, on a LUKS encrypted partition. It might be cool for the static provisioner to handle encrypted volumes too, e.g. with fscrypt, but this isn't possible currently. LVM volumes are managed with Puppet, and then represented to Kubernetes as persistent volumes with an extra few lines of YAML for the local volume provisioner.

In general I use size based storage classes for local static volumes (e.g. large, medium, micro) since resizing local static volumes across the cluster is expensive. By definition static provisioning is an inefficient use of disk vs dynamic network attached storage, so I force workloads to move to new storage classes if they exceed the limits of their PVC. Since every database that is using local static volumes are replicated databases this is easy to do.

There are risks with using local SSDs in a Kubernetes cluster. Excessive disk IO can disrupt workloads, and this can occur just from many pods logging, let alone running a heavily used database. While I'm not dealing with a multi-tenant workloads, my main defence against this is using dedicated machine pools (roles) which limits the workloads that use these nodes.

I did consider not running the databases and/or caches on Kubernetes at all. It would be perfectly possible (and arguably simpler/easier) to run these workloads on the machines directly, as I have done in the past. In the end it is (for me, at least) simpler to run the databases on Kubernetes, rather than having a split-brain way of running software in the infrastructure. I would also add that vendor-provided Helm charts and operators are making it considerably easier to run databases on Kubernetes. We're running CrunchyData's [postgres operator](https://github.com/CrunchyData/postgres-operator) with no issues so far.

The Kubernetes control plane etcd cluster, which is very sensitive to disk IOPS and write latency, runs directly on a LVM volume backed by an SSD.

### Backups

Hetzner do not provide volume backups, but do provide full server backups. These backups are _not_ point-in-time. Therefore if you are running a database you'd probably benefit from using a database-specific backup tool.

I'll briefly cover how I am doing backups in my cluster.

I use VMware's [Velero](https://github.com/vmware-tanzu/velero/) for backing up namespaces, and in-particular, Persistent Volume Claims (PVCs). This is a really useful new feature that you can use to backup a single PVC on a schedule. Velero can backup to another PVC or to S3. You simply need to add an annotation to a pod for Velero backups to kick in on the specified volume.

For Postgres, I use pgBackRest, which works great.

In the spirit of 1-2-3 backups (two mediums, three copies) we backup to multiple S3 instances (two cloud providers) and to Hetzner cloud volumes.

### Additional features

-   We use SealedSecrets for secrets management. Hetzner does not provide a key management system.
    
-   I use Node Exporter, Prometheus, Grafana, and Loki for monitoring the cluster. These use Hetzner volumes.
    
-   For alerting, Alertmanager has a direct integration with Slack which works great.
    

## Thoughts on Kubernetes running on Hetzner

This work too me roughly one week, with four weeks of additional testing and tuning before I migrated our production environment over. I am happy we've moved to Hetzner – the pricing is reasonable and I have faith that it will stay low relative to other providers.

Hetzner has limitations that I have not seen written about much elsewhere, so worth addressing them.

### IP quality

I found that requests to Sentry and Incident (third parties) were failing from pods running on certain nodes. It turned out that both Sentry and Incident run on Google Cloud Platform (GCP) which had blocked the IPs that had been assigned to those nodes.

I saw this affect about 10% of the IPs assigned to our nodes, and it affects everything running on GCP, including things like GitLab CI runners. It turns out that this is a [frequent problem](https://www.reddit.com/r/hetzner/comments/1g445no/hertzner_ip_blocked_from_making_requests_to/) for Hetzner users:

> Unfortunately, some GeoIP databases incorrectly locate some of our IPs in Iran. We cannot influence these databases.
> 
> – Hetzner support

The root cause is unclear; Hetzner maintains that this is only fixable at Google's end, which is _technically correct,_ but doesn't explain why of all the top ~7 cloud providers Hetzner is most affected by this.

There are hints at why this might be affecting 10% of Hetzner's IPs. A GitLab engineer [suggested](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8121) there was something secret that he couldn't share, but implied it's due to some bad behaviour by Hetzner clients that Google is seeing – not that Hetzner IP addresses that used to belong to an Iranian user.

From the above GitLab thread:

> Here is the funny part: I am from Iran and use Hetzner servers for communicating with gitlab registry

It makes sense that folks in Iran would use Hetzner for a VPN to bypass sanctions, it's one of the cheapest VM providers, and this practice is pretty much [out in the open](https://github.com/net4people/bbs/issues?q=hetzner).

Hetzner also host scraping bots on their nodes, despite this being against their terms of service. Note that DataForSeoBot seems to be the top bot user agent (80% of traffic) from Hetzner's AS [for the past year](https://radar.cloudflare.com/traffic/as24940?dateRange=52w). The folks at Cloud66 have a nice [post on their findings](https://blog.cloud66.com/hetzner-connectivity-issues-due-to-sanction-busting-activities) too.

The best way to work around this I have found is to use Wireguard (with Tailscale in my case) to route traffic through non-blocked IPs. I initially tried to prevent being assigned blocked IPs by using node health checks as part of building a new node. But I found it's easier to just accept blocked IPs, since it can take days to be assigned a non-blocked IP. You can also request new IPs from Hetzner without needing to rebuild a node, but it can take a long time to get a fresh IP (hours/days). A long term solution is likely to involve Floating IPs.

We luckily aren't running anything that depends on IP quality (e.g. we send email with providers like SendGrid and PostMark rather than from our own IPs) so this isn't much of an issue, besides creating a small headache when I had to debug this. I'm hosting this blog, Kubernetes, and a couple of SaaS products.

### Customer service

The main complaints I have seen about Hetzner support appear to be from folks used to American service culture. Hetzner is a German company and provides German style customer service.

Hetzner is a low cost provider and you get what you pay for. You will typically get a response in 1-2 business days, depending on urgency, but not always the response that you are hoping for.

In my experience Hetzner provide very good technical support when you email them with a clear technical or business issue that can be only solved with their assistance, with a clear request for action from them. For instance if you need a server threshold raised they will do this for you same day provided you have a good reputation.

Their documentation is limited but decent. The Hetzner subreddit is monitored by their customer service team and they will DM you if you post a problem.

### Feature parity

Hetzner is rapidly releasing new features, but will deprecate non-profitable offerings quickly if they do not have sufficient take up.

Most recently (today, 1st December 2024), their [S3 offering](https://docs.hetzner.com/storage/object-storage/faq/beta) is generally available. It works fine, and is a stripped down version of what you would get from AWS or GCP, but no less than what you would get from another provider like DigitalOcean.

Actually I hit a small issue with their S3. Their S3 was supposed to be GA from November 1st (not December 1st). They sent me an email to this affect, and so I started using it. Unfortunately their S3 went down for about half the weekend in the middle of November, and this caused a couple of issues for me. Since I was using it for pgBackRest backups, my Postgres WALs grew a lot when their S3 was down, since the WAL archiving process couldn't continue until the WAL had been shipped off to S3. It turned out that they had shifted the date for moving S3 to GA from November to December and had not emailed some of their customers (including me) about this.

In general Hetzner has everything we need and I am happy to have made this move.

### Datacenter locations

Hetzner's central European datacenter locations are Falkenstein (Germany), Nuremberg (Germany), and Helsinki (Finland).

Falkenstein is a 1.5 hour drive from Nuremberg (about 100km / 60 miles). Helsinki is a 22 hour drive from Nuremberg, approximately 2,000km (1250 miles).

The ping times are noticeably higher. Nuremberg to Falkenstein is 3ms, while Nuremberg to Helsinki is 25ms. In a funny twist of fate, the only submarine cable between Germany and Finland (the 1173km long [C-Lion1](https://en.wikipedia.org/wiki/C-Lion1)) was cut on November 18th last month. International investigators suspect a Chinese ship's crew of deliberately dragging an anchor 111 miles along the sea bed to sever it. I didn't notice a difference in ping time actually, but Hetzner reported [backbone issues](https://www.reddit.com/r/hetzner/comments/1h2jl2t/clion1_is_repaired/) during the 10 day outage. The cable was repaired in the last few days.

The trouble with such big ping times is that it makes databases with synchronous replication considerably slower for writes when spread across these three availability zones. In the context of Kubernetes, the only impact will be API latency, etcd itself has a default [heartbeat interval](https://etcd.io/docs/v3.5/tuning/#network) of 100ms, which you won't need to tune for Hetzner. In the end I run databases and worker nodes across the two German datacenters with an active-active setup, and use the Finland datacenter for batch workloads. I'm glad Hetzner provide three European datacenters, but it would be nice if they had chosen a closer location for the third one.

## Summary

This migration was smooth, and enabled me to cut infrastructure costs by more than 75%. On top of this I doubled the compute resources available to the cluster.

Would I recommend Hetzner to everyone? No. But if you are looking to cut costs, Hetzner is pretty much guaranteed to deliver that regardless of where you are migrating from (even in light of their [changing prices for US VMs](https://news.ycombinator.com/item?id=42264427)).

Hetzner's open source controllers for managing load balancers and attaching persistent volumes are very convenient and made migrating a cluster to Hetzner straightforward. Even if there are limitations to their current offering, they are not blocking, and are more than made up for in price difference. Many thanks to Hetzner, I'm looking forward to being a customer for years to come.

As an expert summarizer, your task is to condense the following text into a clear and engaging summary in the same language as the source. Begin with a "TL;DR" that captures the core message in a single sentence, followed by a more detailed summary. Use appropriate emojis to highlight key points or emotions, making the summary visually engaging and easy to skim. Ensure the summary accurately reflects the original content's intent and tone.