Kubernetes GitLab Runners on Amazon EKS
By: Marcin Cuber
Tags: Gitops, Gitlab, Containers, Kubernetes, AWS

Find out how to configure GitLab Runners efficiently and trouble free on Amazon EKS following GitOps strategy.

Overview

Custom Gitlab Runners in AWS is probably the best feature of Gitlab, especially when you have a managed cloud GitLab server. I don’t believe such functionality is offered by any other known provider such as CircleCI, TravisCI, TeamCity or disgusting Jenkins. GitLab Runner is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs.

I am using Kubernetes platform to spin up all my Gitlab Runners which is proving to be a very efficient and fast way of spinning them up. Previously, Gitlab Runners were running on AWS EC2, this was proving to be very challenging in terms of configuration and was taking long time to get new runners when needed.

In this article, I will provide details and templates defining how my Gitlab Runners are implemented. Also, I will outline the setup of Amazon EKS cluster and GitOps strategy using Flux that helps hugely with upgrades of the runners.

Note that all configuration details for Gitlab Runners and EKS can be found in the following:

Cluster configuration details

I am currently running latest available version of Amazon EKS 1.16. Full configuration of the cluster can be found in my Github EKS repository.

My cluster is configured with following addons to make Gitlab runners work nicely:

External Secrets — this is used to pull gitlab token from AWS SSM parameter store and populate it as a secret

Reloader — this is extremely useful as it watches changes in ConfigMap and Secret and performs rolling upgrades on Pods

All my Kubernetes addons are deployed in GitOps way using Flux. Flux is a tool that automatically ensures that the state of a cluster matches the config in git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means you don’t need a separate CD tool. It monitors all relevant image repositories, detects new images, triggers deployments and updates the desired running configuration based on that (and a configurable policy). If you want to learn more about it, head to https://github.com/fluxcd/flux.

Flux is a fantastic tool and in my opinion it is much better than Helm, which I am avoiding at all costs. Helm is simply another wrapper around wrapper which to me doesn’t make any sense. In fact, I am hugely surprised that people waste their time trying to fight Helm issues instead of configuring a clean fully working yaml templates for their applications. If you follow GitOps approach you can also get nice release versions of your code in Gitlab or Gihub. In case you want to find out more about Helm madness, please head to https://helm.sh/

Current limitation of Gitlab runners in Kubernetes

Gitlab runners don’t work very well with HPA (Horizonal Pod Autoscaler). Also, metrics server and/or Prometheus custom metrics doesn’t provide good enough metrics to make it work. I confirmed that information with Gitlab support. So, even though Gitlab docs and code suggests that is it possible (https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml#L382) in fact it doesn’t work, so simply don’t waste your time.

My GitOps configuration allows you to easily rollout version upgrades of the runners. Easily upgrade number of required replicas of runners per Gitlab group. And many other simply upgrades which require a minimal modification to yaml template and flux will handle the deployment.

Gitlab Runner Configs

We have reached the essence of this article. Here I am going to share the templates that I use to configure deployment, configmap, namespace, gitlab-token external-secret and RBAC.

Note that all templates are available in my github -> https://github.com/marcincuber/kubernetes-gitlab-runner

Deployment



kind: Deployment

metadata:

labels:

app: gitlab-runner-gitlab-runner

name: gitlab-runner-gitlab-runner

namespace: gitlab

annotations:

reloader.stakater.com/auto: "true"

spec:

progressDeadlineSeconds: 600

replicas: 10

selector:

matchLabels:

app: gitlab-runner-gitlab-runner

strategy:

rollingUpdate:

maxSurge: 50%

maxUnavailable: 50%

type: RollingUpdate

template:

metadata:

labels:

app: gitlab-runner-gitlab-runner

spec:

containers:

- command:

- /bin/bash

- /scripts/entrypoint

env:

- name: CI_SERVER_URL

value:

- name: CLONE_URL

value:

- name: RUNNER_REQUEST_CONCURRENCY

value: "1"

- name: RUNNER_EXECUTOR

value: kubernetes

- name: REGISTER_LOCKED

value: "true"

- name: RUNNER_TAG_LIST

value: "test"

- name: RUNNER_OUTPUT_LIMIT

value: "4096"

- name: KUBERNETES_IMAGE

value: ubuntu:20.04

- name: KUBERNETES_PRIVILEGED

value: "true"

- name: KUBERNETES_NAMESPACE

value: "gitlab"

- name: KUBERNETES_POLL_TIMEOUT

value: "180"

- name: KUBERNETES_POLL_INTERVAL

value: "3"

- name: KUBERNETES_CPU_LIMIT

value: "2"

- name: KUBERNETES_MEMORY_LIMIT

value: "3.5Gi"

- name: KUBERNETES_CPU_REQUEST

value: "1.5"

- name: KUBERNETES_MEMORY_REQUEST

value: "3.5Gi"

- name: KUBERNETES_SERVICE_ACCOUNT

- name: KUBERNETES_SERVICE_CPU_LIMIT

- name: KUBERNETES_SERVICE_MEMORY_LIMIT

- name: KUBERNETES_SERVICE_CPU_REQUEST

- name: KUBERNETES_SERVICE_MEMORY_REQUEST

- name: KUBERNETES_HELPER_CPU_LIMIT

- name: KUBERNETES_HELPER_MEMORY_LIMIT

- name: KUBERNETES_HELPER_CPU_REQUEST

value: "100m"

- name: KUBERNETES_HELPER_MEMORY_REQUEST

value: "128Mi"

- name: KUBERNETES_HELPER_IMAGE

- name: KUBERNETES_PULL_POLICY

value: "if-not-present"

- name: CACHE_TYPE

value: s3

- name: CACHE_PATH

value: gitlab-caches

- name: CACHE_SHARED

value: "true"

- name: CACHE_S3_SERVER_ADDRESS

value: s3.amazonaws.com

- name: CACHE_S3_BUCKET_NAME

value: kubernetes-gitlab-cache

- name: CACHE_S3_BUCKET_LOCATION

value: eu-west-1

- name: RUNNER_CACHE_DIR

value: /opt/gitlab-runner/cache/

- name: RUNNER_BUILDS_DIR

value: /opt/gitlab-runner/builds/

image: gitlab/gitlab-runner:alpine-v13.1.0

imagePullPolicy: IfNotPresent

lifecycle:

preStop:

exec:

command:

- /entrypoint

- unregister

- --all-runners

livenessProbe:

exec:

command:

- /bin/bash

- /scripts/check-live

name: gitlab-runner-gitlab-runner

ports:

- containerPort: 9252

name: metrics

protocol: TCP

readinessProbe:

exec:

command:

- /usr/bin/pgrep

- gitlab.*runner

volumeMounts:

- mountPath: /secrets

name: runner-secrets

- mountPath: /home/gitlab-runner/.gitlab-runner

name: etc-gitlab-runner

- mountPath: /scripts

name: scripts

resources:

requests:

cpu: "50m"

memory: "200Mi"

dnsPolicy: ClusterFirst

initContainers:

- command:

- sh

- /config/configure

env:

- name: CI_SERVER_URL

value:

- name: CLONE_URL

value:

- name: RUNNER_REQUEST_CONCURRENCY

value: "1"

- name: RUNNER_EXECUTOR

value: kubernetes

- name: REGISTER_LOCKED

value: "true"

- name: RUNNER_TAG_LIST

value: "test"

- name: RUNNER_OUTPUT_LIMIT

value: "4096"

- name: KUBERNETES_IMAGE

value: ubuntu:20.04

- name: KUBERNETES_PRIVILEGED

value: "true"

- name: KUBERNETES_NAMESPACE

value: "gitlab"

- name: KUBERNETES_POLL_TIMEOUT

value: "180"

- name: KUBERNETES_POLL_INTERVAL

value: "3"

- name: KUBERNETES_CPU_LIMIT

value: "2"

- name: KUBERNETES_MEMORY_LIMIT

value: "3.5Gi"

- name: KUBERNETES_CPU_REQUEST

value: "1.5"

- name: KUBERNETES_MEMORY_REQUEST

value: "3.5Gi"

- name: KUBERNETES_SERVICE_ACCOUNT

- name: KUBERNETES_SERVICE_CPU_LIMIT

- name: KUBERNETES_SERVICE_MEMORY_LIMIT

- name: KUBERNETES_SERVICE_CPU_REQUEST

- name: KUBERNETES_SERVICE_MEMORY_REQUEST

- name: KUBERNETES_HELPER_CPU_LIMIT

- name: KUBERNETES_HELPER_MEMORY_LIMIT

- name: KUBERNETES_HELPER_CPU_REQUEST

value: "100m"

- name: KUBERNETES_HELPER_MEMORY_REQUEST

value: "128Mi"

- name: KUBERNETES_HELPER_IMAGE

- name: KUBERNETES_PULL_POLICY

value: "if-not-present"

- name: CACHE_TYPE

value: s3

- name: CACHE_PATH

value: gitlab-caches

- name: CACHE_SHARED

value: "true"

- name: CACHE_S3_SERVER_ADDRESS

value: s3.amazonaws.com

- name: CACHE_S3_BUCKET_NAME

value: kubernetes-gitlab-cache

- name: CACHE_S3_BUCKET_LOCATION

value: eu-west-1

- name: RUNNER_CACHE_DIR

value: /opt/gitlab-runner/cache/

- name: RUNNER_BUILDS_DIR

value: /opt/gitlab-runner/builds/

image: gitlab/gitlab-runner:alpine-v13.1.0

imagePullPolicy: IfNotPresent

name: configure

volumeMounts:

- mountPath: /secrets

name: runner-secrets

- mountPath: /config

name: scripts

readOnly: true

- mountPath: /init-secrets

name: init-runner-secrets

readOnly: true

resources:

requests:

cpu: "50m"

memory: "200Mi"

restartPolicy: Always

schedulerName: default-scheduler

securityContext:

fsGroup: 65533

runAsUser: 100

serviceAccount: gitlab-runner-gitlab-runner

serviceAccountName: gitlab-runner-gitlab-runner

terminationGracePeriodSeconds: 300

volumes:

- emptyDir:

medium: Memory

name: runner-secrets

- emptyDir:

medium: Memory

name: etc-gitlab-runner

- name: init-runner-secrets

projected:

defaultMode: 420

sources:

- secret:

items:

- key: runner-registration-token

path: runner-registration-token

name: gitlab-runner-token

- configMap:

defaultMode: 420

name: gitlab-runner-gitlab-runner

name: scripts apiVersion: apps/v1kind: Deploymentmetadata:labels:app: gitlab-runner-gitlab-runnername: gitlab-runner-gitlab-runnernamespace: gitlabspec:progressDeadlineSeconds: 600selector:matchLabels:app: gitlab-runner-gitlab-runnertemplate:metadata:labels:app: gitlab-runner-gitlab-runnerspec:containers:- command:- /bin/bash- /scripts/entrypointenv:- name: CI_SERVER_URLvalue: https://gitlab.com/ - name: CLONE_URLvalue: https://gitlab.com/ - name: RUNNER_REQUEST_CONCURRENCYvalue: "1"- name: RUNNER_EXECUTORvalue: kubernetes- name: REGISTER_LOCKEDvalue: "true"- name: RUNNER_TAG_LISTvalue: "test"- name: RUNNER_OUTPUT_LIMITvalue: "4096"- name: KUBERNETES_IMAGEvalue: ubuntu:20.04- name: KUBERNETES_NAMESPACEvalue: "gitlab"- name: KUBERNETES_POLL_TIMEOUTvalue: "180"- name: KUBERNETES_POLL_INTERVALvalue: "3"- name: KUBERNETES_CPU_LIMITvalue: "2"- name: KUBERNETES_MEMORY_LIMITvalue: "3.5Gi"- name: KUBERNETES_CPU_REQUESTvalue: "1.5"- name: KUBERNETES_MEMORY_REQUESTvalue: "3.5Gi"- name: KUBERNETES_SERVICE_ACCOUNT- name: KUBERNETES_SERVICE_CPU_LIMIT- name: KUBERNETES_SERVICE_MEMORY_LIMIT- name: KUBERNETES_SERVICE_CPU_REQUEST- name: KUBERNETES_SERVICE_MEMORY_REQUEST- name: KUBERNETES_HELPER_CPU_LIMIT- name: KUBERNETES_HELPER_MEMORY_LIMIT- name: KUBERNETES_HELPER_CPU_REQUESTvalue: "100m"- name: KUBERNETES_HELPER_MEMORY_REQUESTvalue: "128Mi"- name: KUBERNETES_HELPER_IMAGE- name: KUBERNETES_PULL_POLICYvalue: "if-not-present"- name: CACHE_TYPEvalue: s3- name: CACHE_SHAREDvalue: "true"- name: CACHE_S3_SERVER_ADDRESSvalue: s3.amazonaws.com- name: RUNNER_CACHE_DIRvalue: /opt/gitlab-runner/cache/- name: RUNNER_BUILDS_DIRvalue: /opt/gitlab-runner/builds/imagePullPolicy: IfNotPresentlifecycle:preStop:exec:command:- /entrypoint- unregister- --all-runnerslivenessProbe:exec:command:- /bin/bash- /scripts/check-livename: gitlab-runner-gitlab-runnerports:- containerPort: 9252name: metricsprotocol: TCPreadinessProbe:exec:command:- /usr/bin/pgrep- gitlab.*runnervolumeMounts:- mountPath: /secretsname: runner-secrets- mountPath: /home/gitlab-runner/.gitlab-runnername: etc-gitlab-runner- mountPath: /scriptsname: scriptsresources:requests:cpu: "50m"memory: "200Mi"dnsPolicy: ClusterFirstinitContainers:- command:- sh- /config/configureenv:- name: CI_SERVER_URLvalue: https://gitlab.com/ - name: CLONE_URLvalue: https://gitlab.com/ - name: RUNNER_REQUEST_CONCURRENCYvalue: "1"- name: RUNNER_EXECUTORvalue: kubernetes- name: REGISTER_LOCKEDvalue: "true"- name: RUNNER_TAG_LISTvalue: "test"- name: RUNNER_OUTPUT_LIMITvalue: "4096"- name: KUBERNETES_IMAGEvalue: ubuntu:20.04- name: KUBERNETES_NAMESPACEvalue: "gitlab"- name: KUBERNETES_POLL_TIMEOUTvalue: "180"- name: KUBERNETES_POLL_INTERVALvalue: "3"- name: KUBERNETES_CPU_LIMITvalue: "2"- name: KUBERNETES_MEMORY_LIMITvalue: "3.5Gi"- name: KUBERNETES_CPU_REQUESTvalue: "1.5"- name: KUBERNETES_MEMORY_REQUESTvalue: "3.5Gi"- name: KUBERNETES_SERVICE_ACCOUNT- name: KUBERNETES_SERVICE_CPU_LIMIT- name: KUBERNETES_SERVICE_MEMORY_LIMIT- name: KUBERNETES_SERVICE_CPU_REQUEST- name: KUBERNETES_SERVICE_MEMORY_REQUEST- name: KUBERNETES_HELPER_CPU_LIMIT- name: KUBERNETES_HELPER_MEMORY_LIMIT- name: KUBERNETES_HELPER_CPU_REQUESTvalue: "100m"- name: KUBERNETES_HELPER_MEMORY_REQUESTvalue: "128Mi"- name: KUBERNETES_HELPER_IMAGE- name: KUBERNETES_PULL_POLICYvalue: "if-not-present"- name: CACHE_TYPEvalue: s3- name: CACHE_SHAREDvalue: "true"- name: CACHE_S3_SERVER_ADDRESSvalue: s3.amazonaws.com- name: RUNNER_CACHE_DIRvalue: /opt/gitlab-runner/cache/- name: RUNNER_BUILDS_DIRvalue: /opt/gitlab-runner/builds/imagePullPolicy: IfNotPresentname: configurevolumeMounts:- mountPath: /secretsname: runner-secrets- mountPath: /configname: scriptsreadOnly: true- mountPath: /init-secretsname: init-runner-secretsreadOnly: trueresources:requests:cpu: "50m"memory: "200Mi"restartPolicy: AlwaysschedulerName: default-schedulersecurityContext:fsGroup: 65533runAsUser: 100serviceAccount: gitlab-runner-gitlab-runnerserviceAccountName: gitlab-runner-gitlab-runnerterminationGracePeriodSeconds: 300volumes:- emptyDir:medium: Memoryname: runner-secrets- emptyDir:medium: Memoryname: etc-gitlab-runner- name: init-runner-secretsprojected:defaultMode: 420sources:- configMap:defaultMode: 420name: gitlab-runner-gitlab-runnername: scripts

I have highlighted some important bits that you may want to reconfigure yourself. That is a S3 bucket which will have a different name and potentially a different region.

There are various environment variables that you should review as well. Setting such as Privileged container is very important. I strongly recommend that you have a dedicated cluster to run all your Gitlab runners. In most case they will need to be privileged. Additionally, all Kubernetes cluster should be treated like cattles.

Lastly, I want to point out that if you are following GitOps model of deployments, upgrades to runners are very simple and you only need to update image field with the new tag of the docker image. Usually upgrades take few seconds depending on rollingUpdate strategy.

ConfigMap

apiVersion: v1

kind: ConfigMap

metadata:

labels:

app: gitlab-runner-gitlab-runner

name: gitlab-runner-gitlab-runner

namespace: gitlab

data:

check-live: |

#!/bin/bash

if /usr/bin/pgrep -f .*register-the-runner; then

exit 0

elif /usr/bin/pgrep gitlab.*runner; then

exit 0

else

exit 1

fi

config.toml: |

concurrent = 1

check_interval = 10

log_level = "info"

configure: |

set -e

cp /init-secrets/* /secrets

entrypoint: |

#!/bin/bash

set -e

mkdir -p /home/gitlab-runner/.gitlab-runner/

cp /scripts/config.toml /home/gitlab-runner/.gitlab-runner/ # Register the runner

if [[ -f /secrets/accesskey && -f /secrets/secretkey ]]; then

export CACHE_S3_ACCESS_KEY=$(cat /secrets/accesskey)

export CACHE_S3_SECRET_KEY=$(cat /secrets/secretkey)

fi if [[ -f /secrets/gcs-applicaton-credentials-file ]]; then

export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-applicaton-credentials-file"

else

if [[ -f /secrets/gcs-access-id && -f /secrets/gcs-private-key ]]; then

export CACHE_GCS_ACCESS_ID=$(cat /secrets/gcs-access-id)

# echo -e used to make private key multiline (in google json auth key private key is oneline with

)

export CACHE_GCS_PRIVATE_KEY=$(echo -e $(cat /secrets/gcs-private-key))

fi

fi if [[ -f /secrets/runner-registration-token ]]; then

export REGISTRATION_TOKEN=$(cat /secrets/runner-registration-token)

fi if [[ -f /secrets/runner-token ]]; then

export CI_SERVER_TOKEN=$(cat /secrets/runner-token)

fi if ! sh /scripts/register-the-runner; then

exit 1

fi # Start the runner

exec /entrypoint run --user=gitlab-runner \

--working-directory=/home/gitlab-runner

register-the-runner: "#!/bin/bash

MAX_REGISTER_ATTEMPTS=30



for i in $(seq 1

\"${MAX_REGISTER_ATTEMPTS}\"); do

echo \"Registration attempt ${i} of ${MAX_REGISTER_ATTEMPTS}\"



\ /entrypoint register \\

--non-interactive



retval=$?



if [ ${retval}

= 0 ]; then

break

elif [ ${i} = ${MAX_REGISTER_ATTEMPTS} ]; then

exit

1

fi



sleep 5

done



exit 0

"

In the configmap template you don’t really need to customise anything. Only things you may want to look at are the namespace (if you decide to change it) and perhaps concurrency. A single Gitlab Runner can run more than one concurrent worker/executer. I set it to exactly to one, this allows me to have a single runner managing exactly one job at a time.

External Secret- Gitlab token

apiVersion: kubernetes-client.io/v1

kind: ExternalSecret

metadata:

name: gitlab-runner-token

namespace: gitlab

spec:

backendType: systemManager

data:

- key: /k8s-gitlab-runner/token/test

name: runner-registration-token

To make use of this, you will need to have external-secrets addon installed. I highly recommend having it, especially when you are running things in AWS. The highlighted line is the important part which points at the name of the AWS SSM parameter. This parameter stores the registration token for Gitlab. Such token can be found in More about SSM parameters can be read in docs.

Namespace and RBAC

apiVersion: v1

kind: Namespace

metadata:

name: gitlab

labels:

app: gitlab-runner-gitlab-runner

---

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

labels:

app: gitlab-runner-gitlab-runner

name: gitlab-runner-gitlab-runner

namespace: gitlab

rules:

- apiGroups:

- ""

resources:

- '*'

verbs:

- '*'

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

labels:

app: gitlab-runner-gitlab-runner

name: gitlab-runner-gitlab-runner

namespace: gitlab

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: gitlab-runner-gitlab-runner

subjects:

- kind: ServiceAccount

name: gitlab-runner-gitlab-runner

namespace: gitlab

---

apiVersion: v1

kind: ServiceAccount

metadata:

labels:

app: gitlab-runner-gitlab-runner

name: gitlab-runner-gitlab-runner

namespace: gitlab

automountServiceAccountToken: true

Just a basic configs for namespace and role. Role is has all permissions assigned, however it is bound to its namespace so doesn’t cause any issues.

Conclusion

I have present you with the latest configuration I am using for Gitlab runners. They are using up-to-date docker image which is version v13.1.0. I have been running this setup for at least 4 months and rolling updates is extremely easy. I had zero issues.

Something I have not mentioned was that my EKS cluster fully utilises spot instances which makes this setup very cheap. As this is all Gitlab related, you can always opt to use cloud managed Gitlab Runners which are provided by Gitlab. For security reasons and easy integration with AWS IAM, it was a must for me to have and run my own runners in AWS.

GitLab runners or in fact GitLab itself doesn’t provide one feature which is very important and this is managed Mac OSX runners. Hopefully, this feature is in works and at some point we will get fully equipped platform with everything we need :).

Thanks for reading, hope this story helped. Stay tuned.

If you have any questions please leave comments or private notes. Otherwise you can find me on https://www.linkedin.com/in/marcincuber/