GitLab/Gitlab Runner

GitLab Runner is an application that works with GitLab to run CI/CD jobs in a pipeline.https://docs.gitlab.com/runner/ For more information see the official GitLab Runner documentation. For information on how GitLab runners are used in production deployments, see Deploying services to production.
=== GitLab Runner types ===
Our GitLab instance offers different types of CI Runners:
;Shared GitLab Runners
:General purpose CI workers. These Runners execute jobs for a wide range of projects inside the /repos group in GitLab. If access to this kind of Runner is needed, consider moving your git repository to the /repos group and make yourself familiar with the details under Shared GitLab Runners. Shared Runners use the tag wmcs but also accept untagged jobs.
;Trusted GitLab Runners
:These offer a platform for CI jobs with additional security needs (like building production artifacts). These Runners run in WMF production infrastructure and access to these Runners is gated and restricted. Access has to be requested on per-project basis, so please take a look on Trusted GitLab Runners on how to get access. Trusted Runners use the tag trusted and do not accept untagged jobs.
;Cloud Runners
:Instance-wide CI workers. They are running on a managed Kubernetes in Digital Ocean. They are available instance wide for every project. Cloud Runners use the tag cloud (and kubernetes) but accept untagged jobs as well.
;Memory-optimized Cloud Runners
: These behave just like regular Cloud runners but feature more RAM than their counterpart. These runners are useful for more intensive building/packaging jobs. To utilize these runners, add the memory-optimized tag.
=== Permission matrix ===
The following table explains which features and permissions are available for which type of runner.
Runner Type 	Trusted Runners 	Dockerfile Runners 	Shared Runner 	Cloud Runners
access level 	access has to be requested per project in projects.json 	access has to be requested per project in projects.json and is restricted to a small set of infrastructure/foundational container projects	all projects in /repos have access 	all projects have access
allowed images 	only images from WMF registry docker-registry.wikimedia.org/**/* 	only images from WMF registry docker-registry.wikimedia.org/**/* 	images from WMF registry, popular upstream Dockerhub images (like debian, golang, node, python), GitLab upstream images, wmflabs images. See profile::gitlab::runner::allowed_images in operations/puppet/hieradata/cloud.yaml	no restriction
allowed Container build frontends	blubber frontend 	blubber and Dockerfile frontend 	blubber frontend 	blubber and Dockerfile frontend
publish images to wmf registry 	Yes (using credentials from jwt-authorizer) 	Yes (using credentials from jwt-authorizer) 	No 	No
allowed job sources 	Jobs from protected tags and branches only 	Jobs from protected tags and branches only 	Jobs from all branches or tags 	Jobs from all branches or tags
allowed WMF services 	GitLab, apt host, doc host, docker registry, Kubernetes staging (see profile::gitlab::runner::allowed_services)	GitLab, apt host, doc host, docker registry, Kubernetes staging (see profile::gitlab::runner::allowed_services) 	GitLab, Zuul 	none
allowed external services 	Https services using the webproxy, Dockerhub is blocked 	Https services using the webproxy 	All public https services 	All public https services
=== Adding new runners (end-users) ===
For special use-cases new runners can be added to any project by end-users. The token needed to register the runner can be obtained under project Settings > CI/CD > Runners (expand). Use the button "New project runner" to add a runner. The registration token will be shown once, so store the token in a secure place. Usage of the registration token varies depending on the type of runner (Docker, VM, Kubernetes). See register a runner upstream documentation. Please note this runners are self-managed, so you have to make sure to install updates and stay compatible with the GitLab server version.
=== Adding new runners (administrators) ===
The process of adding Shared, Trusted or instance-wide runners is different from adding self-managed project runners. The tokens and registration are created programmatically to ensure all runners use the correct configuration (protection status, tags, description and untagged jobs). The tool used to create the tokens is https://gitlab.wikimedia.org/repos/releng/gitlab-settings. The actual registration happens in Puppet or Terraform, depending on the Runner.
To generate new tokens, use the runners generate-tokens command. The script will walk you over every runner configured in runner-config.json. Please note the output of the script contains private tokens.
Example run:
./runners generate-tokens
Please add the following tokens to private puppet.
The token should be added as profile::gitlab::runner::token in either private puppet or WMCS puppet master.
The token for cloud runners should be stored as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd.
The tokens are secrets, do not store them anywhere else!
Do you want to create a new token for shared_runners [y/N]: n
Skipping shared_runners
Do you want to create a new token for trusted_runners [y/N]: y
trusted_runners token: <removed>
As mentioned in the script output, the tokens are then used in Puppet or GitLab CI deployments. So for Shared and Trusted Runners configured by puppet, the token has to added as profile::gitlab::runner::token to private puppet. Runners in Digital Ocean use Terraform for deployments, so the token is added as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd. 
New Runners should pick up a updated token with the next puppet run or Terraform deployment. Existing Runners have to be un-registered and re-registered manually by setting profile::gitlab::runner::ensure to absent, run puppet and setting it to present again. Also the script does not rotate old tokens, it just creates to tokens. So manual deletion of Runners in https://gitlab.wikimedia.org/admin/runners may be required.
=== Evaluation and Design ===
Evaluation sub-pages on the right menu offer more insights into the design and security considerations.