Package: pepperize.cdk-autoscaling-gitlab-runner
AWS CDK GitLab Runner autoscaling on EC2This project provides a CDK construct toexecute jobs on auto-scaled EC2 instancesusing theDocker Machineexecutor.Running out ofRunner minutes,
usingDocker-in-Docker (dind),
speed up jobs withshared S3 Cache,
cross compiling/building environmentmultiarch,
cost effectiveautoscaling on EC2,
deploy directly from AWS accounts (withoutAWS Access Key),
running onSpot instances,
having a biggerbuild log sizeInstallTypeScriptnpminstall@pepperize/cdk-autoscaling-gitlab-runneroryarnadd@pepperize/cdk-autoscaling-gitlab-runnerPythonpipinstallpepperize.cdk-autoscaling-gitlab-runnerC# / .Netdotnet add package Pepperize.CDK.AutoscalingGitlabRunnerJava<dependency><groupId>com.pepperize</groupId><artifactId>cdk-autoscaling-gitlab-runner</artifactId><version>${cdkAutoscalingGitlabRunner.version}</version></dependency>QuickstartCreate a new AWS CDK Appin TypeScript withprojenmkdirgitlab-runnercdgitlab-runner
gitinit
npxprojennewawscdk-app-tsConfigure your project in.projenrc.jsAdddeps: ["@pepperize/cdk-autoscaling-gitlab-runner"],Update project files and install dependenciesnpxprojenRegister a new runnerRegistering runners:For ashared runner, go to the GitLab Admin Area and clickOverview > RunnersFor agroup runner, go toSettings > CI/CDand expand theRunnerssectionFor aproject runner, go toSettings > CI/CDand expand theRunnerssectionOptionally enable:Run untagged jobs[x]
Indicates whether this runner can pick jobs without tagsSee alsoRegistration token vs. Authentication tokenRetrieve a new runner authentication tokenRegister a new runnercurl--requestPOST"https://gitlab.com/api/v4/runners"--form"token=<your register token>"--form"description=gitlab-runner"--form"tag_list=pepperize,docker,production"Store runner authentication token in SSM ParameterStoreCreate a String parameterawsssmput-parameter--name"/gitlab-runner/token"--value"<your runner authentication token>"--type"String"Add to yourmain.tsimport{Vpc}from"@aws-cdk/aws-ec2";import{App,Stack}from"@aws-cdk/core";import{GitlabRunnerAutoscaling}from"@pepperize/cdk-autoscaling-gitlab-runner";constapp=newApp();conststack=newStack(app,"GitLabRunnerStack");constvpc=Vpc.fromLookup(app,"ExistingVpc",{vpcId:"<your vpc id>",});consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(stack,"GitlabRunner",{network:{vpc:vpc,},runners:[{token:token,configuration:{//optionallyconfigureyourrunner},},],});Create service linked role(If requesting spot instances, default: true)awsiamcreate-service-linked-role--aws-service-namespot.amazonaws.comConfigure the AWS CLIAWSumeConfiguring the AWS CLIAWS Single Sign-OnDeploy the GitLab RunnernpmrundeployExampleCustom cache bucketBy default, an AWS S3 Bucket is created as GitLab Runner's distributed cache.
It's encrypted and public access is blocked.
A custom S3 Bucket can be configured:constcache=newBucket(this,"Cache",{//Yourcustombucket});consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,},],cache:{bucket:cache},});Seeexample,GitlabRunnerAutoscalingCachePropsCustom EC2 key pairBy default, theamazonec2driver will create an EC2 key pair for each runner. To use custom ssh credentials provide a SecretsManager Secret with the private and public key file:Create a key pair, download the private key file and remember the created key pair nameGenerate the public key filessh-keygen -f <the downloaded private key file> -yCreate an AWS SecretsManager Secret from the key pairawssecretsmanagercreate-secret--name<thesecretname>--secret-string"{\"<the key pair name>\":\"<the private key>\",\"<the key pair name>.pub\":\"<the public key>\"}"Configure the job runnerconstkeyPair=Secret.fromSecretNameV2(stack,"Secret","CustomEC2KeyPair");newGitlabRunnerAutoscaling(this,"Runner",{runners:[{keyPair:keyPair,configuration:{machine:{machineOptions:{keypairName:"<the key pair name>",},},},},],cache:{bucket:cache},});Configure Docker MachineBy default, docker machine is configured to run privileged withCAP_SYS_ADMINto supportDocker-in-Docker using the OverlayFS driverand cross compiling/building withmultiarch.Seerunners.docker sectioninAdvanced configurationimport{GitlabRunnerAutoscaling}from"@pepperize/cdk-autoscaling-gitlab-runner";import{StringParameter}from"aws-cdk-lib/aws-ssm";consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,configuration:{environment:[],//ResettheOverlayFSdriverforeveryprojectdocker:{capAdd:[],//RemovetheCAP_SYS_ADMINprivileged:false,//Rununprivileged},machine:{idleCount:2,//NumberofidlemachineidleTime:3000,//WaitingtimeinidlestatemaxBuilds:1,//Maxbuildsbeforeinstanceisremoved},},},],});Seeexample,DockerConfigurationBigger instance typeBy default, t3.nano is used for the manager/coordinator and t3.micro instances will be spawned.
For bigger projects, for example withwebpack, this won't be enough memory.consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{manager:{instanceType:InstanceType.of(InstanceClass.T3,InstanceSize.SMALL),},runners:[{instanceType:InstanceType.of(InstanceClass.T3,InstanceSize.LARGE),token:token,configuration:{//optionallyconfigureyourrunner},},],});You may have to disable or configureSpot instancesSeeexample,GitlabRunnerAutoscalingManagerProps,GitlabRunnerAutoscalingJobRunnerPropsDifferent machine imageBy default, the latestAmazon 2 Linuxwill be used for the manager/coordinator.
The manager/coordinator instance's cloud init scripts requiresyumis installed, any RHEL flavor should work.
The requested runner instances by default using Ubuntu 20.04, any OS implemented by theDocker Machine provisionershould work.consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{manager:{machineImage:MachineImage.genericLinux(managerAmiMap),},runners:[{machineImage:MachineImage.genericLinux(runnerAmiMap),token:token,configuration:{//optionallyconfigureyourrunner},},],});Seeexample,GitlabRunnerAutoscalingManagerProps,GitlabRunnerAutoscalingJobRunnerPropsMultiple runners configurationEach runner defines one[[runners]]section in theconfiguration file.
UseSpecific runnerswhen you want to use runners for specific projects.constprivilegedRole=newRole(this,"PrivilegedRunnersRole",{//role1});constrestrictedRole=newRole(this,"RestrictedRunnersRole",{//role2});consttoken1=StringParameter.fromStringParameterAttributes(stack,"Token1",{parameterName:"/gitlab-runner/token1",});consttoken2=StringParameter.fromStringParameterAttributes(stack,"Token2",{parameterName:"/gitlab-runner/token2",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token1,configuration:{name:"privileged-runner",},role:privilegedRole,},{token:token2,configuration:{name:"restricted-runner",docker:{privileged:false,//Rununprivileged},},role:restrictedRole,},],});Seeexample,GitlabRunnerAutoscalingPropsSpot instancesBy default, EC2 Spot Instances are requested.consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,configuration:{machine:{machineOptions:{requestSpotInstance:false,spotPrice:0.5,},},},},],});Seeexample,EC2 spot price,MachineConfiguration,MachineOptions,Advanced configuration - runners.machine.autoscalingCross-Compile with MultiarchTo build binaries of different architectures can also useMultiarchconsttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,configuration:{docker:{privileged:true,},},},],});Configure your.gitlab-ci.ymlfilebuild:image:multiarch/debian-debootstrap:armhf-busterservices:-docker:stable-dind-name:multiarch/qemu-user-static:registercommand:-"--reset"script:-make buildSeemultiarch/qemu-user-staticRunning on AWS GravitonTo run your jobs onAWS Gravitonyou have to provide an AMI for arm64 architecture.consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,configuration:{instanceType:InstanceType.of(InstanceClass.M6G,InstanceSize.LARGE),machineImage:MachineImage.genericLinux({[this.region]:newLookupMachineImage({name:"ubuntu/images/hvm-ssd/ubuntu-focal-20.04-*-server-*",owners:["099720109477"],filters:{architecture:[InstanceArchitecture.ARM_64],"image-type":["machine"],state:["available"],"root-device-type":["ebs"],"virtualization-type":["hvm"],},}).getImage(this).imageId,}),},},],});SeeUbuntu Amazon EC2 AMI LocatorCustom runner's roleTo deploy from within your GitLab Runner Instances, you may pass a Role with the IAM Policies attached.constrole=newRole(this,"RunnersRole",{assumedBy:newServicePrincipal("ec2.amazonaws.com",{}),inlinePolicies:{},});consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{role:role,token:token,configuration:{//optionallyconfigureyourrunner},},],});Seeexample,GitlabRunnerAutoscalingPropsVpcIf no existing Vpc is passed, a cheapVPCwith a NatInstance (t3.nano) and a single AZ will be created.constnatInstanceProvider=aws_ec2.NatProvider.instance({instanceType:aws_ec2.InstanceType.of(InstanceClass.T3,InstanceSize.NANO),//usingacheapergateway(notscalable)});constvpc=newVpc(this,"Vpc",{//Yourcustomvpc,i.e.:natGatewayProvider:natInstanceProvider,maxAzs:1,});consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,configuration:{//optionallyconfigureyourrunner},},],network:{vpc:vpc},});Seeexample,GitlabRunnerAutoscalingPropsZero configDeploys theAutoscaling GitLab Runner on AWS EC2with the default settings mentioned above.Happy with the presets?consttoken=StringParameter.fromStringParameterAttributes(stack,"Token",{parameterName:"/gitlab-runner/token",});newGitlabRunnerAutoscaling(this,"Runner",{runners:[{token:token,configuration:{//optionallyconfigureyourrunner},},],});Seeexample,GitlabRunnerAutoscalingPropsECR Credentials HelperBy default, the GitLabamzonec2 driverwill be configured to install theamazon-ecr-credential-helperon the runner's instances.To configure, override the default job runners environment:newGitlabRunnerAutoscaling(this,"Runner",{runners:[{//...environment:["DOCKER_DRIVER=overlay2","DOCKER_TLS_CERTDIR=/certs",'DOCKER_AUTH_CONFIG={"credHelpers": { "public.ecr.aws": "ecr-login", "<aws_account_id>.dkr.ecr.<region>.amazonaws.com": "ecr-login" } }',],},],});ProjenThis project usesprojento maintain project configuration through code. Thus, the synthesized files with projen should never be manually edited (in fact, projen enforces that).To modify the project setup, you should interact with rich strongly-typed
classAwsCdkTypeScriptAppand
executenpx projento update project configuration files.In simple words, developers can only modify.projenrc.jsfile for configuration/maintenance and files under/srcdirectory for development.See alsoCreate and Publish CDK Constructs Using projen and jsii.