Package: testflows.github.runners
PyPi:VersionsLicense:Apache-2.0Autoscaling GitHub Actions Runners Using Hetzner CloudA simple alternative to Github’sRecommended autoscaling solutions.🔍 Tip:You can easily navigate this documentation page by clicking on any title to jump to theTable of Contents.
Try it out, and remember, if you get lost, just click any title!Thegithub-runnersservice program starts and monitors queued up jobs for GitHub Actions workflows.
When a new job is queued up, it creates a new Hetzner Cloud server instance
that provides an ephemeral GitHub Actions runner. Each server instance is automatically
powered off when the job completes, and then powered off servers are
automatically deleted. Bothx64(x86) andarm64(arm) runners are supported.
SeeFeaturesandLimitationsfor more details.❗Warning:This program is provided on “AS IS” basis without warranties or conditions of any kind. See LICENSE.
Use it at your own risk. Manual monitoring is required to make sure server instances are cleaned up properly
and costs are kept under control.Costs depend on the server type, number of jobs, and execution time. For each job, a new server instance is created
to avoid any cleanup. Server instances are not shared between jobs.✋ Note:Currently, Hetzner Cloud server instances are billed on an hourly basis. So a job that takes 1 min will be billed
the same way as for a job that takes 59 minutes. Therefore, the minimal cost
for any job, the cost of the server for 1 hour plus the cost of one public IPv4 address.Table of ContentsFind out more about,Autoscaling GitHub Actions Runners Using Hetzner CloudTable of ContentsFeaturesLimitationsPrerequisitesInstallationQuick StartGetting Started TutorialInstalling TestFlows Github RunnersCreating GitHub Repository with Actions Workflow and TokenCreating a Hetzner Cloud Project and TokenCreating a Cloud ServiceWaiting for GitHub Actions Job to CompleteInstallation From SourcesBasic ConfigurationSpecifying the Maximum Number of RunnersSpecifying the Maximum Number of Runners Used in Workflow RunRecycling Powered-Off ServersSkipping JobsJobs That Require the Docker EngineSpecifying the Runner Typex64 RunnersARM64 RunnersSpecifying Runner LocationSpecifying the Runner ImageSpecifying the Custom Runner Server Setup ScriptSpecifying Standby RunnersSpecifying Logger ConfigurationListing All Current ServersOpening SSH Client to the ServerDeleting All Runners And Their ServersUsing a Configuration FileSpecifying SSH KeyGenerating a New SSH KeySSH Keys in Cloud DeploymentSpecifying Additional SSH KeysRunning as a ServiceInstalling and UninstallingModifying Program OptionsChecking StatusManual Start and StopChecking LogFollowing The LogSelecting Log ColumnsSelecting the Number of LinesRaw LogRunning as a Cloud ServiceDeployingUsing an ARM64 InstanceUsing x64 InstanceRedeploying Cloud ServiceCloud Service LogCloud Service StatusStopping Cloud ServiceStarting Cloud ServiceInstalling Cloud ServiceUninstalling the Cloud ServiceUpgrading the Cloud Service PackageChanging Cloud Service OptionsDeleting the Cloud ServiceSSH in to Cloud ServiceScaling Up RunnersMaximum Number of RunnersNew ServerThe Setup ScriptThe Start-up ScriptScaling Down RunnersPowered Off ServersUnused RunnersZombie ServersHandling Failing ConditionsProgram OptionsFeaturescost efficient on-demand runners usingHetzner Cloudsupports server recycling to minimize costssimple configuration, no Webhooks, no need for AWS lambdas, and no need to setup any GitHub applicationsupports specifying custom runner server types, images, and locations using job labelsself-contained program that you can use to deploy, redeploy, and manage the service on a cloud instancesupports x64 (x86) and ARM64 (arm) runnerssupports using any Hetzner Cloud server typessupports runners with pre-installed Dockersupports using any standard Hetzner Cloud images and applicationssupports auto-replenishable fixed standby runner pools for jobs to be picked up immediatelysupports limiting the maximum number of runners created for each workflow runsupports efficient GitHub API usage using HTTP caching and conditional requestssimpler alternative to what GitHub lists inRecommended autoscaling solutionsLimitationsGroup runners are not supported✎ However, you can run individual services for each repository using different Hetzner Cloud projects.A unique Hetzner Cloud project must be used for each repository✎ However, unique projects allow you to easily keep track of runner costs per repository.PrerequisitesPython >= 3.7Hetzner CloudaccountGitHub API token with admin privileges to manage self-hosted runnersInstallationpip3installtestflows.github.runnersCheck that thegithub-runnersutility was installed correctly by executing thegithub-runners -vcommand.github-runners-vThegithub-runnersutility is installed in to the~/.local/bin/folder. Please make sure that this folder
is part of thePATH.whichgithub-runners~/.local/bin/github-runnersIf yourPATHis missing this folder, on Ubuntu, modify your~/.profileand add the following section:~/.profile:# set PATH so it includes the user's private bin if it existsif[-d"$HOME/.local/bin"];thenPATH="$HOME/.local/bin:$PATH"fiQuick StartSet environment variables corresponding to your GitHub repository and Hetzner Cloud project.exportGITHUB_TOKEN=ghp_...exportGITHUB_REPOSITORY=vzakaznikov/github-runnersexportHETZNER_TOKEN=GJzdc...Then, start thegithub-runnersprogram:github-runners07/22/2023 08:20:37 PM   INFO MainThread            main 🍀 Logging in to Hetzner Cloud
07/22/2023 08:20:37 PM   INFO MainThread            main 🍀 Logging in to GitHub
07/22/2023 08:20:37 PM   INFO MainThread            main 🍀 Getting repository vzakaznikov/github-runners
07/22/2023 08:20:37 PM   INFO MainThread            main 🍀 Creating scale up service
07/22/2023 08:20:37 PM   INFO MainThread            main 🍀 Creating scale down service
07/22/2023 08:20:38 PM   INFO   worker_2   create_server 🍀 Create server
...Alternatively, you can pass the required options using the command line as follows:github-runners--github-token<GITHUB_TOKEN>--github-repository<GITHUB_REPOSITORY>--hetzner-token<HETZNER_TOKEN>Getting Started Tutorial✅ Launch your first self-hosted runner in:5 minutesThis tutorial will guide you on how to use thegithub-runnersprogram to provide autoscaling GitHub Actions runners
for a GitHub repository and a Hetzner Cloud project that you’ll create.Installing TestFlows Github Runners❶ Before we get started, you will need to installtestflows.github.runnersPython package. See theInstallationsection for more details.pip3installtestflows.github.runners❷ Check that thegithub-runnersutility was installed correctly by executing thegithub-runners -vcommand.github-runners-v1.3.230731.1173142✋ Note:Thegithub-runnersutility is installed in to the~/.local/bin/folder. Please make sure that this folder
is part of thePATH.whichgithub-runners~/.local/bin/github-runnersIf yourPATHis missing this folder, on Ubuntu, you can modify your~/.profileand add the following section:~/.profile:# set PATH so it includes the user's private bin if it existsif[-d"$HOME/.local/bin"];thenPATH="$HOME/.local/bin:$PATH"fiIn order to launch thegithub-runnersprogram, we’ll need to specify the GitHub repository as well as GitHub and
Hetzner Cloud tokens. So, let’s create these.Creating GitHub Repository with Actions Workflow and TokenBefore using thegithub-runners, you need a GitHub repository with a GitHub Actions workflow set up.❶ First, create a GitHub Repository nameddemo-testflows-github-runnersand note the repository name.The repository name will have the following format:<username>/demo-testflows-github-runnersFor me, my GitHub repository is:vzakaznikov/demo-testflows-github-runners❷ Now, create an example GitHub Actions workflow as described in theQuickstart for GitHub Actionsarticle.
Note that we need to modify the example YAML configuration and specify that our job will run on a runner with theself-hostedand thetype-cpx21labels.Explore-GitHub-Actions:runs-on:[self-hosted,type-cpx21]So, the completedemo.ymlthat uses a self-hosted runner is as follows:demo.yml:name:GitHub Actions Demorun-name:${{ github.actor }} is testing out GitHub Actions 🚀on:[push]jobs:Explore-GitHub-Actions:runs-on:[self-hosted,type-cpx21]steps:-run:echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."-run:echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"-run:echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."-name:Check out repository codeuses:actions/checkout@v3-run:echo "💡 The ${{ github.repository }} repository has been cloned to the runner."-run:echo "🖥️ The workflow is now ready to test your code on the runner."-name:List files in the repositoryrun:|ls ${{ github.workspace }}-run:echo "🍏 This job's status is ${{ job.status }}."❸ Finally, you will need to create a GitHub API token with theworkflowprivileges. Make sure to save the token!For me, mydemoGitHub token is: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You should now have your GitHub repository ready.See these steps in action:Creating a Hetzner Cloud Project and TokenNext, you will need to create a Hetzner Cloud project and an API token that we can use to create and manage Hetzner Cloud server instances.❶ Create a new Hetzner Cloud projectDemo GitHub Runners.❷ Now, create an API token and save it.For me, the Hetzner Cloud token for myDemo GitHub Runnersproject is:5Up04IHuY8mC7l0JxKwh3Aps4ghGIyL0NJ9rGlhyAmmkddzuRreR1YstTSTFCG0NYou should now have your Hetzner Cloud project ready.See these steps in action:Creating a Cloud ServiceWith the GitHub repository and GitHub and Hetzner Cloud tokens in hand, we can deploy thegithub-runnersservice
to Hetzner Cloud instance. This way, the service is not running on your local machine.The deployment, we’ll create agithub-runnersinstance in your Hetzner Cloud project on which the service will be running.
See theRunning as a Cloud Servicesection for details.❶ To deploy the service run thegithub-runners cloud deploycommand and specify your
GitHub repository, GitHub and Hetzner Cloud tokens usingGITHUB_REPOSITORY,GITHUB_TOKEN, andHETZNER_TOKENenvironment variables.exportGITHUB_REPOSITORY=exportHETZNER_TOKEN=exportGITHUB_TOKEN=github-runnersclouddeployYou should now have the cloud service up and running.See these steps in action:Waiting for GitHub Actions Job to Complete❶ Thegithub-runnerscloud service is now running. So, now you can just sit back and wait untilgithub-runnersspins up a new runner to complete any queued up GitHub Actions jobs in your GitHub repository.See this step in action:As you can see, our job was executed and completed using our own self-hosted runner!✋ Note:If you run into any issues, you can check the cloud service log using thegithub-runners cloud log -fcommand. For other cloud service commands, see theRunning as a Cloud Servicesection.github-runnerscloudlog-fInstallation From SourcesFor development, you can install from sources as follows:gitclonehttps://github.com/testflows/Github-Runners.git./package&&./installBasic ConfigurationBy default, the program uses the following environment variables:GITHUB_TOKENGITHUB_REPOSITORYHETZNER_TOKENor you can specify these values using the following options:–github-token–github-repository–hetzner-tokenSpecifying the Maximum Number of RunnersThe default maximum number of runners is10. You can set a different value
based on your Hetzner Cloud limits using the-m count, –max-runners countoption. For example,github-runners--max-runners40Specifying the Maximum Number of Runners Used in Workflow RunBy default, the maximum number of runners that could be created for a single workflow run
is not defined.❗Warning:In general, GitHub does not allow you to assign a job to a specific runner, and any available runner
that matches the labels could be used. Therefore, one can’t control how runners are allocated
to queued workflow run jobs, and this is why the–max-runners-in-workflow-runoption will not behave
as one would intuitively expect.If you set the–max-runners-in-workflow-runto some valueX, thengithub-runnerswill create theX * number of queued workflow runsrunners. How these runners will be allocated by
GitHub is out of our control. Therefore, the more runs queued up, the more runners will be created, up to the–max-runnerslimit, to try to complete the jobs faster. However, this does not mean that you will see exactlyXnumber of jobs
being executed in each queued workflow run.For example,github-runners--max-runners40--max-runners-in-workflow-run5will create upto5runners for each queued up workflow run. If there is only one workflow running, then the maximum number of
runners will be5unless more queued up workflow runs appear, which could then speed up the execution of the run in progress.Recycling Powered-Off ServersBy default, recycling of powered-off servers that have completed executing a job is turned on.Recycling allows for minimizing costs by allowing multiple runners to be brought up on
the same server instance as Hetzner Cloud, which bills servers in 1 hour increments.
Therefore, it is inefficient to delete a server if it only executed a job
that runs for a few minutes. Instead, after completing a job, the server is powered off
and if it can be recycled, it is rebuilt from scratch by reinstalling the image
thus providing a clean environment for the next job.Powered off servers are marked as recyclable by changing their name togithub-runner-recycle-{uid}.Recyclable servers are deleted when they reach their end of life period
which is defined by the–end-of-lifeoption, and by default is set to50minutes.
The end of life is calculated on an hourly basis and must be greater than0and less than60.For example, with the default value of the–end-of-lifeoption set to the50minutes,
if the server is running for 2 hours and 50 minutes, then it will be
considered to have reached its end of life and is deleted because it has only10minutes or less of useful life
left in the current hour period.
However, if the server is running for 2 hours and 30 minutes, then it could potentially
has 30 minutes of life left, and it will be kept around to be available for recycling.Sometimes a job might need a server that does not match any recyclable servers,
if the maximum number of runners has been reached, then by default, one of the recyclable servers
will be picked to be deleted to make room for a new server. By default, the recyclable server
that is deleted is picked based on the server’s price per hour and its remaining useful life.
The server with the lowestunused budgetis deleted.Theunused budgetis defined as follows:unused budget:server_life=60-server_age.minutesprice_per_minute=price_per_hour/60unused_budget=server_life*price_per_minute✋ Note:You can also use the–delete-randomoption to randomly pick a recyclable server to be deleted.
Deleting servers at random is a legacy feature.A recyclable server is recycled for a new job if it matches the following:server type matches exactly what the job requires or the default typeserver location matches exactly if a job requests a runner in a specific location or the default location is specifiedserver has matching SSH keys✋ Note:Matching server type exactly means that even if a bigger, more expensive server type
could be potentially recycled if it is not used, even though a job that actually requires
that expensive server might not be queued before the server’s end of life.This is intensional, as we can’t predict when a job that actually requires the more expensive
server type could be queued. If the program would allow recycling of higher server types
than actually requested by a job, then we could run into cases when a job
that requires a smaller and less expensive server runs on a bigger and more expensive server instead.
In this case, a job that actually requires a bigger server would force a new, expensive server to be created
and thus causing more expensive servers to be created than are actually necessary.If needed, you can turn recycling off using the–recycle {on,off}option.github-runners--recycleoffSkipping JobsBy default, a runner will be created for anyqueuedjob.If needed, you can skip creating runners if a job does not have a specified label
using the–with-labeloption.For example,github-runners--with-labelon-demandwill only create runners for jobs that containon-demandlabel and skip any job that is missing
that label.Jobs That Require the Docker EngineFor jobs that require Docker to be installed, you can use the standardHetzner Docker CE applicationwhich can be specified using theimage-label. SeeSpecifying the Runner Imagefor more details about specifying custom runner images.For examplex64:job-name:runs-on:[self-hosted,type-cx11,image-x86-app-docker-ce]ARM64:job-name:runs-on:[self-hosted,type-cax11,image-arm-app-docker-ce]Specifying the Runner Typex64 RunnersThe default server type iscx11which is an Intel, 1 vCPU, 2GB RAM shared-cpu x64 instance.✋ Note:You can use–default-typeoption to set a different default server type.You can specify different x64 server instance type by using thetype-{name}runner label.
The{name}must be a validHetzner Cloud server typename such ascx11,cpx21etc.For example, to use an AMD, 3 vCPU, 4GB RAM shared-cpu x64 instance, you can define theruns-onas follows:job-name:runs-on:[self-hosted,type-cpx21]ARM64 RunnersThe default,server type iscx11, which is an Intel, 1 vCPU, 2GB RAM shared-cpu x64 instance.
Therefore, in order to use ARM64 runners, you must specify the ARM64 server instance type by using thetype-{name}runner label.
The{name}must be a validARM64 Hetzner Cloud server typename such ascax11,cax21etc. which correspond to the Ampere Altra, 2 vCPU, 4GB RAM and
4 vCPU, 8GB RAM shared-cpu ARM64 instances, respectively.For example, to use Ampere Altra, 4 vCPU, 8GB RAM shared-cpu ARM64 instance, you must define theruns-onas follows:job-name:runs-on:[self-hosted,type-cax21]Specifying Runner LocationBy default, the default location of the server where the runner will be running is not specified. You can use the–default-locationoption to force a specific default server location.You can also use thein-{name}runner label to specify the server location for a specific job. Where{name}must be a validHetzner Cloud locationname such asashfor US, Ashburn, VA orfsn1for Germany, Falkenstein.For example,job-name:runs-on:[self-hosted,type-cx11,in-ash]Specifying the Runner ImageBy default, the default image of the server for the runner isubuntu-22.04. You can use the–default-imageoption to force a specific default server image.You can also use theimage-{architecture}-{type}-{name}runner label to specify the server image for a specific job.Where,{architecture}is eitherx86orarm{type}is eithersystem,snapshot,backup, orapp{name}must be a valid Hetzner Cloud image name, forsystemorapptype, such asubuntu-22.04,
or a description, forbackuporsnapshottype.For example,ubuntu-20.04:job-name:runs-on:[self-hosted,type-cx11,in-ash,image-x86-system-ubuntu-20.04]docker-ce app:job-name:runs-on:[self-hosted,type-cx11,in-ash,image-x86-app-docker-ce]snapshot:For snapshots, specifydescriptionas the name. Snapshot descriptions
must be unique.job-name:runs-on:[self-hosted,type-cx11,in-ash,image-x86-snapshot-snapshot_description]Specifying the Custom Runner Server Setup ScriptYou can specify a custom runner server setup script using the–setup-scriptoption.For example,custom_setup.sh:#!/bin/bashset-xecho"Create and configure ubuntu user"adduserubuntu--disabled-password--gecos""echo"%wheel   ALL=(ALL:ALL) NOPASSWD:ALL">>/etc/sudoersaddgroupwheeladdgroupdockerusermod-aGwheelubuntuusermod-aGsudoubuntuusermod-aGdockerubuntu# custom setupapt-get-yupdateapt-get-yinstallca-certificatescurlgnupglsb-releasepython3-pipgitunzipcommand:github-runners--setup-script./custom_setup.shSpecifying Standby RunnersYou can define standby runner groups to always be ready to pick up your jobs using a custom configuration file.✋ Note:Standby runner groups can only be defined using a configuration file.
SeeUsing a Configuration Filefor more details.Standby runners are always active and allow jobs to be picked up immediately.More than one standby runner group can be specified in thestandby_runners. Each group is defined using thestandby_runnerobject
that haslabels,count, andreplenish_immediatelyattributes.schema:"standby_runners":{"type":"array","items":{"type":"object","properties":{"labels":{"type":"array","items":{"type":"string"}},"count":{"type":"integer"},"replenish_immediately":{"type":"boolean"}}}}where,labelsspecifies an array of labels with which standby runners in this group should be createdcountspecifies the count of how many runners should be created for the groupreplenish_immediatelyspecifies if the sandby runners should be replenished as soon as they become busy after picking up a job, default: trueFor example,config.yaml:config:standby_runners:-labels:-type-cx21count:2replenish_immediately:trueSpecifying Logger ConfigurationYou can specify custom logger configuration using a configuration file.✋ Note:A custom logger configuration can only be specified using a configuration file.
SeeUsing a Configuration Filefor more details.The logger configuration is specified in the configuration file using thelogger_configobject.
For more information about the logger configuration, seeConfiguration dictionary schemain Python documentation.Any custom logger configuration must at least definestdoutandrotating_service_logfilehandlers
as well as configuretestflows.github.runnersin theloggers.For example,config.yaml:config:# logging module configlogger_config:version:1disable_existing_loggers:falseformatters:standard:format:"%(asctime)s%(levelname)s%(funcName)s%(message)s"datefmt:"%m/%d/%Y%I:%M:%S%p"handlers:stdout:level:INFOformatter:standardclass:testflows.github.runners.logger.StdoutHandlerstream:"ext://sys.stdout"rotating_service_logfile:level:DEBUGformatter:standardclass:testflows.github.runners.logger.RotatingFileHandlerfilename:/tmp/github-runners.logmaxBytes:10485760backupCount:1loggers:testflows.github.runners:level:INFOhandlers:-stdout-rotating_service_logfileIf the logger configuration is using a custom format for therotating_service_logfile, then a customlogger_formatobject
must be defined to specify the format of the service’s rotating log file which is needed for theservice logandcloud logcommands.For the example above, the customlogger_formatis the following:config:# logger formatlogger_format:delimiter:""default:-column:date-column:time-column:time_ampm-column:level-column:funcName-column:messagecolumns:-column:dateindex:0width:10-column:timeindex:1width:8-column:time_ampmindex:2width:2-column:levelindex:3width:8-column:funcNameindex:4width:15-column:messageindex:5width:80Note that thedate,time, andtime_ampmcolumns come from thedatefmtdefinition, which
defines theasctimeas a three column field consisting ofdate,time, andtime_ampmcolumns
separated by a space.datefmt:"%m/%d/%Y%I:%M:%S%p"Listing All Current ServersYou can list all currently created servers using thelistcommand.
This command will show all the servers that start with thegithub-runnerprefix in their name.For example,github-runnerslistUsing config file: /home/user/.github-runners/config.yaml
11:40:40 🍀 Logging in to Hetzner Cloud
11:40:40 🍀 Getting a list of servers
❌ off        github-runner-5811138574-15753659850
❌ off        github-runner-recycle-1691595565.5396028
❌ off        github-runner-recycle-1691595478.7024605
❌ off        github-runner-5811138574-15753660130
❌ off        github-runner-recycle-1691595481.196499Opening SSH Client to the ServerFor debugging, you can open an SSH client to the current server using thesshcommand and specify the name of the server you would like to connect to.
For thesshcommand to work, you need to specify the–hetzner-tokenand have the correct private SSH key.github-runnersssh<name>For example,github-runnerssshgithub-runner-5811138574-15753659850Deleting All Runners And Their ServersYou can delete all runners, including standby runners, and their servers using thedeletecommand.✋ Note:Thedeletecommand will not delete a cloud service server. If you also want to delete it,
you also need to executecloud deletecommand. For more information, seeDeleting the Cloud Servicesection.github-runnersdelete07/29/2023 07:43:16 PM     INFO       MainThread             all 🍀 Logging in to Hetzner Cloud
07/29/2023 07:43:16 PM     INFO       MainThread             all 🍀 Logging in to GitHub
07/29/2023 07:43:16 PM     INFO       MainThread             all 🍀 Getting repository testflows/testflows-github-runners
07/29/2023 07:43:17 PM     INFO       MainThread             all 🍀 Getting list of self-hosted runners
07/29/2023 07:43:17 PM     INFO       MainThread             all 🍀 Getting list of serversUsing a Configuration FileInstead of passing configuration options using command line arguments, you can use
configuration file. The configuration file uses YAML format, and it is usually namedconfig.yaml. You can find the complete schema
inschema.json.✋ Note:When you mix command line options with a custom configuration file,
explicit command line options take precedence over the values that are defined
for the same parameters in the configuration file.You can specify the default configuration by placing the configuration in the~/.github-runners/config.yamlfile or
pass the path to the configuration file explicitly using the-c path, –config pathoption.The YAML configuration file supports special syntax to specify the value of a property as the value of the environment variable using
the${ENV_VAR_NAME}syntax.For example,github-runners-cconfig.yamlwhere,config.yaml:config:github_token:${GITHUB_TOKEN}github_repository:${GITHUB_REPOSITORY}hetzner_token:${HETZNER_TOKEN}default_server_type:cx11cloud:server_name:"my-github-runners-service"standby_runners:-labels:-type-cx21count:2replenish_immediately:true✋ Note:This is a simple configuration file. You can find a complete example in theexamples/config.yaml.Specifying SSH KeyAll server instances that are created are accessed via SSH using thesshutility and therefore you must provide a valid SSH key
using the–ssh-keyoption. If the–ssh-keyoption is not specified, then the~/.ssh/id_rsa.pubdefault key path will be used.The SSH key will be automatically added to your project using the MD5 hash of the public key as the SSH key name.❗Warning:Given that each new SSH key is automatically added to your Hetzner project, you must manually delete them when no longer needed.Most GitHub users already have an SSH key associated with the account. If you want to know how to add an SSH key, seeAdding a new SSH key to your GitHub accountarticle.Generating a New SSH KeyIf you need to generate a new SSH key, seeGenerating a new SSH key and adding it to the ssh-agentarticle.SSH Keys in Cloud DeploymentIf you are deploying thegithub-runnersprogram as a cloud service using thegithub-runners <options> cloud deploycommand, then
after provisioning a new cloud server instance that will host thegithub-runnersservice, a new SSH key will be
auto-generated to access the runners. The auto-generated key will be placed in/home/runner/.ssh/id_rsa, whererunneris the user under which thegithub-runnersservice runs on the cloud instance. The auto-generated SSH key will be automatically
added to your project using the MD5 hash of the public key as the SSH key name.Specifying Additional SSH KeysIn addition to the main SSH key specified by the–ssh-keyoption, which is used to connect to the servers, you
can specify additional SSH keys using theadditional_ssh_keysproperty in the configuration file.
This is needed in cases where there is more than one user that should have access to the servers used for the runners.✋ Note:Additional SSH keys can only be defined using a configuration file.
SeeUsing a Configuration Filefor more details.Note that the additional SSH keys are defined using only the public key. This enables additional users to hold the matching private key
to connect to the servers.For example,config.yaml:config:additional_ssh_keys:-ssh-rsa AAAAB3Nza3... user@user-node-ssh-rsa AADDDFFFC1... another_user@another-nodeRunning as a ServiceYou can rungithub-runnersas a service.✋ Note:In order to install the service, the user who installed the module must havesudoprivileges.Installing and UninstallingAfter installation, you can useservice installandservice uninstallcommands to install and
uninstall the service.✋ Note:The options that are passed to thegithub-runners <options> service installcommand
will be the same options with which the service will be executed.exportGITHUB_TOKEN=ghp_...exportGITHUB_REPOSITORY=testflows/github-runnersexportHETZNER_TOKEN=GJzdc...github-runnersserviceinstallThe/etc/systemd/system/github-runners.servicefile is created with the following content.✋ Note:The service will use theUserand theGroupof the user executing the program./etc/systemd/system/github-runners.service:[Unit]
Description=Autoscaling GitHub Actions Runners
After=multi-user.target
[Service]
User=1000
Group=1000
Type=simple
Restart=always
Environment=GITHUB_TOKEN=ghp_...
Environment=GITHUB_REPOSITORY=testflows/github-runners
Environment=HETZNER_TOKEN=GJ..
ExecStart=/home/user/.local/lib/python3.10/site-packages/testflows/github/runners/bin/github-runners --workers 10 --max-powered-off-time 20 --max-unused-runner-time 120 --max-runner-registration-time 60 --scale-up-interval 10 --scale-down-interval 10
[Install]
WantedBy=multi-user.targetModifying Program OptionsIf you want to modify service program options, you can stop the service,
edit the/etc/systemd/system/github-runners.servicefile by hand, then reload the service daemon,
and start the service back up.github-runnersservicestopsudovim/etc/systemd/system/github-runners.servicesudosystemctldaemon-reloadgithub-runnersservicestartsChecking StatusAfter installation, you can check the status of the service using theservice statuscommand.github-runnersservicestatus:service status:● github-runners.service - Autoscaling GitHub Actions Runners
     Loaded: loaded (/etc/systemd/system/github-runners.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-07-24 14:38:33 EDT; 1h 31min ago
   Main PID: 66188 (python3)
      Tasks: 3 (limit: 37566)
     Memory: 28.8M
        CPU: 8.274s
     CGroup: /system.slice/github-runners.service
             └─66188 python3 /usr/local/bin/github-runners --workers 10 --max-powered-off-time 20 --max-unused-runner-time 120 --max->

Jul 24 14:38:33 user-node systemd[1]: Started Autoscaling GitHub Actions Runners.
Jul 24 14:38:33 user-node github-runners[66188]: 07/24/2023 02:38:33 PM   INFO MainThread            main 🍀 Logging in to Hetzner >
Jul 24 14:38:33 user-node github-runners[66188]: 07/24/2023 02:38:33 PM   INFO MainThread            main 🍀 Logging in to GitHub
Jul 24 14:38:33 user-node github-runners[66188]: 07/24/2023 02:38:33 PM   INFO MainThread            main 🍀 Getting repository vza>
Jul 24 14:38:33 user-node github-runners[66188]: 07/24/2023 02:38:33 PM   INFO MainThread            main 🍀 Creating scale up serv>
Jul 24 14:38:33 user-node github-runners[66188]: 07/24/2023 02:38:33 PM   INFO MainThread            main 🍀 Creating scale down se>
lines 1-16/16 (END)Manual Start and StopYou can start and stop the service using theservice startandservice stopcommands as follows:github-runnersservicestartsgithub-runnersservicestopor usingservicesystem utilitysudoservicegithub-runnersstartsudoservicegithub-runnersstop✋ Note:Theservice stopcommand will send theSIGINTsignal to thegithub-runnersprocess and will wait for the
program to perform a clean shutdown, which includes stopping scale up and scale down loops.
Given that thegithub-runnersprogram might be in the middle of configuring servers, theservice stopcommand might take sometime to complete.Checking LogYou can get the log for the service using theservice logcommand.Following The LogUse the-f, –followoption to follow the log journal. By default, the last1000lines will be shown and
then the log will be followed, and the new messages will be displayed as they are added to the log.github-runnersservicelog-ffollowed log:github-runners service log -f
Using config file: /home/user/.github-runners/config.yaml
18:11:49 api_watch      INFO     🍀 Consumed 0 calls in 60 sec, 5000 calls left, reset in 3599 sec
18:12:49 api_watch      INFO     🍀 Logging in to GitHub
18:12:49 api_watch      INFO     🍀 Checking current API calls consumption rate
18:12:49 api_watch      INFO     🍀 Consumed 0 calls in 60 sec, 5000 calls left, reset in 3599 sec
18:13:49 api_watch      INFO     🍀 Logging in to GitHub
18:13:49 api_watch      INFO     🍀 Checking current API calls consumption rate
18:13:50 api_watch      INFO     🍀 Consumed 0 calls in 60 sec, 5000 calls left, reset in 3599 sec
...You can dump the full log by omitting the-f, –followoption.github-runnersservicelogfull log:Using config file: /home/user/.github-runners/config.yaml
09:44:28 http_cache     INFO     🍀 Enabling HTTP cache at /tmp/tmp60wo30tc/http_cache
09:44:28 main           INFO     🍀 Logging in to Hetzner Cloud
09:44:28 main           INFO     🍀 Logging in to GitHub
09:44:28 main           INFO     🍀 Getting repository testflows/testflows-github-runners
09:44:28 main           INFO     🍀 Checking if default image exists
09:44:29 main           INFO     🍀 Checking if default location exists
09:44:29 main           INFO     🍀 Checking if default server type exists
09:44:29 main           INFO     🍀 Getting server prices
09:44:30 main           INFO     🍀 Checking if SSH key exists
...Selecting Log ColumnsYou can use the-c name[:width][,…], –columns name[:width][,…]option to specify
a comma separated list of columns to include in the output as well as their optional width.For example,github-runnersservicelog-f-ctime,message:50Using config file: /home/user/.github-runners/config.yaml
Using config file: /home/user/.github-runners/config.yaml
18:13:50 🍀 Consumed 0 calls in 60 sec, 5000 calls left,
         reset in 3599 sec
18:14:50 🍀 Logging in to GitHub
18:14:50 🍀 Checking current API calls consumption rate
...By default, the following columns are available unless you redefine thelogger_formatin your configuration file:datetimelevelintervalfuncNamethreadNamerun_idjob_idserver_namemessageSelecting the Number of LinesYou can select the number of lines you would like to output from the log using the-n [+]number, –lines [+]numberoption. With the–followthe default is10.You can use the+before thenumberto output a log starting with the specified line number.For example,github-runnersservicelog-n50github-runnersservicelog-n+100github-runnersservicelog-f-n1Raw LogBy default, the log is processed and broken up into columns based on thelogger_formatconfiguration.
You can output the raw log by specifying the–rawoption.Running as a Cloud ServiceInstead of runninggithub-runnersprogram locally as a standalone application or as a service.
You can easily deploygithub-runnersto run on a Hetzner Cloud instance.See-h, –helpfor all the available commands.✋ Note:By default, the server name where thegithub-runnersservice will be running
isgithub-runners. If you want to use a custom server name, then
you must use thecloud –nameoption for anycloudcommands.github-runnerscloud-hDeployingYou can deploygithub-runnersas a service to a new Hetzner Cloud server instance, that will be created for you automatically,
using thecloud deploycommand.✋ Note:The options that are passed to thegithub-runners <options> cloud deploycommand
will be the same options with which the service will be executed.exportGITHUB_TOKEN=ghp_...exportGITHUB_REPOSITORY=testflows/github-runnersexportHETZNER_TOKEN=GJzdc...github-runnersdeployYou can specify the version of the package to be installed using the–versionoption. By default, the current local package
version will be installed on the cloud service server. You can also passlatestas the value to install the latest available
version.github-runnersdeploy--versionlatestThedeploycommand will use the following default values:location:ashtype:cpx11image:ubuntu-22.04Thecloud deploycommand uses the following setup script.setup script:set-xapt-getupdateapt-get-yinstallpython3-pipapt-get-yinstallopenssh-clientecho"Create and configure ubuntu user"adduserubuntu--disabled-password--gecos""echo"%wheel   ALL=(ALL:ALL) NOPASSWD:ALL">>/etc/sudoersaddgroupwheelusermod-aGwheelubuntuusermod-aGsudoubuntuecho"Generate SSH Key"sudo-uubuntussh-keygen-trsa-q-f"/home/ubuntu/.ssh/id_rsa"-N""You can customize deployment server location, type, and image using the–location,–type, and–imageoptions.github-runnersdeploy--locationnbg1--typecx11--imageubuntu-22.04The cloud instance that runs thegithub-runnersservice can either be x64 or ARM64 instance. By default,cpx11AMD, 2 vCPU, 2GB RAM, shared-cpu x64 instance type is used.Using an ARM64 InstanceIf you want to deploy thegithub-runnersservice to an ARM64 instance, then you must specify the instance
type using the–typeoption.✋ Note:Currently, Hetzner Cloud has ARM64 instances only available in Germany, Falkenstein (fsn1) location.For example, to use an Ampere Altra, 4 vCPU, 8GB RAM shared-cpu ARM64 instance, you must specifycax21as the value of the–typeas follows:github-runnersdeploy--locationfsn1--typecax21--imageubuntu-22.04Using x64 InstanceBy default, thecpx11AMD, 2 vCPU, 2GB RAM, shared-cpu x64 instance type is used. If you want to use
a different x64 instance, then specify the desired type using the–typeoption.Redeploying Cloud ServiceYou can change the cloud service configuration or cloud service package version without deleting the existing cloud service server
using thecloud redeploycommand.github-runners<options>cloudredeploy✋ Note:The options that are passed to thegithub-runners <options> cloud redeploycommand
will be the same options with which the service will be executed.You can specify the version of the package to be installed using the–versionoption.Cloud Service LogYou can check the log for thegithub-runnersservice running on a cloud instance using thegithub-runners cloud logcommand.
Specify-f, –followif you want to follow the log journal.For example,dump the full log:github-runnerscloudlogfollow the log journal:github-runnerscloudlog-fYou can also specify the–rawoption to output the raw log as well as use the-c name[:width][,…], –columns name[:width][,…]option to specify a comma separated list of columns to include in the output and their optional width.Cloud Service StatusYou can check the status of thegithub-runnersservice running on a cloud instance using thegithub-runners cloud statuscommand.For example,github-runnerscloudstatusStopping Cloud ServiceYou can manually stop thegithub-runnersservice running on a cloud instance using thegithub-runners cloud stopcommand.github-runnerscloudstopStarting Cloud ServiceYou can manually start thegithub-runnersservice running on a cloud instance after it was manually stopped
using thegithub-runners cloud startcommand.github-runnerscloudstartInstalling Cloud ServiceYou can manually force installation of thegithub-runnersservice running on a cloud instance using
thegithub-runners cloud installcommand.✋ Note:Just like with thegithub-runners <options> service installcommand,
the options that are passed to thegithub-runners <options> cloud installcommand
will be the same options with which the service will be executed.You can specify-f, –forceoption to force service reinstallation if it is already installed.github-runners<options>cloudinstall-fUninstalling the Cloud ServiceYou can manually force the uninstallation of thegithub-runnersservice running on a cloud instance using
thegithub-runners cloud uninstallcommand.github-runnersclouduninstallUpgrading the Cloud Service PackageYou can manually upgrade thegithub-runnersservice package running on a cloud instance using
thegithub-runners cloud upgradecommand.If a specific ‘–version’ is specified, then thetestflows.github.runnerspackage is upgraded to
the specified version, otherwise the version is upgraded to the latest available.✋ Note:The service is not reinstalled during the package upgrade process.
Instead, it is stopped before the upgrade and then started back up
after the package upgrade is complete.github-runnerscloudupgrade--version<version>The service is not reinstalled during the package upgrade process.
Instead, it is stopped before the upgrade and then started back upChanging Cloud Service OptionsIf you need to change cloud service options such as the–setup-scriptor the–max-runnersetc.,
you can keep the existing server and usecloud redeploycommand.github-runners<options>cloudredeploy--versionlatestWhen needed, you can also SSH into the cloud service manually and perform changes manually.You can do a complete service teardown using thecloud deleteand then thecloud deploycommands.github-runnersclouddeletegithub-runners<options>clouddeploy--versionlatest✋ Note:A complete teardown will not affect any current jobs, as the service is designed to
be restartable. However, some servers might be left in an unfinished state
but they will be cleaned up when the service is restarted.Deleting the Cloud ServiceYou can delete thegithub-runnerscloud service and the cloud instance that is running on it using
thegithub-runners cloud deletecommand.Thecloud deletecommand, deletes the cloud service by first stopping the service and then deleting the server instance.❗Warning:The default server name where the cloud service is deployed isgithub-runners.
Please make sure to specify thecloud –nameoption if you have deployed the service to a server with a different name.For example,default name:github-runnersclouddeletecustom name:github-runnerscloud--name<custom_name>deleteSSH in to Cloud ServiceYou can open an SSH client to the cloud service using thecloud sshcommand. For example,github-runnerscloudsshYou can also manually SSH into the cloud service using thesshutility. For convenience, you can
retrieve the SSH client command using thecloud ssh commandcommand. For example,github-runnerscloudsshcommandThe output will contain the fullsshcommand including the IP address of the cloud service server.ssh -q -o "StrictHostKeyChecking no" root@5.161.87.21Scaling Up RunnersThe program scales up runners by looking for any jobs that havequeuedstatus.
For each such job, a corresponding Hetzner Cloud server instance is created with the following name:github-runner-{job.run_id}-{job.id}The server is configured using the defaultsetupandstartupscripts. The runner’s name is set
to be the same as the server name so that servers can be deleted for any unused runner that, for some reason
does not pick up a job for which it was created within themax-unused-runner-timeperiod.Note:Given that the server name is fixed and specific for eachjob.id, if multiplegithub-runnersare running in parallel, then
only 1 server will be created for a givenjoband any other attempts to create a server with the same name will be rejected
by the Hetzner Cloud.Also,Note:There is no guarantee that a given runner will pick the job with the exactjob.idthat caused it to be created.
This is expected, and for eachqueuedjob a unique runner will be created the number of runners will be
equal the number of jobs, and therefore, under normal conditions, all jobs will be executed as expected.Maximum Number of RunnersBy default, the maximum number of runners and, therefore the maximum number of server instances are not set and are therefore unlimited.
You can set the maximum number of runners using the–max-runnersoption.New ServerThe new server is accessed using SSH. It boots up with the specified OS image and is configured using
thesetupandstartupscripts.Server Type:The default server type iscx11which is an Intel, 1 vCPU, 2GB RAM shared-cpu x64 instance.You can specify different x64 server instance type by using thetype-{name}runner label.
The{name}must be a validHetzner Cloudserver type name such ascx11,cpx21etc.For example, to use an AMD, 3 vCPU, 4GB RAM shared-cpu x64 instance, you can define theruns-onas follows:job-name:runs-on:[self-hosted,type-cpx21]Server Location:The server location can be specified by using the–default-locationoption or thein-<name>runner label.
By default, location is not set, as some server types are not available in some locations.Image:The server is configured to have the image specified by the–default-imageoption or theimage-{architecture}-{type}-{name}runner label.SSH Access:The server is configured to be accessed usingsshutility, and the SSH public key path is specified using the–ssh-keyoption.Image Configuration:Each new server instance is configured using thesetupand thestartupscripts.The Setup ScriptThesetupscript creates and configuresrunneruser that hassudoprivileges.Setup:set-xecho"Create and configure ubuntu user"adduserubuntu--disabled-password--gecos""echo"%wheel   ALL=(ALL:ALL) NOPASSWD:ALL">>/etc/sudoersaddgroupwheelusermod-aGwheelubuntuusermod-aGsudoubuntuThe Start-up ScriptThestartupscript installs the GitHub Actions runner. After installation, it configures the runner to start in an–ephemeralmode.
The–ephemeralmode causes the runner to exit as soon as it completes a job. After the runner exits, the server is powered off.✋ Note:Thestartupscript is executed asubuntuuser and therefore you must usesudofor any commands that needrootprivileges.The x64startupscript installs and configures x64 version of the runner.x64:set-xecho"Install runner"cd/home/ubuntucurl-oactions-runner-linux-x64-2.306.0.tar.gz-Lhttps://github.com/actions/runner/releases/download/v2.306.0/actions-runner-linux-x64-2.306.0.tar.gzecho"b0a090336f0d0a439dac7505475a1fb822f61bbb36420c7b3b3fe6b1bdc4dbaa  actions-runner-linux-x64-2.306.0.tar.gz"|shasum-a256-ctarxzf./actions-runner-linux-x64-2.306.0.tar.gzecho"Configure runner"./config.sh--unattended--replace--urlhttps://github.com/${GITHUB_REPOSITORY}--token${GITHUB_RUNNER_TOKEN}--name"$(hostname)"--runnergroup"${GITHUB_RUNNER_GROUP}"--labels"${GITHUB_RUNNER_LABELS}"--work_work--ephemeralecho"Start runner"bash-c"screen -d -m bash -c './run.sh; sudo poweroff'"The ARM64startupscript is similar to the x64 script but installs an ARM64 version of the runner.ARM64:set-xecho"Install runner"cd/home/ubuntucurl-oactions-runner-linux-arm64-2.306.0.tar.gz-Lhttps://github.com/actions/runner/releases/download/v2.306.0/actions-runner-linux-arm64-2.306.0.tar.gz#Optional:Validatethehashecho"842a9046af8439aa9bcabfe096aacd998fc3af82b9afe2434ddd77b96f872a83  actions-runner-linux-arm64-2.306.0.tar.gz"|shasum-a256-c#Extracttheinstallertarxzf./actions-runner-linux-arm64-2.306.0.tar.gzecho"Configure runner"./config.sh--unattended--replace--urlhttps://github.com/${GITHUB_REPOSITORY}--token${GITHUB_RUNNER_TOKEN}--name"$(hostname)"--runnergroup"${GITHUB_RUNNER_GROUP}"--labels"${GITHUB_RUNNER_LABELS}"--work_work--ephemeralecho"Start runner"bash-c"screen -d -m bash -c './run.sh; sudo poweroff'"Scaling Down RunnersPowered Off ServersThe program scales down runners by first cleaning up powered off servers. The scaled down service relies on the fact
that thestartupscript starts an ephemeral runner that will pick up only 1 job and then power itself off after the job is complete.The powered off servers are deleted after themax-powered-off-timeinterval which
can be specified using the–max-powered-off-timeoption, which by default is set to20sec.Unused RunnersThe scale down service also monitors all the runners that haveunusedstatus and tries to delete any servers associated with such
runners if the runner isunusedfor more than themax-unused-runner-timeperiod. This is needed in case a runner never gets a job
assigned to it, and the server will stay in the power on state. This cycle relies on the fact that the runner’s name
is the same as server’s name. Themax-unused-runner-timecan be specified using the–max-unused-runner-timeoption, which by default
is set to180sec.Zombie ServersThe scale down service will delete any zombie servers. A zombie server is defined as any server that fails to register its runner within
themax-runner-registration-time. Themax-runner-registration-timecan be specified using the–max-runner-registration-timeoption
which by default is set to180sec.Handling Failing ConditionsThe program is designed to handle the following failing conditions:Server Never Registers a Runner:The server will remain inrunningstate and should be reclaimed by the scale down service when it checks the actual runners registered for current servers.
If it finds a server that isrunningbut no runner is active for it it will be deleted after themax-runner-registration-timeperiod.The./config.shCommand Fails:The behavior will be the same as for theServer Never Registers a Runnercase above.The./run.shCommand Fails:The server will be powered off by thestartupscript and deleted by the scale down service.Creating Server For Queued Job Fails:If creation of the server fails for some reason, then the scale up service will retry the operation in the next interval as the job’s status will remainqueued.Runner Never Gets a Job Assigned:If the runner never gets a job assigned, then the scale down service will remove the runner and delete its server after themax-unused-runner-timeperiod.Runner Created With a Mismatched Labels:The behavior will be the same as for theRunner Never Gets a Job Assignedcase above.Program OptionsThe following options are supported:-h, –helpshow this help message and exit-v, –versionshow program’s version number and exit–licenseshow program’s license and exit-r {on,off}, –recycle {on,off}turn on or off recycling of powered off servers, either ‘on’ or ‘off’, default:on–end-of-life minutesnumber of minutes in 1 hour (60 minutes) period after which a recyclable server
is considered to have reached its end of life and thus is deleted, default:50-c path, –config pathprogram configuration file–github-token GITHUB_TOKENGitHub token, default:$GITHUB_TOKENenvironment variable–github-repository GITHUB_REPOSITORYGitHub repository, default:$GITHUB_REPOSITORYenvironment variable–hetzner-token HETZNER_TOKENHetzner Cloud token, default:$HETZNER_TOKENenvironment variable–ssh-key pathpublic SSH key file, default:~/.ssh/id_rsa.pub–default-type namedefault runner server type name, default:cx11–default-location namedefault runner server location name, default:not specified–default-image architecture:type:name_or_descriptiondefault runner server image type and name or description,
where the architecture is either: ‘x86’ or ‘arm’,
and type is either: ‘system’,’snapshot’,’backup’,’app’,
default:system:ubuntu-22.04-m count, –max-runners countmaximum number of active runners, default:10–delete-randomdelete random recyclable server when the maximum number of servers is reached, by default, server prices are used–max-runners-in-workflow-run countmaximum number of runners allowed to be created for a single workflow run, default: not set–with-label labelonly create runners for jobs that have the specified label,
by default, jobs are not skipped, and runners will be created for any queued job–label-prefix prefixsupport type, image, and location job labels with the specified prefix-w count, –workers countnumber of concurrent workers, default:10–setup-script pathpath to the custom server setup script–startup-x64-script pathpath to the custom server startup script–startup-arm64-script pathpath to the custom ARM64 server startup script–max-powered-off-time secmaximum time after which a powered off server is deleted, default:60sec–max-unused-runner-time secmaximum time after which an unused runner is removed and its server deleted, default:180sec–max-runner-registration-timemaximum time after which the server will be deleted if its runner is not registered with GitHub, default:180sec–max-server-ready-time secmaximum time to wait for the server to be in the running state, default:180sec–scale-up-interval secscale up service interval, default:15sec–scale-down-interval secscale down service interval, default:15sec–debugenable debugging mode, default:Falsecommands:commanddeletedelete all serverslistlist all serverssshssh to a servercloudcloud service commands-n server, –name serverdeployment server name, default:github-runnersdeploydeploy cloud service-f, –forceforce deployment if it already exists–version number|latestservice package version to deploy, either a version number or ‘latest’,
default: current package version-l name, –location namedeployment server location, default:ash-t name, –type namedeployment server type, default:cpx11-i architecture:type:name_or_description, –image architecture:type:name_or_descriptiondeployment server image type and name or description,
where the architecture is either: ‘x86’ or ‘arm’,
and the type is either: ‘system’,’snapshot’,’backup’,’app’,
default:system:ubuntu-22.04–setup-script pathpath to custom deployment server setup scriptredeployredeploy on the same cloud service server–version number|latestservice package version to deploy, either a version number or ‘latest’,
default: current package versionlogget cloud service log-c name[:width][,…], –columns name[:width][,…]comma separated list of columns to include and their optional width–rawoutput raw log-f, –followfollow log journal, default:False-n [+]number, –lines [+]numberoutput the last number of lines, with –follow the default is 10,
use ‘+’ before the number to output the log, starting with the line numbercommanddeletedelete logstatusget cloud service statusstartstart cloud servicestopstop cloud serviceinstallinstall cloud service-f, –forceforce installation if service already existsuninstalluninstall cloud serviceupgradeupgrade cloud service–version versionpackage version, default:the latestsshssh to cloud servicecommandprint ssh command to cloud serviceserviceservice commandsinstallinstall service-f, –forceforce installation if service already existsuninstalluninstall servicestatusget service statuslogget service log-c name[:width][,…], –columns name[:width][,…]comma separated list of columns to include and their optional width-f, –followfollow log journal, default:False-n [+]number, –lines [+]numberoutput the last number of lines, with –follow the default is 10,
use ‘+’ before the number to output the log, starting with the line number–rawoutput raw logcommandformatformat logdeletedelete logstartstart servicestopstop service