nyuuzyou/wikis huggingface dataset:
Portal:Toolforge/Admin/APIs

Toolforge is slowly moving towards an API-oriented model where client tools (such as those installed on bastions) contact custom Toolforge APIs to make changes instead of making them directly.
== API gateway ==
To help decrease setup complexity for new APIs in this model, we've set up an API gateway to run in the Kubernetes cluster. The current implementation is based on Nginx with some Helm-generated configuration running in the Kubernetes cluster, which is then exposed via the HAProxy server pair.
=== Access ===
The base endpoint is https://api.svc.[project].eqiad1.wikimedia.cloud:30003. Services are routed with subpaths, for example /jobs for the Jobs API.
For authentication we currently use client certificates issued by the Kubernetes cluster internal CA via maintain-kubeusers. This might change in the future as we evolve how the APIs are accessed and used.
=== Backends ===
The backends are configured via Helm values. The communication between the gateway and the API gateway must be encrypted via internal CA certificates issued by the api-gateway-backend-ca ClusterIssuer.
Backend services can access the authentication certificate data via the ssl-client-subject-dn header.
== See also ==
* Help:Toolforge/API - user documentation
Portal:Toolforge/Admin/Apt repository

Toolforge has an internal apt repository managed with aptly. The repository exists on the tools-services nodes.
Repositories are declared in puppet, but packages should be added to the aptly repository by hand. We usually have one repository per operating system and project, i.e:
* stretch-tools
* buster-tools
* stretch-toolsbeta
* buster-toolsbeta
Quick example of packages being stored here are:
* https://gerrit.wikimedia.org/r/#/admin/projects/labs/toollabs
* https://gerrit.wikimedia.org/r/admin/projects/operations/software/tools-webservice
* https://gerrit.wikimedia.org/r/admin/projects/operations/software/tools-manifest
(among others)
The repository data, located at /srv/packages is stored in a mounted cinder volume.
== Deployment components and architecture ==
Information on how the setup is deployed, and the different components. 
=== Servers ===
Usually a VM with a cinder volume to store repository data.
=== Addressing, DNS and proxy ===
There is an horizon web proxy called deb-tools.wmcloud.org that should point to TCP/80 on the server. This allows to build docker images using toolforge internal packages.
Other than that, servers don't have any special DNS or adressing. The don't have floating IPs.
Worth noting that these servers in the tools cloudvps project may offer services for the toolsbeta project as well.
=== Puppet ===
The main role in use is role::wmcs::toolforge::services.
== Admin operations ==
Information on maintenance and administration of this setup. 
=== managing aptly repo ===
Is managed as a standard aptly repo.
==== health ====
Some interesting bits to check if you want to know the status/health of the server.
* aptly repos are present, and they contain packages, i.e: sudo aptly repo list and sudo aptly repo show --with-packages=true stretch-tools
* disk is not filled, i.e: df -h /
=== failover ===
We don't have a specific failover mechanism rather than building a new VM and re-attach the cinder volume.
Care should be taken to don't loss aptly repo data, since generating it from scratch can take some time.
== History ==
This was heavily remodeled when migrating the grid to SGE and to Stretch. Previous to the migration, the services nodes used to store Bigbrother (deprecated), and webservicemonitor (moved to cron servers).
Again, when migrating from Stretch to Buster, the 2 VM approach was dropped in favor of storing the data in a cinder volume, see https://phabricator.wikimedia.org/T278354.
== See also ==
* Portal:Toolforge/Admin
Portal:Toolforge/Admin/Envvars Service

Documentation of components and common admin procedures for Envvars Service.
== Components ==
* Envvars client (source code): main entrypoint for users
* Envvars API (source code): main entry-point for clients (users use the cli)
* Envvars admission controller (source code): Injects the envvars into the new pods.
== Alerts ==
None specific yet
* From the cloud UI: https://prometheus-alerts.wmcloud.org/?q=%40state%3Dactive&q=project%3D~^%28tools%7Ctoolsbeta%29
* From the prod UI: https://alerts.wikimedia.org/?q=team%3Dwmcs&q=project%3D~%28tools%7Ctoolsbeta%29
== Dashboards ==
TBD
== Main phabricator board ==
https://phabricator.wikimedia.org/project/board/539/
== Administrative tasks ==
=== Starting a service ===
==== Envvars API ====
This lives in kubernetes, behind the API gateway. To start it you can try redeploying it. To do so follow Portal:Toolforge/Admin/Kubernetes/Components#Deploy (the component is toolforge-envvars-api).
You can monitor if it's coming up with the usual k8s commands:
==== Envvars admission controller ====
Also a k8s component, follow Portal:Toolforge/Admin/Kubernetes/Components#Deploy (the component is envvars-admission-controller).
=== Stopping a service ===
==== Envvars API ====
Being a k8s deployment, the quickest way might be just to remove the deployment itself (will require redeploying to start again).
For a full removal (CAREFUL! Only if you know what you are doing) you can use helm:
==== Envvars admission controller ====
Similar to envvars api:
For a full removal (CAREFUL! Only if you know what you are doing) you can use helm:
=== Checking all components are alive ===
We don't have a unified dashboard yet, but for now you can check each component individually.
 
==== Envvars API ====
You can monitor if it's coming up with the usual k8s commands:
==== Envvars admission controller ====
Also a k8s component, different namespace:
Portal:Toolforge/Admin/Exim

== Email maintenance ==
Currently all mail services run on tools-mail. A few pointers:
* Exim is hard. This documentation is not: http://bradthemad.org/tech/notes/exim_cheatsheet.php
* Most relevantly: exim -q -v to manually run over the queue once
* Log files are in /var/log/exim/*
== Blocking a sender from sending outbound mail ==
Edit /etc/exim4/deny_senders.list and add the envelope from address that you want to block on a new line.
$ echo 'wiktcapt@tools.wmflabs.org' | sudo -i tee -a /etc/exim4/deny_senders.list
wiktcapt@tools.wmflabs.org
$ sudo -i cat /etc/exim4/deny_senders.list
# Add MAIL FROM address to block. One per line
wiktcapt@tools.wmflabs.org
== Stopping all in/outbound mail ==
If an incident seems to be occurring, be safe and stop exim before you spend a lot of time digging into the root cause. Having our MX address put on a blacklist for spamming is worse than some downtime.
$ sudo -i puppet agent --disable "Investigating exim incident -- $USER"
$ sudo -i puppet agent -tv # Check to see that Puppet is actually disabled
$ sudo -i service exim4 stop
$ ps axuww | grep exim # Did it really stop?
Now you can proceed to investigate the queue without more messages going in or out. SMTP is robust to network segmentation so even leaving things down for a few hours is not a huge problem. Messages will be delivered eventually.
== Exim runs as a queue on each exec node ==
Each exec node (and most other hosts actually) run a local copy of exim that queues messages for outbound delivery via the tools-mail smarthost. Even if you have purged the queue on the smarthost before restarting exim following an incident, there may be messages queued across the grid ready to flood in as soon as the service is back online.
== Failover ==
# Provision new node
# Disable Puppet on new node, stop Exim service
# Update Hiera key profile::toolforge::active_mail_relay to point to new instance
# Run Puppet everywhere
# Stop Puppet and Exim on the old node
# Swap floating IP to new node
# Enable Puppet and Exim service on new node
# Decomission old node
== See also ==
* Help:Toolforge/Email - user docs
* General VPS Exim docs
Portal:Toolforge/Admin/Kubernetes/Labels

== This page contains a list of known labels used in the Toolforge Kubernetes cluster. ==
There is an upstream guide on how to use labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
 kubernetes labels
Object	Label 	Values 	Set by 	Read by 	Comments
Pod 	toolforge 	"tool" 	webservice & toolforge-jobs-api	volume-admission-controller, envvars-admission-controller	Used by the volume admission controller to mount the tool home in the pod (unless disabled via toolforge.org/mount-storage). Used by the envvars admission controller to inject the envvars into the pod.
Pod 	toolforge.org/mount-storage 	"all" (default), "none" 	webservice & toolforge-jobs-api	volume-admission-controller 	If the pod should have NFS volumes mounted or not
Pod 	jobs.toolforge.org/filelog 	"yes" 	toolforge-jobs-api 	toolforge-jobs-api 	if a given job should create log files
Pod 	jobs.toolforge.org/emails 	"all", "onfinish", "onfailure"	toolforge-jobs-api 	toolforge-jobs-api & toolforge-jobs-emailer 	if a given job should produce emails
Node 	kubernetes.wmcloud.org/nfs-mounted	"true" 	puppet 	kube-scheduler 	if this node has access to NFS volumes
Node 	kubernetes.io/role 	"ingressgen2" 	manually 	kube-scheduler 	if this node is dedicated for ingress-nginx usage
Portal:Toolforge/Admin/Kubernetes/Upgrading Kubernetes/1.22 to 1.23 notes

= Toolforge Kubernetes 1.23 upgrade =
== prep ==
* [x] run prepare upgrade cookbook
* [x] downtime project via https://prometheus-alerts.wikimedia.org
* [x] update topic on -cloud
== control nodes ==
=== tools-k8s-control-4 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
=== tools-k8s-control-5 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
=== tools-k8s-control-6 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
== worker nodes ==
run upgrade node cookbook for each. it's ok to do a couple in parallel
[x] tools-k8s-worker-30
[x] tools-k8s-worker-31
[x] tools-k8s-worker-32
tools-k8s-worker-33
tools-k8s-worker-34
tools-k8s-worker-35
tools-k8s-worker-36
tools-k8s-worker-37
tools-k8s-worker-38
tools-k8s-worker-39
tools-k8s-worker-40
tools-k8s-worker-41
tools-k8s-worker-42
tools-k8s-worker-43
tools-k8s-worker-44
tools-k8s-worker-45
tools-k8s-worker-46
tools-k8s-worker-47
tools-k8s-worker-48
tools-k8s-worker-49
tools-k8s-worker-50
tools-k8s-worker-51
tools-k8s-worker-52
tools-k8s-worker-53
tools-k8s-worker-54
tools-k8s-worker-55
tools-k8s-worker-56
tools-k8s-worker-57
tools-k8s-worker-58
tools-k8s-worker-59
tools-k8s-worker-60
tools-k8s-worker-61
tools-k8s-worker-62
tools-k8s-worker-64
tools-k8s-worker-65
tools-k8s-worker-66
tools-k8s-worker-67
tools-k8s-worker-68
tools-k8s-worker-69
tools-k8s-worker-70
tools-k8s-worker-71
tools-k8s-worker-72
tools-k8s-worker-73
tools-k8s-worker-74
tools-k8s-worker-75
tools-k8s-worker-76
tools-k8s-worker-77
tools-k8s-worker-78
tools-k8s-worker-79
tools-k8s-worker-80
tools-k8s-worker-81
tools-k8s-worker-82
tools-k8s-worker-83
tools-k8s-worker-84
tools-k8s-worker-85
tools-k8s-worker-86
tools-k8s-worker-87
tools-k8s-worker-88
== ingress nodes ==
* [x] kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=2
* run upgrade node cookbook for each:
[x] tools-k8s-ingress-4
[x] tools-k8s-ingress-5
[x] tools-k8s-ingress-6
* [x] revert afterwards: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=3
== cleanup ==
* [x] remove downtime
* [x] revert topic change
Portal:Toolforge/Admin/Kubernetes/Upgrading Kubernetes/1.24 to 1.25 notes

= Toolforge Kubernetes 1.25 upgrade =
Parent task: phab:T316107
== prep ==
* [x] run prepare upgrade cookbook
* [x] downtime project via https://prometheus-alerts.wikimedia.org
* [x] update topic on -cloud
== control nodes ==
=== tools-k8s-control-7 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
=== tools-k8s-control-8 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
=== tools-k8s-control-9 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
== worker nodes ==
run upgrade node cookbook for each. it's ok to do a couple in parallel
* [x] tools-k8s-worker-nfs-1
* [x] tools-k8s-worker-nfs-2
* [x] tools-k8s-worker-nfs-3
* [x] tools-k8s-worker-nfs-4
* [x] tools-k8s-worker-nfs-5
* [x] tools-k8s-worker-nfs-6
* [x] tools-k8s-worker-nfs-7
* [x] tools-k8s-worker-nfs-8
* [x] tools-k8s-worker-nfs-9
* [x] tools-k8s-worker-nfs-10
* [x] tools-k8s-worker-nfs-11
* [x] tools-k8s-worker-nfs-12
* [x] tools-k8s-worker-nfs-13
* [x] tools-k8s-worker-nfs-14
* [x] tools-k8s-worker-nfs-15
* [x] tools-k8s-worker-nfs-16
* [x] tools-k8s-worker-nfs-17
* [x] tools-k8s-worker-nfs-18
* [x] tools-k8s-worker-nfs-19
* [x] tools-k8s-worker-nfs-20
* [x] tools-k8s-worker-nfs-21
* [x] tools-k8s-worker-nfs-22
* [x] tools-k8s-worker-nfs-23
* [x] tools-k8s-worker-nfs-24
* [x] tools-k8s-worker-nfs-25
* [x] tools-k8s-worker-nfs-26
* [x] tools-k8s-worker-nfs-27
* [x] tools-k8s-worker-nfs-28
* [x] tools-k8s-worker-nfs-29
* [x] tools-k8s-worker-nfs-30
* [x] tools-k8s-worker-nfs-31
* [x] tools-k8s-worker-nfs-32
* [x] tools-k8s-worker-nfs-33
* [x] tools-k8s-worker-nfs-34
* [x] tools-k8s-worker-nfs-35
* [x] tools-k8s-worker-nfs-36
* [x] tools-k8s-worker-nfs-37
* [x] tools-k8s-worker-nfs-38
* [x] tools-k8s-worker-nfs-39
* [x] tools-k8s-worker-nfs-40
* [x] tools-k8s-worker-nfs-41
* [x] tools-k8s-worker-nfs-42
* [x] tools-k8s-worker-nfs-43
* [x] tools-k8s-worker-nfs-44
* [x] tools-k8s-worker-nfs-45
* [x] tools-k8s-worker-nfs-46
* [x] tools-k8s-worker-nfs-47
* [x] tools-k8s-worker-nfs-48
* [x] tools-k8s-worker-nfs-49
* [x] tools-k8s-worker-nfs-50
* [x] tools-k8s-worker-nfs-51
* [x] tools-k8s-worker-nfs-52
* [x] tools-k8s-worker-nfs-53
* [x] tools-k8s-worker-nfs-54
* [x] tools-k8s-worker-nfs-55
* [x] tools-k8s-worker-nfs-56
* [x] tools-k8s-worker-102
* [x] tools-k8s-worker-103
* [x] tools-k8s-worker-104
* [x] tools-k8s-worker-105
* [x] tools-k8s-worker-106
* [x] tools-k8s-worker-107
== ingress nodes ==
* [x] kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=2
* run upgrade node cookbook for each:
* [x] tools-k8s-ingress-7
* [x] tools-k8s-ingress-8
* [x] tools-k8s-ingress-9
* [x] revert afterwards: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=3
== cleanup ==
* [x] remove downtime
* [x] revert topic change
Portal:Toolforge/Admin/Kubernetes/Upgrading Kubernetes/1.25 to 1.26 notes

= Toolforge Kubernetes 1.26 upgrade =
Parent task: phab:T327025
== prep ==
* [x] run prepare upgrade cookbook
* [x] downtime project via https://prometheus-alerts.wikimedia.org
* [x] update topic on -cloud
== control nodes ==
=== tools-k8s-control-7 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
* [x] depool control-8 and -9 via haproxy, check that control-7 is still doing ok
=== tools-k8s-control-8 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
=== tools-k8s-control-9 ===
* [x] run upgrade node cookbook
* [x] check that services start healthy
== worker nodes ==
run upgrade node cookbook for each. it's ok to do a couple in parallel
* [x] tools-k8s-worker-nfs-1
* [x] tools-k8s-worker-nfs-2
* [x] tools-k8s-worker-nfs-3
* [x] tools-k8s-worker-nfs-5
* [x] tools-k8s-worker-nfs-6
* [x] tools-k8s-worker-nfs-7
* [x] tools-k8s-worker-nfs-8
* [x] tools-k8s-worker-nfs-9
* [x] tools-k8s-worker-nfs-10
* [x] tools-k8s-worker-nfs-11
* [x] tools-k8s-worker-nfs-12
* [x] tools-k8s-worker-nfs-13
* [x] tools-k8s-worker-nfs-14
* [x] tools-k8s-worker-nfs-16
* [x] tools-k8s-worker-nfs-17
* [x] tools-k8s-worker-nfs-19
* [x] tools-k8s-worker-nfs-20
* [x] tools-k8s-worker-nfs-21
* [x] tools-k8s-worker-nfs-22
* [x] tools-k8s-worker-nfs-23
* [x] tools-k8s-worker-nfs-24
* [x] tools-k8s-worker-nfs-26
* [x] tools-k8s-worker-nfs-27
* [x] tools-k8s-worker-nfs-28
* [x] tools-k8s-worker-nfs-29
* [x] tools-k8s-worker-nfs-30
* [x] tools-k8s-worker-nfs-31
* [x] tools-k8s-worker-nfs-32
* [x] tools-k8s-worker-nfs-33
* [x] tools-k8s-worker-nfs-34
* [x] tools-k8s-worker-nfs-35
* [x] tools-k8s-worker-nfs-36
* [x] tools-k8s-worker-nfs-37
* [x] tools-k8s-worker-nfs-38
* [x] tools-k8s-worker-nfs-39
* [x] tools-k8s-worker-nfs-40
* [x] tools-k8s-worker-nfs-41
* [x] tools-k8s-worker-nfs-42
* [x] tools-k8s-worker-nfs-43
* [x] tools-k8s-worker-nfs-44
* [x] tools-k8s-worker-nfs-45
* [x] tools-k8s-worker-nfs-46
* [x] tools-k8s-worker-nfs-47
* [x] tools-k8s-worker-nfs-48
* [x] tools-k8s-worker-nfs-49
* [x] tools-k8s-worker-nfs-50
* [x] tools-k8s-worker-nfs-53
* [x] tools-k8s-worker-nfs-54
* [x] tools-k8s-worker-nfs-55
* [x] tools-k8s-worker-nfs-56
* [x] tools-k8s-worker-nfs-57
* [x] tools-k8s-worker-nfs-58
* [x] tools-k8s-worker-nfs-60
* [x] tools-k8s-worker-nfs-61
* [x] tools-k8s-worker-nfs-62
* [x] tools-k8s-worker-nfs-63
* [x] tools-k8s-worker-nfs-64
* [x] tools-k8s-worker-102
* [x] tools-k8s-worker-103
* [x] tools-k8s-worker-105
* [x] tools-k8s-worker-106
* [x] tools-k8s-worker-107
* [x] tools-k8s-worker-108
== ingress nodes ==
* [x] kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=2
* run upgrade node cookbook for each:
* [x] tools-k8s-ingress-7
* [x] tools-k8s-ingress-8
* [x] tools-k8s-ingress-9
* [x] revert afterwards: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=3
== cleanup ==
* [x] remove downtime
* [x] revert topic change
Portal:Toolforge/Admin/Kubernetes/foxtrot-ldap

This page contains information on Toolforge foxtrot-ldap, a project to help setting up a fake LDAP server deployment mainly for Toolforge lima-kilo, local kubernetes environment in a given local machine (like a laptop), to help develop some of the Toolforge internal components.
The foxtrot-ldap project was forked from Striker .
Repository: https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap
Portal:Toolforge/Admin/Kubernetes/lima-kilo

This page contains information on Toolforge lima-kilo, a development and testing environment for Toolforge that reproduces most of the system components used in production.
The lima-kilo project is based on ansible and lima, light-weight linux VMs compatible with both linux and macOS (Intel and ARM) hosts.
Repository: https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo
== Rationale ==
Developing some Toolforge kubernetes components can be challenging because Toolforge is not just a bare k8s deployment. It is Toolforge's kubernetes. There are custom admission webhooks, custom namespaces, a particular security model, some host files are expected to be present, etc.
The lima-kilo recreates most of this service context in a local environment in a way that, thanks to ansible and lima, is lightweight, robust, deterministic, idempotent and easy to use.
Lima-kilo uses kind for its kubernetes component. The lima guest VM runs Debian. So far, lima-kilo has been tested on various linux distributions and both Intel and ARM macOS. It should probably work on WSL2, but this is so far untested.
If you run into compatibility issues while trying to set up lima-kilo on your system, don't hesitate to ask for help. 
The project was mildly inspired by MediaWiki-Vagrant.
== Basic usage workflow ==
See the README file for some hints.
== See also ==
* Portal:Toolforge/Admin/Kubernetes
* Portal:Toolforge/Admin/Kubernetes/Components
* https://docs.ansible.com/ansible/latest/index.html
* The foxtrot-ldap page, fake LDAP for Toolforge internal development
Portal:Toolforge/Admin/Legacy redirector for webservices

In 2020, a change happened to how Toolforge webservices are accessed from the internet.
* The domain changed from tools.wmflabs.org to toolforge.org.
* The URL scheme changed from tools.wmflabs.org/<toolname>/ to <toolname>.toolforge.org/.
To preserve legacy URLs as working endpoints, we deployed a mechanism called legacy redirector that is meant stay online for as long as we consider it useful (forever?).
== How it works ==
In a separate server from the main front proxy, there is an nginx server that loads a LUA script. This script checks if the tool is allowed to have a legacy redirect (we don't want to allow the legacy redirect for newer webservices), by checking a static list.
The legacy FQDN tools.wmflabs.org must point to this VM, so users can access it directly from the internet (TODO: floating IP?)
Then, the end client will see a HTTP status code 301 moved permanently, pointing to the new domain/URL scheme.
The static list is generated by checking the front proxy redis for registered webservices in the grid and by checking ingresses in the kubernetes cluster.
There is a helper script to generate the list, should we need to update it:
 Arturo's page, wmcs-generate-legacy-redirector-map.py script.
== Deployment ==
A simple virtual machine with the puppet role role::wmcs::toolforge::legacy_redirector and the following hiera configuration:
profile::toolforge::canonical_domain: toolforge.org
profile::toolforge::canonical_scheme: https://
profile::toolforge::proxy::do_https: true
Note that these values are the default in the puppet profile manifest, so it's OK if we don't set then explicitly anyway.
In the case of toolsbeta, the hiera config needs to be:
profile::toolforge::canonical_domain: toolsbeta.wmflabs.org
profile::toolforge::canonical_scheme: http://
profile::toolforge::proxy::do_https: false
The most important service in the server is the nginx service, which has a very simplified configuration. Logs can be found at /var/log/nginx/access.log and /var/log/nginx/error.log.
== See also ==
* Toolforge kubernetes networking and ingress
* Toolforge.org announcement
Portal:Toolforge/Admin/Maintenance

This page contains information for common maintenance tasks we administrator do in Toolforge.
Is the Toolforge-specific version of the CloudVPS admin maintenance conterpart wikipage.
== Admin scripts ==
This section contains documentation on several Toolforge-specific admin scripts we have.
=== wmcs-k8s-get-cert ===
This script is for the new Toolforge Kubernetes cluster.
It creates a x509 cert for a given User that can be used to auth against the k8s API. It must be executed from a control plane node.
user@tools-k8s-control-3:~$ sudo wmcs-k8s-get-cert -help
INFO: Usage of this script:
 /usr/local/sbin/wmcs-k8s-get-cert -h/--help 	- show help and exit
 /usr/local/sbin/wmcs-k8s-get-cert <svcname> 	- generate a x509 TLS cert from the kubernetes API
 /usr/local/sbin/wmcs-k8s-get-cert <svcname> -v	- same, but in verbose mode
Example usage:
user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-get-cert prometheus
/tmp/tmp.JYFsVzDnX4/k8s-cert.pem
/tmp/tmp.JYFsVzDnX4/k8s-key.pem
NOTE: x509 certs expire in 1 year usually. See also Portal:Toolforge/Admin/Kubernetes/Certificates
=== wmcs-k8s-enable-cluster-monitor ===
This is a script for the Toolforge Kubernetes cluster, deployed to the control plane nodes.
It will give a specified tool account a service account named $tool-obs that will allow a pod running as it to have "view" access to the entire cluster. This is for building tools similar to Openstack Browser for Kubernetes.
user@tools-k8s-control-3:~$ sudo -i # You'll want full root for k8s credentials
root@tools-k8s-control-3:~# wmcs-enable-cluster-monitor <tool-name>
If that runs successfully, you'll want to run the tool as that service account using a script like k8s_webservice.sh instead of using webservice.
Important point about using a service account: it must use the credentials mounted at /var/secrets/kubernetes.io/serviceaccount in the pod and not $HOME/.kube/config or you'll get the tool permissions instead of the $tool-obs permissions. This is largely because of invoking the pod presets to get the $HOME variable set to /data/project/$tool and the NFS mounted.
== Other scripts ==
Other scripts you may find interesting.
=== wmcs-package-build ===
This script is used to build and publish packages into aptly. More information in the packaging page.
You can locate it in the puppet tree: modules/toolforge/files/wmcs-package-build.py.
It is intended to run from your laptop.