nyuuzyou/wikis huggingface dataset:
Portal:Toolforge/Admin/Monthly meeting

The Toolforge monthly meeting is a one-hour engineering-focused meeting to discuss the current state and the future plans of Toolforge, to unblock some decisions or to get everyone on the same page. It is currently held on the first Tuesday of every month at 17:30 CE(S)T.
The first Toolforge monthly meeting was held in November 2022, and there had been several informal Toolforge-specific meetings from time to time before that. During its history, the meeting has also been known as the Toolforge council meeting or the Toolforge workgroup meeting. The Toolforge workgroup name is now sometimes used to refer to the bi-weekly check-in meeting held by the Cloud Services team staff who regularly work on the Toolforge project.
This meeting is meant to be open in nature:
* The Wikimedia Cloud Services team (WMCS)
* Anyone with root-level access on Toolforge
* Any other interested parties can be invited if requested (reach out on irc/email)
== Next meeting agenda ==
The next meeting is planned for 2025-02-18.
* k8s upgrade workgroup progress
* OKR progress
* <Add yours here!>
You can find the live meeting notes here (they will be moved to the wiki pages below).
== Former meeting minutes ==
Portal:Toolforge/Admin/Monthly meeting/2024-03-12

== Notes ==
=== phab:T306039 - Decision request - Toolforge external infrastructure domain usage ===
* We have some services that need to be accessed externally, and we did not have a convention for those, the proposal is `svc.toolforge.org`
** Approved as proposed
* Bryan has reserved the tool name in 2019
* Nova proxy will support select subdomains (pre-determined, not user-specified)
=== phab:T314664 Two days to grid engine shut down ===
* Komla: copypatrol has migrated using python2, still a few tools, and most of the rest are non-responsive, good to go
* The grid deprecation tool is not working as expected, but we will defer fixing it (until it's not needed xd)
* Toolschecker grid stuff is already gone? -yes \o/
=== Jobs api refactor ===
* At a high level, separate the frontend that users interact with from the backend that interacts with Kubernetes primitives such that k8s details do not leak out to the frontend.
** Our "continuous job" can then map to more than a single k8s object
** Allow keeping our metadata separate from k8s metadata in the sense that we won't have to force everything we store into the abstractions allowed by k8s
* AB: we should look at https://www.crossplane.io/ as a potential way to make the custom storage be a k8s CRD
** https://www.crossplane.io/why-control-planes explains a bit about how this approach can leverage built-in k8s operations for reconciliation
== Action items ==
* Taavi to close decision request task and implement required novaproxy support
* Taavi to shut down the grid as originally planned
Portal:Toolforge/Admin/Monthly meeting/2024-05-14

=== Agenda ===
* Bird's eye view of toolforge ideas: https://wikitech.wikimedia.org/wiki/User:David_Caro/Toolforge_high_view
=== Notes ===
==== David presents ‘Toolforge high view’ ====
* Anything on the diagram not connected to another box can be worked on async
* Arturo suggests that we publish the consolidated API definition asap
** We don’t have authentication yet (sort of)
*** Auth via certificates sort of works but isn’t very portable
** (debate ensues about releasing APIs now vs solidifying plans so we don’t have to announce and wait to make changes)
* Andrew requests that release points be added to the plan so we can get some sort of push-to-deploy product out soon
* Arturo asks about current push-to-deploy auth
** Web hook w/token that bypasses the ssl cert
** API call to create web hook which generates the token
* (side discussion about benefits and risks of having a project manager – sounds like it isn’t happening anytime soon anyway)
* Trove may never be reliable enough to use as a first class dbaas solution. K8s might have provide a better solution in the long run, or self-managed things in persistent volumes.
==== Hackathon recap ====
* Some positive feedback about the k8s migration from a traditionally grumpy user
* No real outages or bottlenecks during the hackathon, users got what they needed
==== Can we stop requiring manual new toolforge account approval? ====
* Taavi says, not until there’s better visibility on what tools are doing
* Everyone wants this anyway
* We can partially audit via what buildpacks people are using
* Arturo says: there’s real DOS danger with people overwhelming e.g. maintainkubeusers
=== Action items ===
* [] Publish the toolforge high view doc
** E.g. like https://meta.wikimedia.org/wiki/Community_Tech/Tool_Labs_support/Tool_Labs_vision
* [] Get visibility (via email?) about new tool/tool account creation
** [] Then plan an experimental period of automatic account creation w/out requiring manual approval
Portal:Toolforge/Admin/Monthly meeting/2024-07-09

=== Attendees: ===
* Andrew Bogott
* Arturo Borrero
* Bryan Davis (bd808)
* David Caro
* Raymond Ndibe
* Seyram Komla Sapaty
* Slavina Stefanova
=== Agenda: ===
* Toolforge KR6.3.1: Sustainability scoring brainstorm (see live document)
=== Notes ===
* "Sustainability" is ideally for the users, but a balance of the time that Toolforge admins can provide (?Did I get that right?)
* 3 part KR: this, improved deployment, web ui for Toolforge
* "By the end of Q2, introduce a sustainability scoring system for the Toolforge platform across a variety of technical and social factors. By the end of Q4, improve one of its key technical factors by 50%."
* Some clarification in terminology: when we’re saying ‘admins’ in the KR, that refers to toolforge roots (not users/tool maintainers)
Portal:Toolforge/Admin/Monthly meeting/2024-11-05

== Attendees ==
* Arturo Borrero
* Bryan Davis
* David Caro
* Francesco Negri
* Sarai Sanchez
* Seyram Komla Sapaty
* Slavina Stefanova
== Agenda ==
* k8s upgrade workgroup progress
* KR/hypothesis progress
* Toolforge standard committee
== Notes ==
=== k8s upgrade workgroup progress ===
Upgraded to 1.28 yesterday, quite smooth
Some nodes had a small issue where the cookbook tries to evacuate the node, but is unable to, though deleting the pods manually worked (all user tools, 2 deployments 1 cronjob)
There was an issue (T378976) where a tool was not able to start, that might be related to NFS, the user restarted the webservice and worked (probably in a different worker?), keep an eye for any node having stuck processes
Next is 1.29, Francesco and Arturo, will probably not get to it before the end-of-year PTO but will start working on it
Toolsdb mariadb upgrade has priority, so that might take the time instead
Missing the webservice deploy for the latest code - will happen tomorrow
=== KR/hypothesis progress ===
==== 6.3.5 components-api ====
Have working token auth in local dev. Deploy to toolsbeta next
Want to generate a async client library from the OpenAPI spec to use in the components-api service
Will create a limited components-api client (command line), mainly to be able to write functional tests easily 
==== 6.3.7 Sustainability score ====
Delivery of sustainability score pushed to Q3
==== 6.3.6 hypothesis ====
Looking for pain points in user journeys. 
Need to gather and incorporate user feedback. Thinking about QuickSurveys as a possible tool to gather information. Need to find out if that can be enabled on Wikitech. 
Looking at some 3rd party platforms like fly.io and Heroku to see how they do things
Evaluate creating high-level designs representing UI-based workflows
=== Toolforge Standard Committee ===
8 folks went through the process and they’re all set up on the existing mailing list (we kicked out previous members).
How do we avoid being in the same situation where after 7 years we have to start the committee over from scratch? Some discussions are ongoing.
Committee members have started looking through the Phabricator backlog and closing some old tasks.
A difference from before: there is a tool named “toolforge-standards-committee” and everyone in the committee is a co-maintainer for that tool. If there is an abandoned tool, the “toolforge-standards-committee” tool can be added as a co-maintainer for the abandoned tool.
Should we invite the members of the committee to this monthly meeting?
David to draft an invite with some info about the meeting 
Bryan can deliver the invite via the committee's mailing list
Portal:Toolforge/Admin/Monthly meeting/2025-01-14

== Attendees ==
* Arturo Borrero
* Bryan Davis
* Cindy Cicalese
* David Caro
* Francesco Negri
* Lucas Werkmeister
* MusikAnimal
* Parthiv Menon
* Sarai Sanchez
* Seyram Komla Sapaty
* Slavina Stefanova
== Agenda ==
* Around the room introductions
* k8s upgrade workgroup progress - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Ongoing_Efforts/Toolforge_Upgrade_Workgroup/Upgrades_Overview
** Francesco is the driver of the next upgrade, Arturo is shadow (though in sick leave)
** Move things this Jan, and upgrade in Feb
* OKR progress
** Trying to lay out toolforge planning things (https://miro.com/app/board/uXjVLti6_Lw=/)
** Toolforge UI https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_UI
== Notes ==
=== K8s upgrade workgroup progress ===
* Francesco is driving the upgrade to K8s 1.29, hopefully will be completed by the next meeting (scheduled for Feb 11).
** David might replace Arturo as the “second person” for this upgrade (see table), as Arturo is not available this month.
=== OKR progress ===
* Push to Deploy: David started a Miro board.
* Toolforge UI: discussion page was shared in a couple newsletters. We should share it to cloud-announce if we haven’t yet.
Portal:Toolforge/Admin/Redis

Toolforge Redis is running on three nodes (tools-redis-5/6/7). One of them is a master and rest of them are replicas. Keepalived makes sure a virtual IP address is always assigned to the master that the clients can connect to.
If the current master goes down, Redis Sentinel should notice that within five seconds and automatically fail over to a replica. It might take additional 10 seconds for the floating IP to move to the new master.
If Sentinel does not fail over to a new node (use redis-cli info replication to check), look into /var/log/redis/redis-sentinel.log on any alive node. If the IP address does not move, check sudo systemctl status keepalived and check that the /usr/local/bin/wmcs-check-redis-master script has exit code 0 on the master and 1 on the replicas.
Note that Sentinel requires a quorum to perform any actions - that means that it will not function with two nodes down. Additionally Redis has been configured to not accept any writes on the replicas or on the master if no replicas are connected.
== Systemd unit ==
We are not using the default Systemd unit redis-server.service that comes with the Debian package. We are using a custom unit named redis-instance-tcp_6379.service that is deployed via Puppet.
== Manual failover ==
If you need to force a failover or perform other Sentinel actions, you can connect to it using redis-cli on port 26379:
taavi@toolsbeta-redis-1:~$ redis-cli -p 26379
127.0.0.1:26379>
Sentinel commands are listed at redis, use toolforge as the "master name".
The most useful command is sentinel failover toolforge which forces a failover to any other available node. You can alternatively add the IP address of the node to fail over to.
== Puppet configuration ==
Puppet is configured to never update (replace => false) the config files /etc/redis/tcp_6379.conf and /etc/redis/sentinel-toolforge.conf, to prevent clashes with Redis Sentinel, which can also modify those files.
This means that if you change or add a config value in modules/profile/manifests/toolforge/redis_sentinel.pp, it will not end up in the actual config files, unless you manually remove them in all Redis server, and let Puppet recreate them.
# # Repeat on all Redis hosts, starting from the current primary
# rm /etc/redis/sentinel-toolforge.conf /etc/redis/tcp_6379.conf
# run-puppet-agent
The commands above will cause Redis to restart and likely cause a short Redis outage.
A possible improvement to the current setup is tracked in phab:T366365.
== See also ==
* Toolforge Redis user documentation
Portal:Toolforge/Admin/Runbooks

This page contains basic resources for developers who want to author and publish runbooks for Toolforge on Wikitech.
==Before you begin==
Permissions: Some processes documented in runbooks for Toolforge will need varying levels of permission to perform. Make sure you have appropriate permissions to complete your task.
==What is a runbook?==
A runbook is a detailed set of instructions that explain how to perform a common task or procedure, so it can be easily and accurately repeated by others. Runbooks are particularly useful for incident response operations. Creating runbooks in response to specific incidents, makes it possible for people to repeat the steps in response to similar incidents. Because they are often used to help people respond to incidents quickly, they should be easy to read and follow, consistent, and accurate.
==When should you use a runbook?==
A runbook should be used whenever a common task or procedure may need to be repeated by mulitple people.
==Tips for building useful runbooks==
*Keep them up-to-date. Revisit the instructions after each incident to make sure they are clear and accurate.
* Keep them simple. These are sets of detailed instructions, and they do not require extra history or context.
** Publish a separate runbook for each issue or incident
*Test them. Make sure the instructions are repeatable by others by asking others to follow them and provide feedback.
* Follow a template. Make sure your runbooks follow a template, so that people can find and update information easily.
===Cloud VPS runbook templates===
Instructions for each issue or alert should have its own page in order to reduce unnecessary information. You can see an example here with the issue Check for VMs leaked by the nova-fullstack test. Note that the page addresses one issue only.
You can find a template/outline for creating runbooks here: Portal:Cloud_VPS/Admin/Runbooks/Runbook_template
== Where to publish runbooks for Toolforge ==
Where to publish Toolforge runbooks
Portal:Toolforge/Admin/Runbooks/(PAGE NAME)
Note: This is a subpage of admin documentation. Some procedures may require advanced admin permissions to complete.
Note: Many runbooks for Toolforge will include procedures that can only be followed by individuals with admin access. In order to avoid confusion and frustration for general users, you should note at the content level when a procedure will require admin permissions.
When an entire runbook requires admin permissions to complete procedures, mark it with the following template:
When runbooks include information for general users and special instructions for admins, mark any instructions (inline) for admins with the following template:
== Existing runbooks ==
==See Also==
*:Category:Runbooks - Wikitech pages marked as runbooks.
*WMCS Admin documentation - Advanced documenation for WMCS administrators.
*Runbooks entry on Wikipedia
Runbooks
Toolforge
 Toolforge
Portal:Toolforge/Admin/Runbooks/BuildsApiDown

This happens when the builds-api pod in the builds-api namespace of tools/toolsbeta k8s cluster is down or can't be reached.
== Error / Incident ==
This happens when the jobs-api pod in the jobs-api namespace of tools/toolsbeta k8s cluster is down or can't be reached.
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
The first most likely step is to ssh to tools/toolsbeta (depending on the project the alert is from) k8s-control servers (i.e toolsbeta-test-k8s-control-4.toolsbeta.eqiad1.wikimedia.cloud). From there you can:
* check that the pods are running:
* You can also check the log of the pod's deployment with kubectl logs -n builds-api deploy/builds-api.
* You can force a restart of all the pods with a rollout kubectl rollout restart -n builds-api deployment/builds-api
* It might also make sense to check if there has been any recent code change and re-deployment attempts. Again a good place to start is by looking at the recent commits in builds-api gitlab repo, or the toolforge-deploy gitlab repo.
* If the pods don't exist or the deployment does not exist, you can try redeploying the builds-api by following the instructions in the toolforge repo (it will do nothing if there's nothing to do).
=== Doing a manual curl for the stats ===
You can try doing a curl directly to the pods for the statisticts, by checking the configuration of prometheus, you'll get the cert, key and url:
Then you can curl directly the pods by name, like:
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If build-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* Builds service docs
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
Portal:Toolforge/Admin/Runbooks/BuildsApiUpMetricUnknown

This happens when prometheus has no data from k8s on the builds-api pod.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmflabs.org/tools/graph?g0.expr=up%7Bjob%3D%22builds-api%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If build-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* Karma UI (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/EnvvarsAdmissionDown

This happens when there's no ready envvars-admission pods in the envvars-admission namespace of tools/toolsbeta k8s cluster, or no information about it (no metrics).
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
The first most likely step is to ssh to tools/toolsbeta (depending on the project the alert is from) k8s-control servers (i.e toolsbeta-test-k8s-control-4.toolsbeta.eqiad1.wikimedia.cloud). From there you can:
* check that the pods are running:
* You can also check the log of the pod's deployment with kubectl logs -n envvars-admission deploy/envvars-admission.
* You can force a restart of all the pods with a rollout kubectl rollout restart -n envvars-admission deployment/envvars-admission
* It might also make sense to check if there has been any recent code change and re-deployment attempts. Again a good place to start is by looking at the recent commits in envvars-admission gitlab repo, or the toolforge-deploy gitlab repo.
* If the pods don't exist or the deployment does not exist, you can try redeploying the envvars-admission by following the instructions in the toolforge repo (it will do nothing if there's nothing to do).
=== Checking if stats are coming in ===
You can go directly to prometheus and check there if there's any stats coming in for that pod (the actual link the alert is using is in the alert UI, under the time, named wmcloud.org):
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If envvars-admission seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* Envvars API docs
* Karma UI (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
Portal:Toolforge/Admin/Runbooks/EnvvarsApiDown

This happens when the envvars-api pod in the envvars-api namespace of tools/toolsbeta k8s cluster is down or can't be reached.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
The first most likely step is to ssh to tools/toolsbeta (depending on the project the alert is from) k8s-control servers (i.e toolsbeta-test-k8s-control-4.toolsbeta.eqiad1.wikimedia.cloud). From there you can:
* check that the pods are running:
* You can also check the log of the pod's deployment with kubectl logs -n envvars-api deploy/envvars-api.
* You can force a restart of all the pods with a rollout kubectl rollout restart -n envvars-api deployment/envvars-api
* It might also make sense to check if there has been any recent code change and re-deployment attempts. Again a good place to start is by looking at the recent commits in envvars-api gitlab repo, or the toolforge-deploy gitlab repo.
* If the pods don't exist or the deployment does not exist, you can try redeploying the envvars-api by following the instructions in the toolforge repo (it will do nothing if there's nothing to do).
=== Doing a manual curl for the stats ===
You can try doing a curl directly to the pods for the statisticts, by checking the configuration of prometheus, you'll get the cert, key and url:
Then you can curl directly the pods by name, like:
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If envvars-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* Envvars API docs
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
Portal:Toolforge/Admin/Runbooks/EnvvarsApiUpMetricUnknown

This happens when prometheus has no data from k8s on the envvars-api pod.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmflabs.org/tools/graph?g0.expr=up%7Bjob%3D%22envvars-api%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If envvars-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* Karma UI (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
*