nyuuzyou/wikis huggingface dataset:
Wikimedia Cloud Services team/EnhancementProposals/Decision record T336376 Find a standard place for all the toolforge-related config files

Origin task: phab:T336376
Date of the decision: 2023-05-26
There was no meeting, decision made in the task.
== Decision taken ==
Option 1 was chosen.
=== Rationale ===
The option of having a config file per-client instead of mixing all the configurations by default on one file makes it easier to manage with debian packages and being able to fallback to the common one keeps the option to reuse open.
== Problem ==
Currently we have a bunch of configuration files under /etc or /etc/toolforge, each with some information for some of the clients/tools that toolforge has, but with no specific pattern to it.
=== Constraints and risks ===
* With the addition of APIs and clients using those APIs, we have new tools needing some configuration (endpoints, auth, etc.), not having a clear place and way to put it make it confusing for new developments and debugging.
* Without a pattern, it might be harder to reuse code between tools.
== Options considered ==
=== Option 1 ===
I propose having the following structure:
```
/etc/toolforge <directory>
|- <client1_name>.yaml
|- <client2_name>.yaml
\- common.yaml
```
where `client1_name` is the name of the cli as installed (toolforge-webservice, toolforge-build, toolforge-jobs, ...) or the generic 'toolforge-cli' one for the top level wrapper.
The config should be read hierarchically, from lower priority to more priority:
* common.yaml
* toolforge-cli.yaml <- not sure this one is needed though, I'd remove it until it's clear it's needed
* <client_name>.yaml
And the config option put at the highest priority level that it can (if two clients need it -> common.yaml).
That can be paired with introducing in the toolforge_weld library a function to load your config, in that way and reused between client.
Pros:
* Clear configuration places and hierarchy
* Common way to load the config
* Changes in one cli config are hard to affect other clis
Cons:
* A bit of work refactoring the current code in puppet and clients
* A bunch of files to look into, but at least all together
=== Option 2
Having one single config file `/etc/toolforge.yaml` with different subsections:
```
common:
 api_gateway_url: "https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/api/v1"
builds:
 builds_endpoint: "builds/v1"
 builder_image: tools-harbor.wmcloud.org/toolforge/heroku-builder-classic:22
jobs:
 api_url: https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/api/v1
 kubeconfig: "~/.kube/config"
...
```
Paired as before for a common function to load the config.
Pros:
* Clear configuration places and hierarchy
* Common way to load the config
* One single file to look for things
Cons:
* Easier to mess up the other clients configs with a bad patch (ex. removing what you shouldn't, etc.)
* A bit of work refactoring the current code in puppet and clients
Wikimedia Cloud Services team/EnhancementProposals/Decision record T336669 How to provide a way to install system dependencies for buildpack-based images

Origin task: phab:T336669
Date of the decision: 2023-06-08
There was no meeting, decision made in the task..
== Decision taken ==
Option 2 - limiting packages to OS provided ones
=== Rationale ===
We decided to go with allowing anyone to use the Apt buildpack because, even though it makes it easier to abuse the system, it also makes it easier to maintain (at least initially) and to develop, and we can always add the extra bits to enforce good usage later if that ever becomes a problem.
We are limiting though packages existing on the official repos of the build/run image (as of writing this it's Ubuntu Jammy).
== Problem ==
Sometimes tools need some system dependencies like:
 imagemagik
 pstools
And currently buildpacks don't allow to install any system packages.
=== Constraints and risks ===
* Users will not be able to use the buildservice and request adding extra packages to the existing toolforge docker images
* Users might not be able to migrate out of the grid without a big refactor
== Options considered ==
== Options ==
=== Option 1 ===
Allowing to install apt packages using upstream apt buildpack (https://github.com/heroku/heroku-buildpack-apt).
This can be done by injecting that buildpack to the generated groups.toml (after detection).
We should put this feature behind an access list, enabled for certain projects after request.
Pros:
* Unblocks any users that need extra dependencies
* Still pushes users to use the recommended way to pull dependencies with buildpacks (pip/composer/bundler/...)
Cons:
* For users that need it, they will have to do an extra request to enable it
* A bit more complicated code-wise (we have to implement some sort of allowlist, we might use it for other things too ex. multistack/custom buildpacks)
=== Option 2 ===
Allowing to install apt packages using upstream apt buildpack (https://github.com/heroku/heroku-buildpack-apt).
This can be done by injecting that buildpack to the generated groups.toml (after detection).
Enabled for everyone.
Pros:
* Unblocks any users that need extra dependencies
* Users that need it have it right away
* No need for allowlist implementation
Cons:
* We enable installing any package from anywhere to everyone potentially welcoming non-opensource code to run on toolforge
* Images will be bigger (we currently have 1G limit per tool set on harbor, so will not be bigger than that).
=== Option 3 ===
Allowing only selected buildpacks for specific libraries, and not the "apt" buildpack.
E.g. this buildpack adds a bunch of additional libraries https://github.com/heroku/heroku-geo-buildpack
Many buildpacks can be found online, and we could create more ourselves.
Pros:
* More control on what people can install in their images
* Might be solved the same way that multistack/specific buildpacks would be
Cons:
* We would need to add those buildpacks individually, as people request them
* Some libraries might not be available as a buildpack, and creating a custom buildpack is possible but not easy
Wikimedia Cloud Services team/EnhancementProposals/Decision record T362224 What to use for toolforge components api task execution

Origin task: phab:T362224
Date of the decision: 2024-07-11
People in the decision meeting (alphabetical order):
* User:Andrew_Bogott
* User:David_Caro
* User:Slavina_Stefanova
* User:Raymond_Ndibe
== Decision taken ==
Option 2 was chosen, with the caveat of moving to option 4 if any blockers are found.
There's open the question for option 2 of how to keep track of the status of the background tasks that we will have to figure out (probably updating the deployment model or similar).
== Problem ==
To be able to manage pipelines in the components api (build 1 + build 2 -> deploy component 1 -> deploy component 2 for example) we need some way to handle execution pipelines.
=== Constraints and risks ===
* The pipelines don't need to be huge, dozens of components would be the biggest
== Options ==
=== Option 1 ===
No asynchronous task processing
Pros:
* Easiest to implement
Cons:
* If the request breaks, the whole pipeline fails
=== Option 2 ===
After-request asynchronous task processing on the same thread (ex. https://fastapi.tiangolo.com/tutorial/background-tasks/)
Pros:
* Easy to implement and setup (no extra components/services/etc. needed)
Cons:
* If the service gets restarted (ex. OOM, moving to a different worker) the pipeline breaks
=== Option 3 ===
Using tekton pipelines
Pros:
* Already used for build service
* Good pipeline support (made for it)
Cons:
* Relatively complex to setup
* Pipelines written in tekton yaml + custom image/shell script
* Needs interfacing with tekton (similar to what we do with builds service)
=== Option 4 ===
Using celery with redis
Pros:
* Very common pattern
* Good pipeline support (made for it)
* Pipelines written in the same code than the service
Cons:
* Needs a redis instance (we should not reuse the user-facing one for security reasons I think)
=== Option N ===
Add you option here!
Wikimedia Cloud Services team/EnhancementProposals/Decision record T363683 kubernetes upgrade workgroup

Origin task: phab:T363683
Date of the decision: 2024-06-06
People in the decision meeting (alphabetical order): There was no meeting, decided in the task
== Decision taken ==
Option 2 was chosen with some comments:
* Create an overview wiki page for the coming upgrades
=== Starting workgroup members (alphabetical order) ===
* User:Arturo_Borrero_Gonzalez
* User:David_Caro
* Raymond Ndibe
* Slavina Stefanova
* User:FNegri
=== Rationale ===
We want to catch up with the k8s versions specially as we can't jump versions yet, so we have to put in the effort to speed up until we get there, and to stay there once we catch up.
== Problem ==
We are several years behind kubernetes upgrades, and in order to catch up, we need to upgrade faster than upstream releases for some time.
=== Constraints and risks ===
* All the problems of running old software (security, bugs, stability, ...)
 
=== Extra info ===
* Current upgrade process documentation - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes
* Latest upgrade task - phab:T362869 (the rest are subtasks of it)
* Upstream releases - https://kubernetes.io/releases/
== Decision record ==
In progress
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T302593_How_do_we_make_decisions
== Options ==
=== Option 1 ===
Do nothing
Pros:
* No extra effort needed
Cons:
* We never catch up
=== Option 2 ===
Create a dedicated opt-in workgroup to focus on monthly Kubernetes upgrades until we catch up (as aim, some upgrades might take more), and continue with regular updates thereafter.
Pros:
* K8s upgrade progress greatly improves
* we spread upgrade knowledge in the team
* we setup a working group that can then take over the regular updates (3/year)
* automation improvement and refinement
Cons:
* Considerable effort sometimes when api deprecations happen to affect us
* Are monthly updates compatible with other work streams?
Wikimedia Cloud Services team/EnhancementProposals/Decision record template

Origin task: phab:T12345
Date of the decision: 2022-02-22
People in the decision meeting (alphabetical order):
* User:Andrew_Bogott
* User:Arturo_Borrero_Gonzalez
* User:Bd808
* User:David_Caro
* User:FNegri
* Raymond Ndibe
* User:Nskaggs
* Slavina Stefanova
* User:Vivian_Rook
* ...
== Decision taken ==
Option 1 was chosen.
=== Rationale ===
Given this and that and compared with this other thing, we chose to lean in favor of this so option 1 gets us closer to the goal.
== Problem ==
Some description of the problem, this might include documents, images, diagrams, links, etc.
=== Constraints and risks ===
* It has to be done before this date because this
* Has to be under this budget because that
* We only have this many people to work on it
* If this event happens and this is not done, this might be the consequences
== Options considered ==
=== Option 1 ===
Some explanation of option 1
For this option we also tested this and that and we found this.
Pros:
* Pro 1
Cons:
* Con 1
...
Wikimedia Cloud Services team/EnhancementProposals/Production Readiness Checklist

A list of items that we should think about before deploying new services/components/changes in production.
This is a working in progress, feel free to contribute.
== Support Lifecycle ==
=== Service Level ===
Document support lifecycle of the solution (roadmap, timeline, expected SLA/SLO, etc)
=== Infrastructure components ===
Document all components that make up the solution and their supported lifecycle for improvements, bug fixes, security fixes, etc.
If we're using LTS versions or not, how long they're supported, by whom, etc.
== Resource Limits ==
All resources consumed by users must have an upper limit to avoid unbounded resource usage.
* CPU/memory/disk usage should have clear limits per user/application
* API endpoints should be rate limited
== Performance Tests ==
It's necessary to understand what levels of performance the solution can provide.
=== User Facing ===
Benchmark and document critical user-facing points in the solution to understand system behavior globally.
=== Building Blocks ===
Benchmark and document low-level building blocks (servers, memory throughput network-attached storage, local disks, network, etc).
== Documentation ==
=== User Documentation ===
* Documentation for most common use cases
* Frequently Asked Question page
* Contact page
=== Admin Documentation ===
* Infrastructure diagrams
* High-level overview of data flow, request timeline, etc
* Runbooks
** How to deploy from scratch
** How to deploy new changes
** How to restart
** Creating/deleting/changing resources (users, apps, projects, etc... any manageable object)
* Troubleshooting steps
== Monitoring ==
=== Servers ===
=== Critical Service Components ===
=== Blackbox Monitoring ===
=== Whitebox Monitoring ===
== Backups ==
== See also ==
* Portal:Cloud VPS/Admin/Deployment confidence checklist
* How to deploy code
* mw:Manual:Pre-commit checklist
* mw:Review queue#Checklist/Process
* mw:Best practices for extensions
* mw:API:Client code/Gold standard
== External Resources ==
* [[[List here other companies' production readiness checklists for comparison. ]]]
Wikimedia Cloud Services team/EnhancementProposals/Toolforge UI/Preload

1) How do you think a graphic interface could improve your experience with Toolforge? What tasks in your current workflow would be the most streamlined if performed in a centralized UI?
2) Of the features mentioned (e.g., tool creation, deployment logs, health monitoring), which are most critical to your workflow?
3) Are there additional features not listed that you believe would significantly enhance the usefulness of Toolforge's web platform?
Wikimedia Cloud Services team/EnhancementProposals/Toolforge container image configuration

This document proposes to establish a single place to configure Toolforge container images.
__TOC__
== Overview ==
Currently there are two separate components that have their own separate copy of the list of container images available for tools (webservice and jobs-framework-cli), and in the future that list might be needed in a growing number of places (for example :phab:T311917). To simplify operations, I propose creating a single location for this configuration.
== Proposed solution ==
=== Format===
The following is a quick sketch of what the data format might like:
{
 "images": {
 "python3.9": {
 "state": "stable",
 "aliases": ["tf-python39"],
 "variants": {
 "jobs-framework": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base"
 },
 "webservice": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-python39-sssd-web"
 }
 }
 },
 "python3.9-pwb": {
 "state": "beta",
 "variants": {
 "jobs-framework": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-python39-sssd-pwb"
 }
 }
 },
 "python3.7": {
 "state": "deprecated",
 "variants": {
 "jobs-framework": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base"
 },
 "webservice": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web"
 }
 }
 },
 "php7.4": {
 "state": "stable",
 "variants": {
 "jobs-framework": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-php74-sssd-base"
 },
 "webservice": {
 "image": "docker-registry.tools.wmflabs.org/toolforge-php74-sssd-web"
 }
 }
 }
 }
}
Note how in the example, python3.9-pwb does not have a webservice variant. A configuration like this means that the specific image is available via the jobs framework but not to webservices.
The aliases field mostly exists to allow non-breaking migration from the current names used by webservice and jobs-framework.
For the final product, we should also create a versioned schema to ensure implementations in different tools are compatible with each other.
=== Location of the file ===
This one I'm not sure. We have multiple options:
# Standalone Git repository, fetched on runtime with a gerrit/gitlab raw url
# Standalone Git repository included as a submodule on build time for the tools that need it
# Protected Wikitech page fetched on runtime
# File on disk provisioned by Puppet
# Kubernetes ConfigMap (managed with our standard Helm tooling?)
# Some sort of a Toolforge metadata API webservice (?)
I personally prefer (5), (1) or (3) in that order, but am up to other options as well.
== Other ==
One more thing I'm considering adding to the spec would be default resources allocated to the pod. The main use case is allocating more memory to Java apps since the JDK tends to be a memory hog.
A future version of this spec could be used when implementing T213641 Design mechanism and process for upgrading Kubernetes container runtimes.
Wikimedia Cloud Services team/Ownership

Service 	AKA 	Phabricator Tag 	Category 	Admin docs 	Alerts	Support level
PAWS 	 	paws 	Tool 	 	 	
Quarry 	 	quarry 	Tool 	 	 	
Superset 	 	superset.wmcloud.org	Tool 	 	 	
Toolhub 	 	toolhub 	Tool 	toolhub.wikimedia.org 	 	
Toolsadmin 	Striker 	striker 	Tool 	toolsadmin.wikimedia.org 	 	
OpenStack Browser 	 	 	Tool 	Tool:Openstack-browser 	 	
K8s status 	 	 	Tool 	 	 	
Tool:Fourohfour 	 	 	Tool 	Tool:Fourohfour 	 	
Toolforge (k8s) 	 	toolforge 	Platform Service 	 	 	
Toolforge Jobs Service 	 	toolforge 	Platform Service 	 	 	
Toolforge Envvars Service 	 	toolforge 	Platform Service 	 	 	
Toolforge Build Service 	 	toolforge 	Platform Service 	 	 	
Toolsdb (Db-as-a-service) 	 	 	Managed Service 	 	 	
Volumes (storage-as-a-service) 	OpenStack Cinder 	 	Managed Service 	 	 	
Databases (Db-as-a-service) 	OpenStack Trove 	 	Managed Service 	Portal:Cloud VPS/Admin/Trove 	 	
Kubernetes (k8s-as-a-service) 	OpenStack Magnum 	 	Managed Service 	Portal:Cloud VPS/Admin/Magnum 	 	
Compute (VM-as-a-service) 	OpenStack Nova 	 	Managed Service 	 	 	
DNS (DNS-as-a-service) 	OpenStack Designate	 	Managed Service 	Portal:Cloud VPS/Admin/DNSPortal:Cloud VPS/Admin/DNS/Designate	 	
Cloud Dashboards 	OpenStack Horizon 	 	Managed Service 	Portal:Cloud VPS/Admin/Horizon 	 	
Puppet/nova integration + Horizon UI	 	 	Managed Service 	 	 	
Nova proxy service 	 	 	Managed Service 	 	 	
VM backups 	 	 	Managed Service 	Portal:Cloud VPS/Admin/Instance backups 	 	
Cinder Backups 	 	 	Managed Service 	 	 	
Openstack 	 	 	Foundational infrastructure	 	 	
metricsinfra 	 	 	Foundational infrastructure	 	 	
Ceph 	 	 	Foundational infrastructure	Portal:Cloud VPS/Admin/Ceph 	 	
read/write NFS 	 	 	Foundational infrastructure	 	 	
=== Support levels ===
Fully supported. These services should be working and available to users at all times; any downtime will be publicly announced and documented. Users can rely on these services as reliable, long-term foundations for their work.
Partial support. Service should be useful for some use cases and running most of the time but some features may be broken or consciously neglected. Outages will be tracked but not necessarily addressed immediately. Approach with caution when relying on these services for major projects.
Little staff support. Service may be experimental, a work in progress, volunteer-maintained, or in the process of being phased out. Outages may be ignored entirely. Do not rely on stability of these services for your work.
== See also ==
* Portal:Cloud VPS/Admin/Skill matrix
Wikimedia Cloud Services team/Python coding

See the newer page for the latest practices.
We write a lot of python, so it seems like that would be the place to introduce some common practices for the WMCS team. In the interest of clean, readable code across the team, it is common practice by development groups to share formatting and other development practices in order to facilitate collaboration. A common example is shared linter settings, like what is used in operations/puppet for puppet-lint.
This is intended as Team Practices, not an overall standard. The movement has mw:Manual:Coding conventions/Python which provides some additional guidance.
==Formatting==
While PEP8 provides most of this, and the flake8 linter enforces it on CI, flake8 is pretty liberal, and formatting is hardly consistent. Using the auto-formatter black provides a more narrow interpretation of PEP8, can be integrated into all of our text editors, and makes the reasoning of the code stand out over the format. Black is configurable if we have disagreements with its defaults.
We have, as a rule, set black to a line length of 80 to comply with PEP8 and make things easier in gerrit reviews. Otherwise, we prefer black's conventions to flake8s when there is a conflict.
== Configuration Data ==
In a repository where a pyproject.toml is appropriate or doesn't clash with other settings, we can use the settings:
[tool.black]
line-length = 80
If that repository is set to check for 79 char lines because of some old flake8 setting, like in tools-webservice, we use that for now.
Category:Cloud Services
Wikimedia DNS/Monitoring

== Wikidough Basic Check ==
==== What does it mean? ====
If this check fails, it means that a Wikidough host is not responding on ports 443 and/or 853 on its IPv4 and/or IPv6 address. This happens when the host is down, depooled, or the dnsdist.service is inactive or has failed.
==== Resolving this message ====
* Head to Icinga and check if the host is up and if there are other checks that are failing (which may indicate a problem with the host itself).
* Is the dnsdist.service active? Check with: systemctl status dnsdist.service.
** If it is has stopped or failed, try restarting it. If it fails again, check the journal output to see why it is failing.
* Since this check queries the host IP and not the anycast IP, it is unlikely that there is an issue with anycast-healthchecker or the bird service.
** Nevertheless, checking the status of the above two services might be worthwhile.
== Service Restart Check ==
==== What does it mean? ====
A failure of this check indicates that the configuration file for either dnsdist or pdns-recursor was changed but the service itself was not restarted. A CRITICAL alert is raised if the time delta between the configuration file change and service restart exceeds 24 hours. 
This check is meant to be a warning alert and does not signify an error in the service. 
==== Resolving this message ====
Please do not perform the steps below without contacting the Traffic team first as restarting any of these services clears the cache.
From a cumin host, restart the service mentioned in the alert on the Wikidough hosts:
 sudo cumin -b 1 -s 5 'A:wikidough' 'systemctl restart dnsdist.service'
or,
 sudo cumin -b 1 -s 5 'A:wikidough' 'systemctl restart pdns-recursor.service'
Wikimedia DNS/Notes

== Troubleshooting ==
=== Block an IP address or multiple IP addresses ===
ATTENTION: This step should only be undertaken for preventing abuse of the service.
==== Add an IP block rule ====
To block a specific IP address from connecting to Wikidough, run the command below replacing the example IP with the actual IP:
 $ sudo cumin 'A:wikidough' "dnsdist -e addAction({'172.217.12.238/32'}, DropAction())"
To block multiple IPs:
 $ sudo cumin 'A:wikidough' "dnsdist -e addAction({'172.217.12.0/24', '1.1.1.1/32'}, DropAction())"
==== Remove an IP block rule ====
On any Wikidough host (let's go with doh1001), find the rule number you want to remove (first column of the output of the command below):
 sukhe@doh1001:~$ sudo dnsdist -e "showRules()"
Note the rule number you want to delete (assuming for example it is 5) and then rmRule it on all hosts:
 $ sudo cumin 'A:wikidough' "dnsdist -e rmRule(5)"
==== Notes ====
* This is a dynamic rule and does not require a restart of the dnsdist service.
* Please specify the IP address in CIDR format. This means that 1.1.1.1/32 is fine, 1.1.1.1 is not.
=== Rate-limit IP addresses ===
==== Add rate-limiting rule ====
To rate-limit IP addresses (maximum number of queries allowed per address), use the MaxQPSIPRule() selector. For example, to set the allowed queries per second to 40 (with an equivalent burst), matching on a /32 IPv4 and /64 IPv6:
 $ sudo cumin 'A:wikidough' "dnsdist -e addAction(MaxQPSIPRule(40, 32, 64), DropAction())"
==== Remove rate-limiting rule ====
To remove a rate-limiting rule, follow the same steps as in removing an IP block rule.