nyuuzyou/wikis huggingface dataset:
Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull

The ToolsNfsAlmostFull alert fires when the Toolforge NFS server is almost out of disk space. This happens surprisingly often as the NFS share has no quotas. When that happens, generally a task similar to is created with administrator work logged on it and a tree of user tasks that we assign to end users to clean up their tool shares or project shares with some advice and assistance where possible.
== Error / Incident ==
The Toolforge NFS server is almost out of disk space. This generally means that some space needs to be freed up. As of 2024-01-03 the nfs server is tools-nfs-2.tools.eqiad1.wikimedia.cloud.
Note that this alert comes in multiple severity levels, a warning alert means that there's much more space available than for a critical or a page alert.
Admin actions include, but are not limited to:
* Truncate automatically-created *.out and *.err files that were created by job file logs. For example:
 sudo ionice -c 3 nice -19 find /srv/tools -type f -size +10M -name "*.err" -exec truncate -s 1M {} \;
* Log files generated by the webservice command such as access.log and error.log can be treated similarly.
* Other files should probably be checked with the user before deleting unless the situation is very urgent (usually asking the user in the phabricator task is enough).
* If a service is consistently filling up NFS volumes, and users cannot be reached, it could be shut down as a danger to the overall service. We should make our best effort to avoid needing to do that, of course.
* Remove older files (or, if necessary, all files) from project/.shared/cache -- some tools (mostly cewbot) use that for temporary storage and often fail to clean up afterwards.
The #Debugging section has some tips to find what's filling up the volume.
== Debugging ==
=== Try what was done last time ===
If the alert fires after a very short time (about a week or so) after the last time cleanup was done, it is usually caused by the same thing as the last time. Look at the task for that cleanup and look what was done there. Cleanup those again and nudge those maintainers.
=== Locate disk hogs ===
# ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" | sort -h > tools_large_files_$(date +%Y%m%d).txt
This will take a few hours to complete.
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Data_Services/Admin/Shared_storage
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState

This happens when the primary ToolsDB instance is down, or is up but in read-only mode.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
== Debugging ==
=== General cluster overview ===
You can run the following cookbook to get a cluster overview:
dcaro@urcuchillay$ wmcs-cookbooks wmcs.toolforge.toolsdb.get_cluster_status --cluster tools
=== Checking the systemd unit status ===
SSH to the instance and check the systemd status for mariadb.service
$ ssh tools-db-1.tools.eqiad1.wikimedia.cloud
fnegri@tools-db-1:~$ sudo systemctl status mariadb.service
If SSH does not work (for example because of phab:T349681) you can use virsh console: find the "instance name" and "host" in Horizon, then SSH to the cloudvirt host, and run virsh console {instance name}.
== Common issues ==
Add new issues here when you encounter them!
=== MariaDB process killed by OOM killer ===
If this is the case, you usually see a log message like the following one in the mariadb logs:
$ ssh tools-db-1.tools.eqiad1.wikimedia.cloud
fnegri@tools-db-1:~$ sudo journalctl -u mariadb |grep -i kill
Oct 24 09:23:39 tools-db-1 systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Oct 24 09:23:39 tools-db-1 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Oct 24 09:23:39 tools-db-1 systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Sometimes the mariadb logs only include "Main process exited", without any mention to OOM, but you can verify if the process was killed by the OOM killer looking at dmesg:
fnegri@tools-db-1:~$ sudo dmesg -T |grep Killed
[Tue Oct 24 09:19:23 2023] Out of memory: Killed process 2437 (mysqld) total-vm:64835688kB, anon-rss:64103256kB, file-rss:0kB, shmem-rss:0kB, UID:497 pgtables:126460kB oom_score_adj:-600
Check if systemd restarted the "mariadb.service" automatically with systemctl status mariadb, otherwise run systemctl start mariadb.
Finally, set the server to read-write, as it is configured to start in read-only mode for extra safety:
$ sudo mariadb
MariaDB [(none)]> SET GLOBAL read_only=OFF;
== Related information ==
* Portal:Data_Services/Admin/Toolsdb
* Toolforge admin docs
== Support contacts ==
The main discussion channel for this alert is the #wikimedia-cloud-admin in IRC.
If the situation is not clear or you need additional help, you can also contact the Data Persistence team (#wikimedia-data-persistence on IRC).
== Old incidents ==
Add any incident tasks here!
* phab:T353093
* phab:T349695
* phab:T344298
Portal:Toolforge/Admin/Runbooks/k8s-haproxy

These tests are defined here: https://gerrit.wikimedia.org/g/operations/puppet/+/c85d36363c69c1d01092726af556812138475857/modules/profile/manifests/toolforge/k8s/haproxy.pp#56
These checks verify the behavior of the toolforge web proxy. There are two different endpoints tested:
=== https://admin.toolforge.org/healthz ===
This tests a known, minimal k8s webservice that when working properly returns 'ok' at the healthz route.
When this alert fires it is probably because the tool has crashed. Restart on a toolforge bastion with:
 $ sudo su -
 # become admin
 # webservice stop # and now wait a minute, just doing a quick restart seems insufficient
 # webservice start
 # tail -f error.log # should be fairly quiet
=== https://this-tool-does-not-exist.toolforge.org/ ===
As explained by the endpoint, this is a check of the proxy behavior when accessing a non-existent tool. It ought fall through to the 404 handler and report a useful "this tool does not exist" message.
As of 2022-09-02 this test is known to be flaky due to the 404 handler being a bit broken. A (apparently insufficient) attempt to fix this is https://gerrit.wikimedia.org/r/c/operations/puppet/+/826779/
Portal:Toolforge/Admin/SSL certificates

This page contains information on SSL certificates for Toolforge.
For LE certificates, we use acme-chief.
== Usage ==
There is currently 1 certificate in use.
* *.toolforge.org (also know as star.toolforge.org) which also includes *.tools.wmflabs.org.
=== *.toolforge.org / *.tools.wmflabs.org ===
This certificate was introduced in T235252
This certificate is in use by Toolforge web proxies as well as the k8s master and docker registry, and is physically deployed in several servers:
* tools-proxy-*.tools.eqiad1.wikimedia.cloud
*tools-k8s-master-*.tools.eqiad1.wikimedia.cloud
*tools-docker-registry-*.tools.eqiad1.wikimedia.cloud
The private key is generated in the acme-chief servers (tools-acme-chief-*). These servers are responsible for requesting new certificates, renewal, etc. They contain credentials that can be used to make the DNS changes necessary to prove domain ownership.
Example URL using this SSL certificate: https://tools.wmflabs.org/
=== See also ===
* Main Toolforge admin docs
Portal:Toolforge/Admin/Striker

toolsadmin.wikimedia.org is where maintainers of Toolforge tools can manage their tools. The software that runs this interface is codenamed Striker.
== Features ==
* Wikimedia developer account creation wizard
* SSH public key management
* Associate Wikimedia developer account, Wikimedia SUL account, and Phabricator account
* Create new tool accounts
* Create/update metadata describing a tool. This metadata is a superset of the data needed for Hay's directory and published there.
* Create Git repositories in GitLab associated with a tool.
* Create Phabricator projects associated with a tool.
== Service ==
Striker is an Django application for managing Toolforge tools. A blacksmith's striker is an assistant to a blacksmith whose job is to swing a heavy hammer at the direction of the blacksmith. Similarly, Striker performs various tasks at the direction of a tool maintainer.
=== Production deployment ===
Toolsadmin.wikimedia.org is provisioned via the role::striker::web Puppet class on cloudweb1003 and cloudweb1004. This class sets up Docker and a service to run Striker as a Docker container. The application is configured by environment variables that are provisioned by Puppet and populated with data from hiera. The application's local database is hosted on the m5 section. The application also connects to the LDAP server for authn/authz and data lookup, Phabricator's public API, metawiki for OAuth authentication, and OpenStack's keystone API.
=== Testing deployment ===
https://striker.wmcloud.org in the Striker project hosts a testing deployment that is used for beta testing. This deployment is managed with docker-compose similarly to the local development environment, but uses a production ready container for the striker service itself.
=== Local development ===
The local development environment uses docker-compose and GNU Make to manage a container hosting the project. Additional containers are provisioned to provide:
* a MariaDB database service
* an OpenLDAP service
* a Phabricator instance
* a Keystone service
* a MediaWiki wiki acting as a meta.wikimedia.org substitute
* a MediaWiki wiki acting as a wikitech.wikimedia.org substitute
See contrib/docker/README.md for additional manual steps needed once the initial startup is complete.
=== Contributing ===
Submit patches to the labs/striker gerrit project. Tests and linting can be done locally using tox.
== See also ==
* Frequently (un)asked questions
* Deployment log
* Build and deploy process
* Tool Labs vision
* Phab:T136256 (Deploy "Striker" Tool Labs console to WMF production)
* Tool creation added to toolsadmin.wikimedia.org (28 August 2017 blog post)
* Nova Resource:Striker
* The foxtrot-ldap page, fake LDAP for Toolforge 
== External link ==
* https://toolsadmin.wikimedia.org/
* https://gerrit.wikimedia.org/g/labs/striker
** https://github.com/wikimedia/labs-striker
Category:Services
Striker
Portal:Toolforge/Admin/Striker/FAQ

Frequently (un)asked questions about using the Toolforge admin console.
== Configure Diffusion to mirror a GitHub repo ==
# Create Diffusion repository using toolsadmin.wikimedia.org
# Go to new repository on Phabricator by following the "browse repository on Phabricator" link
# Click "Manage Repository" button
# Click "URIs" section in left column
# Click "New URI" button
#* URI: enter the https URL to your "https://github.com/{OWNER}/{REPO}.git"
#* I/O Type: "Observe: Copy from a remote."
#* Display Type: "Visible: Show as a clone URI."
# Click "Create Repository URI" button
Note: Currently only GitHub (and Gerrit) repositories are supported for mirroring, see phab:T143969 for discussion about adding more hosts.
== Mirror a Diffusion repo to GitHub ==
# Create a repo on GitHub
# Grant a push bot write permissions to the GitHub repo.
#* This will be easiest if the repo is in the wikimedia or toolforge organization. You should not store credentials for your personal GitHub account in Phabricator.
#* If the repo is in the toolforge organization, go to the "Settings" > "Collaborators & teams" tab and add the "Push bots" team with write permissions. This will grant the Toolforge push bot user push access.
#* If the repo is in the wikimedia organization, the Phabricator bot user will automatically have push access.
# Go to the repository on Phabricator
# Click the "Actions" > "Manage Repository" menu item
# Click "URIs" section in left column
# Click "New URI" button
#* URI: enter the https URL to your "https://github.com/{OWNER}/{REPO}.git"
#* I/O Type: "Mirror: Push a copy to a remote."
#* Display Type: "Visible: Show as a clone URI." or "Hidden: Do not show a clone URI."
# Click "Create Repository URI" button
# Click "Set Credential" menu item
# Choose the credentials for a push bot that can write to your GitHub repo
#* toolforge org: "K33 GitHub - toolforge-push - password (2020-01-22)"
#* wikimedia org: "K32 wmfphab github token"
Portal:Toolforge/Admin/System Overview

This page contains a system overview of Toolforge from the engineering point of view. Find here high level information of the different bits and pieces that make Toolforge what it is.
= Summary =
Toolforge is a Platform as a Service. It is meant to offer pre-made and managed computing facilities to its users.
At the core of its service, there is a computing engine. Well, as of this writing there are two, Grid Engine (to be deprecated) and Kubernetes.
Users log in into Toolforge via SSH and deploy tools (mainly webservices and jobs) into one of the computing engines.
The About Toolforge user-oriented page contains more information on the basic aspects of the service.
All pieces of Toolforge are deployed inside a Cloud VPS project (or tenant) called tools. The staging/development project is called toolsbeta.
= Use cases =
This section captures the most common use cases that Toolforge supports.
=== webservices ===
TODO.
=== jobs ===
TODO.
= They key components =
Information on the several key pieces that make Toolforge what it is.
=== LDAP ===
This is where user accounts are stored.
=== Bastions ===
Bastion servers are used by toolforge user as the entry point for the service. They are Cloud VPS virtual machines that allow SSH connections from the internet. Usual FQDNs are login.toolforge.org and dev.toolforge.org.
A strong shared-system policy is enforced to ensure that users don't easily hog system resources (like CPU or RAM).
=== NFS ===
As of today, NFS is at the core of Toolforge. We use it for many purposes, including:
* to store toolforge user's home directories (remember, we allow SSH to the bastions)
* to store toolforge user's source code and distribute it to the computing backends (grid engine or kubernetes)
* to store toolforge user's credentials, like kubernetes TLS certs or wiki replicas database account credentials
* to store toolforge user's logs, generated at runtime from webservices and jobs
* to distribute wiki dumps
* to store toolforge tool temp files and similar
Please note that NFS is an old technology, and we have been working for years in reducing dependency on it with the ultimate goal of stop using NFS at all in the future.
=== Kubernetes ===
We have a particular Kubernetes deployment, which is described in its own page.
There are a number of custom components deployed into Toolforge kubernetes, see Portal:Toolforge/Admin/Kubernetes/Custom_components.
==== RBAC and security ====
See main page: Portal:Toolforge/Admin/Kubernetes/RBAC_and_PSP.
==== jobs framework ====
See main page: Portal:Toolforge/Admin/Kubernetes/Jobs_framework.
=== Network ===
Toolforge has a particular network setup, see Portal:Toolforge/Admin/Kubernetes/Networking_and_ingress.
See also Portal:Toolforge/Admin/Dynamicproxy.
= Other components =
There are a number of components that are part of the Toolforge offering beyond the key building blocks.
=== striker ===
The entry point for account creation.
TODO.
=== databases ===
TODO. Talk here about.
* toolsdb
* wiki replicas
= Some history =
TODO:
* toolserver
* grid-only system
= See also =
TODO.
Portal:Toolforge/Admin/Toolforge-sync-meeting

= Toolforge sync: plans =
Notes from meeting's etherpad we had on 2018-11-07 to sync on Toolforge plans.
Meeting attendants:
* brooke
* gtirloni
* bryan
* andrew
* arturo
== plans & TODOs ==
* we can't mix both grids!
* misctools dependency on mariadb on stretch !!
* building a grid in toolsbeta cloudvps project, stretch with son of grid engine
* stretch test host with all packages and libraries to do early testing by tool owners?
** probably worth waiting until we have a minimal son of grid engine skeleton
* try a k8s worker node from toolsbeta in eqiad1-r to see if that can work cross-region etc
* plans for abandoned projects? contact James, open phab tasks
=== speculative timeline ===
* 2018-11-07: toolforge sync meeting
* 2018-11-15: have all stretch packaging done by this time (misctools, etc)
* 2018-11-30: skeleton son of grid engine grid working in toolsbeta
* 2018-12-01: email server rebuilt as stretch by this time
* 2018-12-21: skeleton son of grid engine grid working in tools
* 2019-01-01: start sending emails and opening phab tasks for tools users.
* 2018-01-15: figure out actual timeline and deadline for the rest of the process
* 2019-04-30: trusty deprecation deadline
== info on concrete nodes ==
Components:
* Grid engine
* cron server (grid submit host)
* exec nodes
** the new libraries are going to be very interesting +1
* k8s cluster --- try to keep this in jessie? apparently not related to grid engine at all.
** test stretch bastion
** we may have to move k8s to eqiad1-r prior to further operations on the new toolforge
* mail server (removing grid dependency) --- once the refactor is complete, we could be able to just rebuild in stretch
* redis instance? --- aparently not a grid submit host
* services nodes (aptly, bigbrother, updatetools, etc) --- ongoing refactor. Grid submit host bc bigbrother
** Bryan thinks we can "deprecate" bigbrother by not setting up the service on the new Stretch grid +1
* bastions
* checker (??)
* clushmaster (already jessie)
* elastic (already jessie)
* flannel (already jessie)
* paws (already stretch)
* ...
Portal:Toolforge/Admin/Toolforge roots and Toolforge admins

The terms Toolforge root and Toolforge admin are used somewhat interchangeably to describe users who do administrative work in Toolforge.
These users usually have all the set of elevated permissions listed in this page, but they must be all granted individually so it's possible for someone to only have a subset of these.
Ideally, we should find a way to add these permissions automatically by adding people to a single group in LDAP or elsewhere.
=="tools" project in Cloud VPS==
Adding a user with "member" privileges to the tools project in Cloud VPS.
This allows a user to add and delete other users from the Toolforge project.
There is also a separate toolsbeta project which is used by the Toolforge staging environment.
==sudo policy "roots"==
This allows a user to use sudo to become root on Toolforge instances.
Members of the admin tool in Toolforge can log into infrastructure instances and perform tasks as the admin tool.
An equivalent toolsbeta.admin tool exists on the "toolsbeta" staging deployment. Note that for toolsbeta you will need to add it through the command line using modify-ldap-group toolsbeta.admin from mwmaint1002.
==Gerrit group "toollabs-trusted"==
Users that are part of this group in Gerrit can +2 changes in Gerrit repositories related to Toolforge.
==GitLab group repos/cloud/toolforge==
Users that are part of this group in Gitlab can push changes to Gitlab repositories related to Toolforge.
==See also==
* Cloud roots and Cloud admins
Portal:Toolforge/Admin/Toolschecker

Toolschecker is a Flask application that runs various active checks on Toolforge and Cloud VPS infrastructure in response to HTTP requests. Each check is exposed as a separate URL on the checker.tools.wmflabs.org host. These URLs are monitored by Icinga for alerting purposes (see "checker.tools.wmflabs.org").
Each endpoint is served by a different uwsgi service behind an nginx running on the VM (currently tools-checker-04.tools.eqiad1.wikimedia.cloud). Configured by puppet.
== Servers ==
* tools-checker-04.tools.eqiad1.wikimedia.cloud
This list is defined in the profile::toolforge::checker_hosts Hiera key and is used in configuring the ferm rules for Toolforge's Kubernetes etcd clusters.
== Checks ==
=== /dns/private ===
=== /etcd/k8s ===
=== /ldap ===
=== /nfs/dumps ===
=== /nfs/home ===
=== /redis ===
=== /self ===
Portal:Toolforge/Admin/emergency guides/irc bot deployment

emergency deployment of grrrit-wm and wikibugs on tools-webproxy-01 during nfs outage
1. Find a host that is still running and can access
* the internet (both)
* tools-redis (for wikibugs)
2. open up a screen
3. git clone https://github.com/wikimedia/labs-tools-grrrit
4. cd labs-tools-grrrit, apply (we don't need to identify, and ircnotify is also not importatnt)
diff --git a/config.yaml b/config.yaml
index a2a1d6b..4301dd4 100644
--- a/config.yaml
+++ b/config.yaml
@@ -1,6 +1,8 @@
 nick: grrrit-wm
+password: blah
 server: irc.freenode.net
 userName: lolrrit
+ircnotify-pass: none
 realName: GRRRit the Terrible
 default-channel: "#wikimedia-dev"
 firehose-channel: "#mediawiki-feed"
5. you may have to strip channels from the channel list (e.g. ##wmt)
6. 
sudo apt-get install nodejs
npm install irc redis ssh2 yaml
cd src
nodejs relay.js
7. in another screen tab, cd ~
8. git clone https://github.com/wikimedia/labs-tools-wikibugs2
9. copy config.json from https://phabricator.wikimedia.org/P872
10. python redis2irc.py
11. in another tab, maybe python wikibugs.py
Portal:Toolforge/Admin/emergency guides/single tool webservice

During the june 2015 NFS outage, we did an emergency deployment of the demw wiki loves earth jury tool on toolsbeta. This is a step-for-step guide what we did at that point, although some intermediate steps might be missing.
1) Find a suitable host. 
* We chose a host on Toolforge because it had on-line instances that could be deleted safely afterwards (spinning up new instances took an hour or more). 
* We chose a host with a suitable Security Group setup -- with port 8080 open to 10.0.0.0/8.
* We chose a host with a running web service setup so we could have an environment akin to the existing Toolforge one
 
Best host: toolsbeta-webgrid-04
 
2) Disable NFS and restart host (human intervention by Yuvi)
3) create basic structure in /data/project:
 /data/project/jury
 /data/project/jury/public_html (containing salvaged code)
 /data/project/jury/replica.my.cnf
4) hack around the safeguards in lighttpd-starter 
* lighttpd-starter checks whether the server is allowed to start. It also tries to write in places where we don't have permission to write
--- /usr/local/bin/lighttpd-starter 2015-04-05 06:13:33.878601295 +0000
+++ lighttpd-starter 2015-07-03 21:45:21.945025980 +0000
@@ -23,14 +23,10 @@
 
 tool="$1"
 port="$2"
-prefix=$(/bin/cat /etc/wmflabs-project)
+prefix=tools #$(/bin/cat /etc/wmflabs-project)
 user="$prefix.$tool"
-home=$(getent passwd $user | cut -d : -f 6 | sed -e's/\/$//')
-if [ "$(getent group $user | cut -d : -f 1)" !="$user" ]; then
- echo "$0: $1does not appear to be a tool" >&2
- exit 1
-fi
-spool="/var/run/lighttpd"
+home=/data/project/jury #$(getent passwd $user | cut -d :-f 6 | sed -e 's/\/$//')
+spool="/data/project/jury/lighttpd"
 runbase="$spool/$tool"
 
 if ["$home" = "" -o ! -d "$home/public_html" ]; then
 
5) start the webserver in a screen
sudo su tools.jury
script /dev/null
screen
./lighttpd-starter jury 8080
 
6) Test if the webservice is running (ssh tunnel)
 
7) Disable webservice on Toolforge (requires root login)
* stop webservice, make sure it stays stopped (no service.manifest, but that's hard without NFS)
 
8) set up proxy to redirect to temporary host
* ssh to primary webproxy
* redis-cli
* hset prefix:<toolname> '.*' 'http://hostname:port'
9) test on tools.wmflabs.org
Portal:Toolforge/Admin/emergency guides/toolforge down notification

How to deploy the emergency Toolforge error page (503).
1) Set up Hiera: Hiera:Tools
* "dynamicproxy::error_enabled": true
* "dynamicproxy::error_details": <details you want to add to the page>
* "dynamicproxy::error_message": <to override the standard 'this server is having technical issues' notice>
You can use HTML for this
2) puppet apply -tv on tools-webproxy-XX
3) if this fails:
3a) edit /etc/nginx/sites-available/proxy manually:
* search for "# To enable the 'This server is experiencing technical difficulties' message:"
* remove comment in code line below
* remove rest of file
3b) edit /var/www-error/errorpage.html to fill in information
4) if you need to serve other files from the webserver, this is possible. Put them in /var/www-error and they should just work. Non-existent URLs get a 503 instead of a 404.