nyuuzyou/wikis huggingface dataset:
Portal:Toolforge/Admin/local packages

Created using PAWS and https://public-paws.wmcloud.org/User:BDavis_(WMF)/List%20tools-local%20packages.ipynb
package 	jessie	stretch	buster 	bullseye
jobutils 	 	1.43 	1.43 	
kubernetes-node 	 	 	1.4.6-7 	
logster 	 	 	0.0.10-2~jessie1	
misctools 	 	1.43 	1.43 	
pack 	 	 	0.14.2-1 	
toolforge-jobs-framework-cli	 	4 	7 	4
toolforge-webservice 	0.82 	0.82 	0.82 	0.82
toollabs-webservice 	0.75 	0.77 	0.77 	0.77
tools-manifest 	 	0.24 	0.24
Portal:Toolforge/Admin/replagstats

#!/usr/bin/perl -w
use encoding 'utf8';
use strict;
use warnings;
use DBI;
use Socket;
# Database servers with most active databases.
my %Servers = ('s1' => 'enwiki_p',
 's2' => 'itwiki_p',
 's3' => 'mlwiki_p',
 's4' => 'commonswiki_p',
 's5' => 'dewiki_p',
 's6' => 'ruwiki_p',
 's7' => 'eswiki_p');
# Loop forever.
while (1) {
 # Iterate over all servers.
 foreach my $Server (keys (%Servers)) {
 # Connect to server.
 my $DB = DBI->connect ("DBI:mysql:database=$Servers{$Server};host=$Server.labsdb;mysql_read_default_file=" . (getpwuid ($<)) [7] . "/replica.my.cnf",
 undef,
 undef);
 if (!defined ($DB)) {
 warn ("$0: Could not connect to $Server: " . DBI->errstr ());
 next;
 }
 # Retrieve replication lag.
 my $r = $DB->selectall_arrayref ("SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(rc_timestamp) FROM recentchanges ORDER BY rc_timestamp DESC LIMIT 1;");
 if (!defined ($r)) {
 warn ("$0: Could not retrieve replication lag for $Server: " . $DB->errstr ());
 $DB->disconnect ();
 next;
 }
 # Disconnect from server.
 $DB->disconnect ();
 }
 # Wait a minute.
 sleep (60);
}
Portal:Toolforge/Admin/toolhistory

#!/bin/bash
# Dump and commit the list of servicegroups to the repository at
# ~/var/lib/git/servicegroups hourly for backup and history.
# Abort on error.
set -e
# Loop forever.
while true; do
 TMPWC="$(mktemp -d)"
 git clone ~/var/lib/git/servicegroups "$TMPWC"
 cd "$TMPWC"
 ldaplist -l servicegroups > servicegroups.ldif
 git add servicegroups.ldif
 if git commit -m Snapshot; then
 git push
 fi
 cd
 rm -Rf "$TMPWC"
 # Sleep for an hour.
 sleep 1h
done
Portal:Toolforge/Contributing

If you are looking to contribute to the Toolforge project itself, you are welcome!
== Where is the source code? ==
The source code for Toolforge is complex and is composed of several different components.
* Most of the code lives in gitlab:repos/cloud/toolforge/
* Some other code pieces are located in gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production
== Work on a single task, fix a bug, add/improve some docs or implement a small feature ==
You can start by checking the current open issues on phabricator and picking one that sounds interesting.
== Work with us on specific efforts ==
For more continued contributions, you can find more information and ways to contribute under the ongoing efforts page. A list of subpages:
== Propose something new or improve an existing system ==
If you have new ideas for improvements or new projects proposals, you can follow the process specified by the Cloud Services Proposals Phabricator project.
== Join Toolforge Workgroup meetings ==
Period, informal engineering meetings focused on Toolforge, to unblock decisions or to get everyone on the same page. Details at Portal:Toolforge/Admin/Workgroup.
== Get in touch! ==
If you have any questions or want to learn more, get in touch with us:
Contributing
Category:Documentation
Portal:Toolforge/Ongoing Efforts/Toolforge Upgrade Workgroup

= Toolforge Upgrade Workgroup =
 About
 Upgrades Overview
 Feedback
== About ==
The Toolforge Upgrade Workgroup is a focused workgroup to tackle the upgrades of toolforge kubernetes until we catch up with the running version (whatever that is when we catch up).
It's formed by some WMCS team members (as of writing this), but any volunteer is welcome to participate. Note that you are expected to exclusively dedicate some time to it.
Currently the workgroup syncs on-task/wiki and on the wider toolforge workgroup check-in meetings.
== Background ==
Upgrades for toolforge have been "best effort" for a long time, and we found ourselves unable to catch up with upstream cadence, so we decided to create a focused workgroup to catch up and build traction to remain up to date.
== Current members ==
In alphabetical order:
* User:Arturo_Borrero_Gonzalez
* User:David_Caro
* User:FNegri
* Raymond Ndibe
* Slavina Stefanova
Category:Toolforge admin
Category:Documentation
Portal:Toolforge/Ongoing Efforts/Toolforge Upgrade Workgroup/Upgrades Overview

= Toolforge Upgrade Workgroup =
 About
 Upgrades Overview
 Feedback
== Upgrades Overview ==
=== Extra info ===
* Current upgrade process documentation - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes
* Upstream releases - https://kubernetes.io/releases/
* Fancy deprecation UI - https://kube-api.ninja/
=== Upcoming upgrades ===
Pool of members (for easy copy-paste):
* User:Arturo_Borrero_Gonzalez
* User:David_Caro
* User:FNegri
* User:Raymond_Ndibe
* Slavina Stefanova
Version	Task 	Changelog	Big changes 	Assignees 	Expected timeline
1.32 	phab:T379047	Changelog	TBD 	User:Raymond Ndibe and User:FNegri 	TBD
1.31 	phab:T372697	Changelog	TBD 	User:Arturo Borrero Gonzalez and User:Raymond Ndibe	TBD
1.30 	phab:T362869	Changelog	Will move back to PAVs and drop kyverno.	User:David_Caro and User:Arturo Borrero Gonzalez 	March 2025
1.29 	phab:T362868	Changelog	TBD 	User:FNegri and User:David Caro 	February 2025
Category:Toolforge admin
Category:Documentation
Portal:Toolforge/Ongoing Efforts/Toolforge Workgroup

= Toolforge Workgroup =
 About
 Project Overview
 Reports
 Feedback
== About ==
The Toolforge Workgroup is a focused group dedicated to maintaining and evolving the Toolforge service.
It is currently formed by some WMCS team members but any volunteer is welcome to participate, as long as you are prepared to commit dedicated time and join regular sync meetings. For smaller contributions, contact us via IRC or Phabricator for guidance, or feel free to choose any task you'd like to tackle.
== Current members (Q1 2024-2025) ==
This is meant to be updated when someone wants to participate/stop participating during the current quarter (inverse alphabetical order):
* User:Raymond_Ndibe
* User:Slavina_Stefanova
* User:Komla Sapaty
* User:FNegri
* User:David_Caro
* User:Arturo_Borrero_Gonzalez
* User:Andrew_Bogott
== Background ==
We started a workgroup for the Toolforge Build Service that worked pretty well, and at the same time, we started having toolforge admins periodical meetings to sync between ourselves.
Some time after, the Build Service evolved and it started needing and affecting more and more other Toolforge components, to the point that it was not about just the Build Service anymore. At that point we decided to widen the scope of the workgroup to the whole of Toolforge.
There's still the need to discuss and communicate between Toolforge admins what are the next steps and changes needed for Toolforge, so the toolforge admins periodical meetings will still happen, but there will be now a group of people coordinating to plan and implement the outcomes (this effort).
Category:Toolforge admin
Category:Documentation
Portal:Toolforge/Ongoing Efforts/Toolforge Workgroup/Overview

= Toolforge Workgroup =
 About
 Project Overview
 Reports
 Feedback
== Project Overview ==
See the last iteration of the toolforge phabricator board.
== Project organization ==
The project is currently being led by the Wikimedia Cloud Services team and welcomes technical contributors! 
The team is working in two-week iterations, with pre-selected tasks that are estimated depending on the complexity, effort, and time they might take. Iterations can be tracked on the Phabricator board.
Current iteration: Iteration 12
== Milestones ==
* phab:T194332 [Epic,builds-api,orchestrator,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
* phab:T356262 [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs
* phab:T348755 [jobs-api,webservice] Run webservices via the jobs framework
==== How to read user stories ====
User stories help facilitate expectations and communication within the team and to the users. Each user story for Toolforge Build Service is accompanied by a "When do we know it's done?" section that helps list the different scenarios that have to be taken into account in order to elaborate the consequent subtasks the team will be working on to accomplish them.
=== Future milestones ===
TBD
Category:Toolforge admin
Category:Documentation
Portal:Toolforge/Previous Efforts/Toolforge Build Service

= Toolforge Build Service Project =
NOTE: This effort has been merged into Portal:Toolforge/Ongoing_Efforts/Toolforge_Workgroup
 About
 Project Overview
 Documentation
 Reports
 Feedback
== About ==
The Toolforge Build Service project aims to enable Toolforge users to deploy and update their projects without relying on the underlying implementation. It will achieve this goal by technically decoupling service and implementation providing a more stable interface.
For users, this means setting up a project will be simpler, faster, and require no in-depth, technical knowledge about how our services work and will experience less interruptions and migrations due to the infrastructure evolution.
For the WMCS team, the project will help decouple Toolforge from NFS, enabling them to deploy container images with user code directly instead of sharing that code through NFS. This in turn will get the team closer to making the service independent from NFS and its problems (scaling, maintenance, sensitivity to errors).
If the goal is reached, users will be able to:
* Push their code using git and see that code being deployed on Toolforge.
* Set this up easily.
* Avoid needing any knowledge of how it happens (currently Kubernetes and custom Toolforge tweaks).
* Get their own code built using widely adopted industry standards, like buildpacks.
== Background ==
See Toolforge build to deploy and Toolforge Buildpack Implementation for more information on the past efforts for the project.
== Report a bug ==
If you're encountering any issues, you can use this Phabricator template to file a bug report. Just make sure to be as detailed as possible in your description, including any error screenshots that may help explain the problem.
Category:Toolforge admin
Category:Documentation
Portal:Toolforge/Previous Efforts/Toolforge Build Service/Documentation

= Toolforge Build Service Project =
 About
 Project Overview
 Documentation
 Reports
 Feedback
== Documentation ==
=== Internal ===
* Phabricator task
* Enhancement proposal: Toolforge push to deploy
* Enhancement proposal: Toolforge Buildpack Implementation
==== Archived documents/pages ====
* 2022-12 Toolforge build service - user stories brainstorm
* 2022-12 Toolforge build service - Beta release user stories
=== External ===
* Generic buildpack info
* Paketo buildpacks (sample apps using them)
* Google cloud buildpacks
Category:Toolforge admin
Category:Documentation
Portal:Toolforge/Previous Efforts/Toolforge Build Service/Reports

= Toolforge Build Service Project =
 About
 Project Overview
 Documentation
 Reports
 Feedback
== Reports ==
Reports include both decision records and progress reports on the implementation of Toolforge Build Service (beta). These reports are intended to keep team members and interested community members updated on progress of the project.
=== Decision records ===
===== Ongoing =====
* Decision request – Toolforge (re)architecture - Ongoing
===== Decided =====
* Decision request - How to allow installing system libraries on buildpack-based images - Decided on 2023-06-08
* Decision request - Find a standard place for all the toolforge-related config files - Decided on 2023-05-26
* Decision request - Toolforge envvars/secrets service - Decided on 2023-05-26
* Decision request - What buildpacks to allow and include for toolforge build service beta - Decided on 2023-03-10
* Decision request - Choose a lang for the toolforge build service API - Decided on 2023-02-23
* Decision request - Toolforge build service to move to an API design - Decided on 2023-02-06
* Pre-create harbor namespaces - Decided on 2022-12-15
* How to manage quotas in Toolforge Build Service - Decided on 2022-05-04
=== Progress reports ===
Read the latest sync meeting notes.
==== 2023 ====
* 2023-09-13: Focus on the beta rollout, improve user experience and add metrics to the services
* 2023-08-29: Focus on the second beta rollout, database credentials on envvars and harbor stability
* 2023-08-08: Focus on getting the database credentials on the envvars service to do the second beta rollout
==== 2022 ====
* 2022-09-27: We started working on the Toolforge Build Service [beta] project!
Category:Toolforge admin
Category:Documentation
Portal:Wikitech

The production part of this wiki, also known as Wikitech, is for technical documentation of internal infrastructure at the Wikimedia Foundation.
For technical documentation of the MediaWiki software, see mediawiki.org.
Guides 	Getting help from SRE · Get involved · Runbooks · How-Tos · Profiling · Puppet coding · Service Level Objectives
Servers 	Server Lifecycle · LDAP · Virtualization · PartMan
OSs 	Debian · Distribution upgrades · Puppet · APT repository · Debian packaging
War 	Security
Communication	Mail · Mailing lists · OTRS · PGP Keys · IRC
Network 	RANCID · Proxy access to cluster · Reporting a connectivity issue
Other 	Batch jobs ·Status monitors · Phabricator · Renaming users · Password and 2FA reset · Fundraising
Wiki 	Browse root category · :Category:Wiki · :Category:Templates
* :Category:Bot and monitoring
* Presentations
* Software deployments
== See also ==
* Grafana monitoring: grafana.wikimedia.org
* Wikimedia servers on Meta-Wiki
* System administrators on Meta-Wiki
__NOTOC__ Wikitech
Postfix

== Troubleshooting ==
=== Metrics ===
* All: Mail - Postfix servers
* Outbound: Mail - Postfix servers
* Inbound: Mail - Postfix servers
=== Logs ===
* All: Postfix
* Outbound: Postfix mx-out
* Inbound: Postfix mx-in
=== Queues ===
* active: in memory queue, messages that are ready to be delivered
* incoming: disk queue of received messages
* deferred: messages that failed delivery, they are retried with exponential backoff
Inbound mail is mostly redirected to our Gsuite instance and VRTS, so you should expect fewer that 20 messages in the deferred queue. Outbound mail primarily consists of email notifications from our projects to users. The deferred queue for outbound will be around 1000 messages per server as external mail servers are unreachable for a variety of reasons. Under normal load you should expect the active and incoming queues to be mostly empty, with brief spikes as mail is received and then immediately delivered. To obtain a quick overview
of the active and incoming queues use the qshape tool:
$ sudo qshape 
 T 5 10 20 40 80 160 320 640 1280 1280+
 TOTAL 0 0 0 0 0 0 0 0 0 0 0
The value in the T column is the total mail in the incoming and active queues. Each following numbered column displays the count of mail that is the corresponding number of minutes old. 
==== Show queues ====
# View the incoming and active queues 
$ sudo qshape
# View the deferred queue
$ sudo qshape deferred
# List all mail in all queues
$ postqueue -p 
# Show queue count totals
$ postqueue -j | jq -s 'group_by(.queue_name) | map({"queue": .[0].queue_name, "count": length})'
==== Flush the mail queue ====
$ postqueue -f
==== Deliver just one specific email from the queue ====
$ postqueue -i <QUEUE_ID>
==== Search for a specific email from a sender ====
$ postqueue -j | jq 'select(.sender == "wiki@wikimedia.org")'
==== Search for a specific email to a recipient ====
$ postqueue -j | jq 'select(.recipients[].address == "butter@example.com")'
==== Count number of emails to a specific recipient ====
$ postqueue -j | jq -s 'map(select(.recipients[].address == "butter@example.com")) | length'
==== Remove emails to a specific recipient ====
$ postqueue -j | jq -r ' select(.recipients[0].address == "user@example.com")
 | select(.recipients[1].address == null)
 | .queue_id' |
 sudo postsuper -d -
==== View a message in the queue ====
$ postcat -q <QUEUE_ID>
== Configuration ==
Postfix has two main config files:
# main.cf: Specifies config options for the postfix daemons
# master.cf: Specifies the daemons postfix should run
Category:Email
Postgres

PostgreSQL is used for specific services in the foundation's infrastructure. Currently Maps, Netbox, Airflow and PuppetDB use Postgres to store data. 
== Monitoring ==
=== Syncing Postgres replica ===
After a postgres slave replica has been re-imaged or it has fallen behind in replication due to network issues or other problems, it will first need to sync all data before it can be placed in production. The sre.postgresql.postgres-init cookbook can be used to resync a host
$ sudo cookbook sre.postgresql.postgres-init --replica ${replica_server} --reason "Resync after planet sync" --depool --pgversion 11
=== POSTGRES_HOT_STANDBY_DELAY ===
This alert indicates that a replica has fallen behind the master database in replication. This is not immediately a cause for panic, as a host can recover assuming it is not left out of replication for long. However, the postgres logs should be examined and connectivity between individual replicas and the master should also be checked. 
If many or all hosts in the maps cluster are issuing this alert, it could be the case that the OSM dataset is being refreshed - during this operation it is likely that hosts will fall behind as large numbers of changes happen on the master during import. Check the cluster masters for import processes - generally this is a passing problem and will self-resolve over time. Follow the restart process if hosts need to be restarted.
=== requested WAL segment XXXXXXXXXXXXXXXX has already been removed ===
In this case the WAL segments (roughly the postgres' journal) have been deleted on the master, that rightfully tells the replica about their absence causing the replication to break. If this happens, you need to resync the replica from the master, using the sre.postgresql.postgres-init cookbook. https://phabricator.wikimedia.org/T383114 shows an example of an occurrence of the issue on puppetdb2003.
=== Dashboards ===
We have a number of Grafana dashboards that can be helpful:
* https://grafana-rw.wikimedia.org/d/000000828/postgresql-database
* https://grafana-rw.wikimedia.org/d/000000469/postgres?orgId=1
* https://grafana-rw.wikimedia.org/d/wGgaPlciz/postgresql-overview?orgId=1
Primary database switchover

See Switch master/Archive for the previous version.
=== Updates ===
These parts need updating in case of a switchover:
* dbctl (for mediawiki config)
* puppet
* DNS
* heartbeat
* orchestrator tags
* zarcillio
* query killer
== Steps ==
There are previous steps before doing the topology change, use this task as reference:
https://phabricator.wikimedia.org/T313383
The key part is to first change the topology to get all the hosts to replicate from the new master:
Assuming old master: db1181 and new master: db1136
 sudo db-switchover --timeout=25 --only-slave-move db1181 db1136
Once that is finished, we are ready to get the section into read-only and then change the master:
To interact with mediawiki dependencies we use dbctl:
 sudo dbctl --scope eqiad section sX ro "Maintenance until 06:15 UTC - TXXXX"
 sudo dbctl config commit -m "Set sX eqiad as read-only for maintenance - TXXXX"
Now we can perform the switch:
 sudo db-switchover --skip-slave-move db1181 db1136
Assuming the above went fine, we can set the section back to WR and promote the new master
 sudo dbctl --scope eqiad section sX set-master db1136
 sudo dbctl --scope eqiad section sX rw
 sudo dbctl config commit -m "Promote db1136 to sX primary and set section read-write TXXXX"
The rest of the steps provided db-switchover by the after the switchover need to be followed in order to ensure that the post-switchover tasks are completed successfully. 
Check this task for reference: https://phabricator.wikimedia.org/T313383
Please also see the main dbctl article
Project Honeypot

:Andrew doesn't think this is running any more and SAL indicates it was disabled 2009-08-13.
We send data from spam.log to an external party, Project Honeypot, in exchange for them sending us abuse data.
It's run as a script, /home/andrew/send-honeypot-data.pl in Andrew's crontab on hume.
Here's the general outline they gave of how to send data to them:
curl -k 'https://feeds.projecthoneypot.org/rc3.php' \
 -F 'access_key=[See the script]' \
 -F 'op=FILE_GENERIC' \
 -F 'logfile=@/absolute/path/to/logfile'
It uses .last-honeypot-pos in /home/andrew to figure out the last line it sent, so as not to send the same data more than once.
Contacts at Project Honeypot:
* Lee Holloway, first name at unspam dot com.
* Matthew Prince, first name at unspam dot com.
Projects

Projects is a generically used term in Wikimedia's documentation. It may refer to:
* Wikimedia projects, the different Wikimedia communities, as groups (e.g. Wikipedias) or individually (e.g. French Wikisource). 
** See also Complete list of Wikimedia projects
* Phabricator projects, the #tags for grouping tasks by topic area
* Cloud VPS projects, see a complete listing at https://openstack-browser.toolforge.org/project/ 
* WikiProjects, community organized work around a specific topical focus
** See a listing on Wikidata at d:Q4234303 or the largest example set on English Wikipedia at w:en:WP:WikiProject
* Team projects, organized work by the Wikimedia Foundation or an Affiliate organization
== See also ==
* SRE project wishlist, the content previously at this page title
* Upstream projects, software projects we rely on which are developed by neighboring communities
Category:Disambiguation pages
Projects for September 2009

* order!
* SSL poke stuff
* Roll over items from Projects for August 2009
== SSL ==
* bugzilla:16822 Provide SSL/HTTPS interface to upload.wikimedia.org and use it for SSL-served pages
* bugzilla:20643 Serve SSL/HTTPS sites out of same domain names as HTTP access: https://en.wikipedia.org/
** This is potentially harder :)
** Taipei experiment's issues w/ parser cache shouldn't be a problem if we set this up right
* bugzilla:20454 -- cleanup for SSL favicons
* bugzilla:225 -- once that's all up we can see about enforcing SSL for all logins
== Squid capacity upgrade ==
* New Squids with SSDs, both Tampa and (a bit later) esams
== Datacenter project ==
* Ongoing investigation and planning for our main dc, and some caching centers
== Miscellaneous servers for esams ==
* The current servers are old and well out of warranty, we'll want ~10 or so new ones
== Puppet ==
* Version control
* Slow but ongoing integration
== Move Wikitech ==
* To server sage (at knams), with a (read-only?) replica
== esams cleanup ==
* Cleanup and documentation of the move
== pmtpa management network ==
== LVS ==
* Reaching its capacity limits, needs an overhaul with more efficient kernel connection scheduling, or a new hw based solution