nyuuzyou/wikis huggingface dataset:
Php-wmerrors

php-wmerrors is a PHP extension that extends how PHP errors are handled and displayed. It is specifically designed for situations where the standard PHP API is not sufficient for userland code to render an error page and log stacktraces (e.g. out of memory errors). More technical details can be found in T187147#4298721 (last evaluated in 2018 for PHP 7.1+).
== Service at WMF ==
=== Configuration ===
Error page:
The bulk of the logic is in the php-wmerrors extension. At runtime, we inject the php7-fatal-error.php (maintained in Puppet) which defines the HTML template, and takes care of the stats counter and normalized ingestion to Logstash, synthetising a message under type:mediawiki AND channel:exception, structured the same way as uncaught exceptions that userland code in MediaWiki is able to report directly.
Appservers:
* puppet: profile::mediawiki::php (install packages, set and create runtime files, set and create INI file)
* puppet: mediawiki::web::yaml_defs (pass runtime file contents as YAML to k8s builder)
mw-on-k8s (ref T288851)
* deployment-charts: mediawiki/templates/lamp/deployment.yaml (materialize runtime files)
* docker-images: php/7.2/fpm-multiversion-base/conf/wmerrors.ini (set and create INI file, duplicated from puppet)
=== Deployment ===
For the official Pecl packages, refer to How to PECL on mediawiki.org.
For production, refer to Debian Packaging.
== Code stewardship ==
* Issue tracker
* See mw:Developers/Maintainers
==External links==
*Source code
Category:MediaWiki production
Ping offload

Service status: Production
Documentation status: Ready
Goal: Lower the high ICMP load on LVS/CP servers by offloading echo requests to a dedicated server.
Linux has internal ICMP rate limiters that can cause the kernel to drop valuable ICMP packets. By offloading ICMP echo, we make sure the "important" ICMP (eg PMTU discovery) doesn't get dropped.
=== Deployment ===
Deployment task: https://phabricator.wikimedia.org/T190090
Rebuilt in: https://phabricator.wikimedia.org/T295767
eqiad: ping1002.eqiad.wmnet
codfw: ping2002.codfw.wmnet
esams: ping3002.esams.wmnet
=== Monitoring ===
Icinga: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=ping1001&style=hostservicedetail (and ping2001)
Grafana dashboard: https://grafana.wikimedia.org/d/000000513/ping-offload
External monitoring: Ping to VIPs configured in Watchmouse
==== InAddrErrors alert ====
From the Grafana dashboard alerting.
This means the server is receiving packets for an IP not existing on the server.
# Run ip addr to check if all the redirected IPs are present on the loopback interface
## If not, they can manually be added temporarily with ip addr add <ip>/32 dev lo:ping_offload
# If the IPs are present, use tcpdump to find the IP in question (eg. filter out all the present IPs)
# In any cases or if the troubleshooting takes too long, disable the redirect (see bellow)
=== How-to ===
==== Deploy a new host ====
# Create a VM, see existing VMs on https://netbox.wikimedia.org/virtualization/virtual-machines/?q=ping
# Assign the ping_offload role in Puppet (eg. https://gerrit.wikimedia.org/r/c/operations/puppet/+/564873)
# Add the target VIP to its configuration (eg. https://gerrit.wikimedia.org/r/c/operations/puppet/+/564908)
# Set the VIP and ping host in Homer (eg. https://gerrit.wikimedia.org/r/c/operations/homer/public/+/564917)
==== Temporarily stop the ICMP echo redirect ====
If the system is showing signs of issues or needs to go down for maintenance. 
On both cr1 and cr2 routers of the target site, enter the following commands:
# deactivate firewall family inet filter border-in4 term offload-ping4
# deactivate firewall family inet filter transport-in4 term offload-ping4
Then verify that the changes about to be made are correct, the output should be similar to:
# show | compare
[edit firewall family inet filter border-in4]
!       inactive: term offload-ping4 { ... }
[edit firewall family inet filter transport-in4]
! inactive: term offload-ping4 { ... }
Finish by committing the changes (replace <TASK #> with a phabricator task ID or relevant comment):
# commit comment "<TASK #>"
To confirm that the change is effective, monitor tcpdump on the ping host (for example sudo tcpdump -i ens5 icmp -nn) or the dashboard.
To re-activate the redirect, re-do the similar changes as above but replace deactivate with activate
=== Possible improvements ===
* Use BGP flowspec to automatically advertise/remove the redirect
* Add IPv6 support
* Have multiple ping servers per site for redundancy
Category:Services
Category:Runbooks
PipelineLib

PipelineLib is a container based CI workflow specification and engine. It executes any number of discrete CI tasks—building images, running tests, deploying images, etc.—based on a simple YAML file provided in each project repo. It is similar in spirit to .travis-ci.yml and .gitlab-ci.yml for GitHub and GitLab, but provides constructs specific to WMF needs and supports a more advanced DAG based execution scheme.
Below are links to tutorials, user guides, conceptual/architecture documentation, and configuration reference that should help everyone from WMF/volunteer developers wanting to quickly get their projects using a .pipeline/config.yaml to CI maintainers wishing to reimplement the current Groovy/Jenkins based implementation of PipelineLib.
PipelineLib/Concepts

= Overview =
The WMF is moving to deploying services via Docker images to Kubernetes, and pipelinelib was developed to support that.
The WMF CI system supports a form of self-serve for the software we maintain and deploy to our production systems. The basic idea is that each repository contains all configuration necessary for CI to do the following.
# Build and containerize all applications implemented in the repo.
# Gate applications by executing a dependency graph of pipeline stages, which:
## Build and run the application's test images.
## Deploy application images to a testing cluster and perform end-to-end tests.
## Publish application images to a public WMF Docker registry for subsequent production deployment.
This system makes some assumptions and and has some constraints:
* Each repository can specify how it is built and tested, and how the Docker image needed to deploy it is built.
* The details of where and the Docker image it deployed are specified to CI by sysadmins, not developer of individual services. Deployment is done using helm and its charts. The charts are provided by SRE or RelEng.
* We only support building of Docker images via Blubber, another in-house tool. Docker images may only be based on our own, vetted base images, not arbitrary base images from the Internet.
* We publish any Docker images we build to our own Docker registry.
* We only support microservices.
* For pipelinelib, we don’t care about CI/CD outside of WMF. This is not meant to be a generic solution.
The Blubber software supports Docker images with multiple variants. Typically our projects have variants for building, testing, and using in production.
The repository is expected to have a .pipeline/config.yaml file at its root. That file is read and inspected by CI via pipelinelib. If the file does not exist, CI does nothing for the project.
config.yaml specifies one or more independent pipelines. Each pipeline builds its own Docker image, or variant, which can be its own service, indepenent of any other services built from the same repository. To build (and test) the software, each pipeline specifies a directed acyclic graph or DAG, consisting of stages. Each stage is a step to build the software, test it, or otherwise move towards the goal of getting the software running in production.
== Example .pipeline/config.yaml ==
pipelines:
 test:
 blubberfile: blubber.yaml
 stages:
 - name: test
 - name: candidate
 build: production
 rehearse:
 blubberfile: blubber.yaml
 stages:
 - name: test
 - name: candidate
 build: production
 publish:
 image: true
 - name: rehearsal
 deploy:
 chart: https://releases.wikimedia.org/charts/blubberoid-0.0.9.tgz
 image: '${candidate.imageName}'
 tag: '${candidate.imageTag}'
 test: true
 publish:
 blubberfile: blubber.yaml
 stages:
 - name: test
 - name: candidate
 build: production
 publish:
 image: true
 - name: rehearsal
 deploy:
 chart: https://releases.wikimedia.org/charts/blubberoid-0.0.9.tgz
 image: '${candidate.imageName}'
 tag: '${candidate.imageTag}'
 test: true
 - name: production
 publish:
 image:
 id: '${candidate.imageID}'
 tags: [stable]
PipelineLib/Guides/How to define a golang test pipeline

Welcome to How to define a Go test pipeline, a simple PipelineLib guide to get your Go project patchsets tested and linted.
= Prerequisites =
* A basic understanding of the Blubber image build tool. The Blubber tutorials can help you with that.
* An understanding of .pipeline/config.yaml basics, namely how to define a simple pipeline that will build a test image variant and run it. Some of the PipelineLib tutorials should help you learn these basics.
And of course, don't hesitate to get in touch with the WMF Release Engineering Team for help with following this guide.
= Guide =
The basic steps to getting your Go test/lint tasks running in CI using Blubber and PipelineLib are:
# Define image variants in .pipeline/blubber.yaml that invoke your test and lint entry points.
# Define a new pipeline that includes stages for building and running your image variants.
# Tell our CI system about your new pipeline by following the How to configure CI for your project supplemental guide.
== Define test and lint image variants ==
dev@laptop:~$ cd src/project/foo
dev@laptop:foo$ mkdir -p .pipeline/
dev@laptop:foo$ vim .pipeline/blubber.yaml
version: v4
variants:
 build:
 base: docker-registry.wikimedia.org/golang:latest
 apt: { packages: [ca-certificates, g++, gcc, git] }
 builder:
 requirements: [go.mod, go.sum]
 command:
 - sh
 - -c
 - >-
 go mod download &&
 go mod verify &&
 go get -u golang.org/x/lint/golint
 copies: [local]
 runs:
 insecurely: true
 lint:
 includes: [build]
 entrypoint: [golint, ./...]
 test:
 includes: [build]
 entrypoint: [go, test, ./...]
This .pipeline/blubber.yaml gives you two runnable image variants, one that invokes golint ./... and another that invokes go test ./.... In addition, both variants use a base build variant that installs commonly needed development packages from our APT repository and downloads/verifies your project's Go module dependencies.
== Define a simple test pipeline ==
Now that you have two variants for linting and testing, define a test pipeline that builds and runs them.
dev@laptop:foo$ vim .pipeline/config.yaml
pipelines:
 test:
 blubberfile: blubber.yaml
 stages:
 - name: lint
 - name: test
== (Optionally) Perform lint and test in parallel ==
Depending on your testing requirements, you may wish to run these two (or more) tasks in parallel. Configuration of execution is how you accomplish that.
dev@laptop:foo$ vim .pipeline/config.yaml
pipelines:
 test:
 blubberfile: blubber.yaml
 stages:
 - name: build
 build: build
 run: false
 - name: lint
 - name: test
 execution:
 - [build, lint]
 - [build, test]
This configuration essentially means: There are three distinct stages in two branches of execution. Execute build first, then branch out and execute lint and test independently in parallel. As a visual graph, it ends up looking something like:
 [set-up]
 ↓
 build
 ⇙ ⇘
lint test
 ⇘ ⇙
 [tear-down]
= Next up =
Now that you have a test pipeline defined for your project, head on over to How to configure CI for your project to tell our CI how to run it for each patchset submitted to Gerrit.
Pixel/Runbook

This is the runbook for deploying and debugging Pixel problems.
==Meta==
*Issue tracker (Phabricator): web_team_visual_regression_framework
*Documentation: Pixel
== Update to new version of Pixel ==
Pixel is auto updating on the production server to the latest tag. If you make a new release, make sure to add a tag and push it to git.
// TODO Monte please share the correct way to update the changelog!
git tag -a vX.Y.Z -m "Release version X.Y.Z"
git push origin vX.Y.Z
== Revert to a old version of Pixel ==
If the current tag version of Pixel is broken and you cannot easily fix it, you need to revert to the latest working version. 
# Disable the auto update on the server. 
# Login to the server ssh production.pixel.eqiad1.wikimedia.cloud 
# Switch to the pixel user sudo su - pixel 
# Disable the auto update nano /home/pixel/pixel.sh and comment out the line git checkout "tags/$latest_tag"
# Go to the repository: cd /home/pixel/pixel
# Checkout the last working tag: git checkout "tags/vX.Y.Z"
When there's a fix, make sure to enable the auto update again.
== Prio 1, 2 or 3 job is failing ==
You receive an alert email from Grafana with the subject "Pixel: Prio X job are failing qte". This means that the Pixel job failed running on https://pixel.wmcloud.org/. 
# Checkout the failure reported in Grafana. Go to https://grafana.wikimedia.org/d/lC3anj1Iz/pixel and check which jobs are failing.
# Login to the server: ssh production.pixel.eqiad1.wikimedia.cloud
#
Category:Runbooks
Planet.wikimedia.org

== What is planet? ==
An RSS feed aggregator. See meta:Planet_Wikimedia for details.
== How do I access it? ==
en.planet.wikimedia.org and several other languages, *.planet.wikimedia.org.
== Which languages exist? ==
The current list of languages is:
*Arabic / العربية
*Catalan / Català
*Czech / Česky
*German / Deutsch
*English
*Spanish / Español
*French / Français
*Scandinavian
*Indonesian / Bahasa Indonesia
*Italian / Italiano
*Polish / Polski
*Portuguese / Português
*Romanian / Română
*Russian / Русский
*Serbian / Српски / srpski
*Chinese / 中文
=== How to add a new language? ===
Make a change in the operations/puppet repo in ./modules/profile/manifests/planet/venus.pp . You'll see the existing examples.
== Where should I report issues? ==
On Phabricator. In product Wikimedia, component Planet
== Where does it run? ==
On planetX001 where X={1, 2} varying on the datacenter (eqiad, codfw). Before that it was on zirconium in eqiad. The even older planet was on singer.
== Where's the code? ==
In git, in the operations/puppet repo:
#./manifests/role/planet.pp
#./manifests/misc/planet.pp
and 
#https://offog.org/code/rawdog/
== How are planets being updated? ==
Puppet creates cron jobs, running as user "planet".
 crontab -u planet -l
 example: 0 0 * * * /usr/bin/planet -v /usr/share/planet-venus/wikimedia/de/config.ini > /var/log/planet/de-planet.log 2>&1
There is also /var/lib/planet/update-planets to update all planets at once. run with sudo -u planet though to not mess up permissions on log files.
== How do I add or remove feed URLs? ==
# Clone the operations/puppet Git repository: git clone https://gerrit.wikimedia.org/r/operations/puppet;
# go to modules/planet/templates/feeds/ and edit one of the [language]_config.erb files (where [language] matches the code of the language of the new feed);
# submit to Gerrit and have somebody review and it.
#: → Use "planet-feeds" as the topic (branch) when submitting. Moreover, prepend "planet:" to the commit message, e. g. planet: Add XYZ feed (861452). Currently, in November 2022, user Dzahn is responsible for reviewing and merging respective change sets and may be added as reviewer. See the Git/Reviewers #operations/puppet MediaWiki.org page for reference.
You can see the existing lists of feeds that are included at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/planet/templates/feeds/
== How do I change HTML of the index pages? ==
#see above, edit ./puppet/templates/planet/index.html.tmpl.erb 
== How do I add/change translations in the index pages or add a new language? ==
#see above, edit ./puppet/manifests/role/planet.pp (find $planet_languages=...)
== How do I request changes if I can't or don't want to submit a change set myself? ==
Ask on meta:Planet_Wikimedia#Requests_for_Update_or_Removal
Category:Services
Planet/pl

ERROR:planet:Error 500 while updating feed <http://blog.kangel.info/rss/cat/wikipedia/content/html/10>
ERROR:planet:Error 404 while updating feed <http://bigosnamadagaskarze.blogspot.com/feeds/posts/default/-/Wikipedia>
ERROR:planet:Error 500 while updating feed <http://wikiźródła.pl/blog/?feed=rss2>
ERROR:planet.runner:Update of <http://tarplwiki.wordpress.com/feed/> failed
ERROR:planet:Error 404 while updating feed <http://historiaimedia.org/tag/wikipedia/feed/>
ERROR:planet.runner:Update of <http://wikismoki.wordpress.com/feed/> failed
ERROR:planet.runner:Update of <http://plwiki.wordpress.com/feed/> failed
Podman

Podman is a daemonless container engine to run OCI containers. It is notably able to run containers in rootless mode. For a lot of simple use case, it is a drop-in replacement to Docker (alias docker=podman).
It does not support BuildKit as far as I know.
== Debian Bullseye ==
As of Feb 8th 2023, the Debian packages for Debian Buster are outdated and have some breakage. Among others, having containers to reach each others via a DNS lookup does not work out of the box. One has to install golang-github-containernetworking-plugin-dnsname and dnsmasq-base. Then the need to add a symbolic link: ln -s /usr/lib/dnsname /usr/lib/cni/dnsname debian #985548 and debian #985549.
Random notes:
For docker-compose use the version 2 (v1 is end of life):
sudo apt install -y docker-compose-plugin
sudo sudo ln -s /usr/libexec/docker/cli-plugins/docker-compose /usr/local/bin/docker-compose
Boot up a podman service which emulates the Docker daemon. The systemd unit can be run as a normal user and podman will then yields rootless containers:
sudo systemctl --user start podman
It listens on a socket at $XDG_RUNTIME_DIR/podman/podman.sock which upon receiving traffic will spawn podman system service. I had some issue with the socket vanishing due to a poor interaction with Podman and the systemd podman.socket. Apparently resolved by installing the latest version of Podman manually.
Or maybe I had to instruct the service to not deal with the socket by passing --time=0
Then set in your env:
export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"
And you can use docker-compose
== Install from source ==
Upstream has a list of installation methods one being building from sources.
Portada

Wikitech es el hogar de la documentación técnica de la infraestructura y los servicios de Wikimedia Foundation. Esto incluye clústeres de producción, Servicios en la nube de Wikimedia, alojamiento Toolforge y el Beta Cluster.
Para editar esta wiki, necesita una cuenta de desarrollador de Wikimedia. Wikitech no usa el inicio de sesión único unificado de Wikimedia (SUL) (como en Wikipedia) y no permite la edición sin una cuenta.
Portal:Cloud VPS/Admin

Documentation of admin procedures for Cloud VPS. See Help:Cloud VPS for user-facing documentation.
== Admin permissions ==
Performing admin procedures requires having admin permissions on Cloud VPS. There is not a single "admin" flag, but a set of interrelated permissions you can be granted. These are described in detail in the page Cloud roots and Cloud admins.
== Deployments ==
Cloud VPS is made of two customized OpenStack deployments: eqiad1 (production) and codfw1dev (testing). Both deployments are managed by the Cloud Services team together with other WMF staff members and with the help of external technical volunteers.
== Useful links ==
* Cloud VPS Runbooks
* Who to ask
* Packaging software
* Aptly: A guide to setting up a local debian repo for installing your own .deb packages within a labs project
* Disabling an account
* Logout all users
* Suspending instances
* Using nova-manage (OpenStack cli tools)
* DNS with Designate + PDNS
* LDAP
* Testing deployment (Cloud VPS test environment, currently hosted in codfw)
*cloudweb1003: WMCS web UIs - Horizon and Striker
== All subpages ==
Cloud VPS
 
Admin
Portal:Cloud VPS/Admin/Cloudinfra

The cloudinfra project is a very special project intended to hold infrastructure relating to running the rest of Cloud VPS. Access to it is restricted under Help:Access policies.
== Instances ==
Types of instances it contains currently:
=== ntp ===
These instances serve the rest of Cloud VPS with NTP updates.
=== mx-out ===
These instances serve the rest of Cloud VPS with outbound mail handling.
=== cloud-puppetmaster ===
These instances serve as the puppet master for projects that do not use project-local puppet masters.
=== cloudinfra-db ===
These instances contain MariaDB databases that keep state for important Cloud VPS infrastructure hosted within Cloud VPS itself. Currently that only includes the Puppet ENC database (labspuppet).
We don't yet have regular backups of that cluster (:phab:T301640). :(
=== cloudinfra-internal-puppetmaster ===
This is a small project-local puppetmaster that keeps secrets (such as cumin ssh keys and enc database passwords) for some of the cloudinfra instances.
=== enc ===
These instances host the Puppet ENC API that was previously hosted on the cloud-puppetmasters.
== DNS ==
This project also has an important role with regards to DNS. See Portal:Cloud_VPS/Admin/DNS for more info.
== codfw1dev ==
The equivalent of this project in the codfw1dev deployment is called cloudinfra-codfw1dev, and has the same role within the deployment.
Portal:Cloud VPS/Admin/Cloudlb

The cloudlb hosts provide load balancing for hardware providing cloud-realm services (see Cross-Realm traffic guidelines#Case 4: cloud-dedicated hardware).
As of 2024-01-09 this includes:
* OpenStack control plane APIs
* OpenStack metadata MariaDB database
* radosgw for object storage
* Wiki Replicas client access (see Portal:Data Services/Admin/Wiki Replicas)
== Overview ==
These are fairly simple hosts, with two main services:
* HAProxy which does the actual load balancing
* Bird which announces the used service VIPs via BGP to the cloudsw switches
== Failover ==
These hosts are active-active by default. Anycast-healthchecker should automatically depool any misbehaving servers, if not, disabling Puppet and stopping bird.service will manually do that.
Portal:Cloud VPS/Admin/DB

This page describes the DB (database) usage by our Openstack deployments.
= Databases =
Production deployments must use a production database server/service, probably m5-master.eqiad.wmnet in coordination with the DBA team.
Some deployments use a local mysql database in the control boxes.
Each component of Openstack uses a different database: nova, nova-api, glance, keystone, neutron, etc.
== GRANTS ==
Openstack services require a specific set of database GRANTs to work:
TODO: elaborate
= Connection throttling =
We had some issues in the past because Openstack assumes unlimited usage capabilities in the database. This is not true in our case, specially in production, since
m5-master.eqiad.wmnet is shared with other different services.
TODO: elaborate in what has been done by Brooke to throttle connections.
= See also =
* Portal:Cloud_VPS/Admin/Deployments
Portal:Cloud VPS/Admin/Devstack magnum

= Overview =
Devstack can be used to prototype and test openstack magnum for kubernetes and other orchestration
= Installation guides =
= Magnum info =
== Create a template ==
openstack coe cluster template create my-upgrade --image Fedora-CoreOS --external-network public --fixed-network private --fixed-subnet private-subnet --dns-nameserver 8.8.8.8 --network-driver flannel --docker-storage-driver overlay2 --volume-driver cinder --docker-volume-size 30 --master-flavor m1.small --flavor m1.medium --coe kubernetes --labels kube_tag=v1.21.8-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/,cloud_provider_enabled=true
kube_tag can be found at https://hub.docker.com/r/rancher/hyperkube/tags
== scale a cluster ==
openstack coe cluster resize <cluster name> <number of workers>
== upgrade a cluster ==
openstack coe cluster upgrade <cluster name> <template to upgrade to>
If this gets stuck you may need to uncordon any cordoned nodes and then run:
openstack coe cluster resize <cluster name> <current number of workers>
to reset the cluster status. Then run the upgrade again.
While updating kubectl won't work, but services will still be running and available.