nyuuzyou/wikis huggingface dataset:
WebPageReplay/Runbook

This is the runbook for deploying new versions of WebPageReplay/Browsertime/Firefox/Chrome.
==Meta==
*Issue tracker (Phabricator): synthetic-performance-testing
*Documentation: WebPageReplay
== Update to new version ==
Firefox and Chrome are bundled in the sitespeed.io Docker container. When there's a new version, check the changelog and update like this:
# Clone the Gerrit repo: ssh://USER@gerrit.wikimedia.org:29418/performance/synthetic-monitoring-tests.git
#Go into the new cloned repo: cd synthetic-monitoring-tests
#Create a new branch named with the new version number: git checkout -b my_new_version
# Edit the run.sh file and change one of the first lines that looks something like this: DOCKER_CONTAINER=sitespeedio/sitespeed.io:10.3.2
#Change the version number (10.3.2 in this case) to your new version.
# Commit the file and send in the commit for a review.
#When the change is approved, the new version will be automatically picked up the next iteration of running the tests.
==Deploy first time==
On a new server you need to install the dependencies (Docker) and follow the instructions. 
== Debug missing metrics ==
If metrics stops arriving in Grafana the reason can be two different things: Either something is wrong with Graphite or something is broken on the WebPageReplay server. First check the #synthetic-tests-error-reporting Slack channel and see if you have any related errors in that channel. 
If you can't see any errors, then focus on the WebPageReplay server. You can see which server that run which test at WebPageReplay#Servers. Log into the server and check of if any tests is running. Do that by running docker ps
If everything is ok it should look something like:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ae4f0e6946c sitespeedio/sitespeed.io:35 "/start.sh --config …" 6 minutes ago Up 6 minutes sitespeedio
We start (and stop) the container for every new test so a container should have been created for maximum a couple of minutes ago. If the created is a couple of hours (or days) ago, something is wrong. The container is stuck, probably something happened with the browser. You can fix the test by killing the container: docker kill sitespeedio
The root cause (that the container got stuck) is still there but restarting the test usually works. After you killed the container, wait a minute and check again that a new container is running by using docker ps
Check Grafana the coming hour to make sure it doesn't get stuck again.
Category:Runbooks
WebPageTest/Run locally

Running a WebPageTest instance on your local machine has been made a lot easier in 2017 with the support for Linux agent. The agent is still experimental but it is a good start for us to be able to fix bugs and test things locally. But we are still running the Windows version on our test agents (we will switch when we feel Linux supports what we need).
=== Mac OS X ===
On my local I run a server and an agent. I rarely (or never) do any changes on the server so I run them using the provided Docker containers.
# Start the server on port 4001
docker run -p 4001:80 --rm webpagetest/server
# Switch to another terminal
# Start the agent
docker run --rm \
-e SERVER_URL="http://192.168.65.1:4001/work/" \
-e LOCATION="Test" \
-e SHAPER="none" \
--cap-add=NET_ADMIN \
webpagetest/agent
Access http://localhost:4001/ and you should see the server interface. Choose the Test location and run your test.
WebPageTest will kill the current browser sessions, so if you want to run your test on Firefox, go to the server in Chrome and vice verse.
==== Known issues ====
* The video isn't working at all at the moment when you run locally in Docker (on Mac OS X).
* I've seen issues where the video becomes all kind of colors and not recording the browser, but I haven't been able to pinpoint the problem.
Category:Wikimedia Performance
WebPageTest/UpdateToLinux

=== Background ===
In June (or July) 2018 our current WebPageTest Windows agent will stop to work because of changes in Chrome and the old agent will not be upgraded to work with the change, meaning we need to move to the new agent multi platform agent. 
The new multi platform agent works both on Windows and Linux. We've been trying out the new one on Linux for on and off for half a year and filing upstream bugs and soon it is time for us to do the switch.
We will move our agent to Linux and there are a couple of reasons: The Linux agent is cheaper to run on AWS (than Windows), Linux makes it easier for us to test out ourself and do pull requests if we fix bugs. And it also makes it possible for us to move the agent in-house in the future. And we can add more testing for more Wikis and more pages.
We will miss out on performance tests on Internet Explorer, however our old agent couldn't be updated so we are testing Internet Explorer 11 with no HTTP/2 support.
=== Status ===
Testing the new Linux version has been going on for a long time and it tracked in https://phabricator.wikimedia.org/T165626
==== Timeplan ====
We aim to remove the Windows agent the 26 of March 2018.
==== Known issues ====
There are a couple of bugs and they are tracked in https://phabricator.wikimedia.org/T186466.
What's important for you as a WebPageTest user is that the Linux instance seems faster than before (most of our pages now have faster first visual change and lower Speed Index values). 
What also good to know is that for some pages on desktop we have seen a bigger variance than before. However for our alerting we are aiming to move to our WebPageReplay setup.
=== Migrating your WebPageTest Grafana dashboards ===
Migrating your old dashboards to start using the metrics from our new Linux agent is pretty straight forward. We have changed the name of the location from us-east-1 to us-east. That means that in all your dashboards you need to change us-east-1 to us-east.
There's one annoying thing and that is that the Chrome browser names has changed between the old WebPageTest agent and the new one. That means that you need to be careful when you do the change in Grafana because key parts will not be populate themselves when you change the location. It's easiest to Toggle edit mode and use the plaint text mode to do the change.
The browser names changes:
Google_Chrome -> Chrome
Google_Chrome-emulateMobile -> Chrome-emulateMobile
Firefox are the same.
You can checkout https://grafana.wikimedia.org/d/000000210/webpagetest as an working example of a dashboard that moved to the new Linux agent.
Category:Wikimedia Performance
WebPageTest/WPTBulkTester

WPT Bulk Tester is a Open Source script to bulk test many URLs using WebPageTest and get the result in a Google spreadsheet. It's created by Andy Davies.
The bulk tester is perfect when we want to test changes in a systematic way and use our own instance to test each URL many many times (WebPageTest.org has a limit of 9 runs per URL).
=== Start ===
You can see Andy's script and read his blog post. Do that before you start. If you follow the instructions there, it works. We also have a template that is pre-filled with API keys and server names and example how to test Wikipedia, contact the Performance Team to get help.
=== Gotchas ===
There are some things I learned the hard way so read it before you start.
==== Connectivity ====
If you want to set the connectivity on your run (and you sure want to do that) the recommended way according to the docs is adding it to the location. Say for example that we run our tests from eu-west-1:Chrome and want to run it using 2G then we change the location to eu-west-1:Chrome.2G and the connectivity will be 2G. Well, in the GUI it's 2G but checking the actual metrics like start rendering tells another story. Running on WebPageTest.org our page on 2G start renders after 12 s, setting it to 2G using the bulk tester we got rendering in 4 seconds so there's a mismatch in the latency and down/up-loading speed. 
But you can fix that. Checkout the source code configuration in WebPageTest and take the bwIn, bwOut and latency from there and add it to your run. Then it will work.
==== Verify ====
Before you start testing your changes with 30+ runs make sure that it works as expected first. Always test with one run, go to the WebPageTest result page, verify that the URL is the right one, that the screenshots are ok, the video looks good and the metrics seems to be ok compared with how you configured the runs.
Then when you made sure one run created the metrics you wanted, change the number of runs to a large number.
==== Scripted tests ====
You can script your tests, that works perfectly fine. The URL field will not have meaning so just skip that. A scripted test can look like this (set a specific header, navigate to that page).
addHeader X-Wikimedia-Debug: 1
navigate https://en.wikipedia.org/speed-tests/pages.html
Category:Wikimedia Performance
What is a database schema change

= What is a database schema change and why might I need one? =
This page is meant to give enough information to anyone who is considering a database schema change as part of their code development that will eventually be deployed into production on WMF servers (MediaWiki schema changes and standards will not be covered here).
== What is a schema change ==
Good database schema design is essential for security and performance purposes. As our overall code base evolves over time, this means that occasionally we will need to update databases and indexes. Here's some examples of typical database schema changes: 
 * [add complex and simple examples here]
== What is *not* a schema change ==
* Creating new tablesThis is not a dangerous process for developers to do, and can be done without DBA assistance. However, giving the DBA's a notification about the new table would be appreciated to allow for confirmation that accidental leakage of private data will not be possible.
* Dropping tablesThis action is a fairly low priority action and normally does not block anyone. This does not mean that it isn't complex - please add any tables that will be dropped to or as a subtask of that ticket.
* Other "cleanup" schema changes (e.g. a column that is stopped being used but it is not blocking code)While these are real database schema changes, if this cleanup does not block a code deployment, tag the request with the #DBA tag to prioritize this action appropriately.
* Updating your own tableIf this db schema change is for your own database (and not affecting other teams / individuals), you probably don’t need to go through this process.
== How long does a schema change take ==
Database schema changes can be quite time consuming and can be quite complex or, alternatively, somewhat easy to do. However, implementing schema changes on live databases should not be taken casually. Schema changes need to be requested early enough to allow for discussion (as needed), taking into account ongoing work and general task prioritization while supporting the overall team.
Some changes can take a week, whereas other changes can take several months (or even years) to fully be deployed in production. Generally, updates / alterations to database tables will be performed as soon as possible using the available team resources.
If in doubt, request all schema changes early — once a proposed feature has been scoped and is ready to be started. This might mean requesting a complex change months in advance of deployment.
== Are you ready to request a database schema change? ==
The SRE team follows a rotating three (3) month work schedule for their team goals and priorities. Having early (but not too early) knowledge of upcoming work that will determine future priorities is essential.
Click here for step by step workflow instructions.
What is a database schema change/workflow

= Workflow for requesting a database schema change =
The following is a step-by-step instruction guide to requesting database schema changes from the DBAs (part of the SRE team in the Technology Department)
== How to request a database schema change ==
# Create a Phabricator task with the #schema-change tag to propose a change to a table’s design.
# Add in relevant people that would need to see this request in order to provide constructive feedback in a timely manner. 
#* As needed, add tags #dba, #core_platform_team_backlog, #analytics and #dumps-generation to your ticket
# Have a conversation (usually on the Phabricator task, but occasionally a meeting or two will be needed) by the developers and engineers to agree on what is to be done, you’ll need to create an additional task ticket that will have the exact details of the changes that are needed.
# Create this new ticket as a subtask of the original Phabricator task:
#* Add applicable tags, such as: #dba, #core_platform_team_backlog, #analytics, #schema-change and #dumps-generation 
#* FYI: #blocked-on-schema-change is only used once you are actually blocked
# Provide the specific information that is needed (each batch group will have its own ticket, even if the alters are done at the same time):
## The ALTER TABLEs to run (link to a commit diff)
## Where to apply the schema changes (a specific dblist file or complete list of wikis) 
## When to apply the schema changes
##* If it depends on a particular code deployment; if it can be done at any time or; if it needs to be done by a certain future date
# Confirm that the schema change is backwards compatible
#* If it is compatible with current code and, if it can be completed at any time
# Confirm that the schema change has been tested 
#* On a test or beta wiki (Note: ss a last test, the change can be applied to testwiki before going into production)
# Confirm that the data should be made available on lab replicas and/or dumps:
#* If the schema change involves new columns or tables; if the data contains private or sensitive data (consult legal if you are unsure); and if it involves deletion of data that was previously available on labs
== Want specific examples and more background information on making database schema changes? ==
Click here for more detailed information.
Wiki Replica redaction

This page is to document how the data is sanitized for the Wiki Replicas public databases that Wikimedia Cloud Services provides.
This page provides a high-level overview for Wiki Replicas users. The detailed admin docs are in Portal:Data Services/Admin/From_production_to_Wiki_Replicas.
Sanitization happens in two steps: first some private data is removed by Sanitarium hosts, then more private data is removed through Wiki Replicas views.
== Sanitarium hosts ==
The replication into the Sanitarium hosts uses triggers and filters to remove sensitive columns, tables and databases in the simple case where there are no conditions (e.g. ensures user_password does not end up in Wiki Replicas).
There is also a check_private_data_report script to make sure redaction happened properly. This runs weekly via cron and emails the DBAs the results when a mismatch is found.
== Wiki Replicas views ==
Wiki Replicas hosts contain both tables and views. Wiki Replicas users only have permissions to access the views. The file maintain-views.yaml contains the view definitions that define what is public and available to end users. This contains conditional redactions that cannot be done by the Sanitarium triggers and filters (e.g. revision delete), and also serves as defense in depth in case one of the Sanitarium redactions fail.
== Document redaction decisions ==
TODO: include documentation/rationale on any info publicly exposed that is not publically exposed by MW.
Category:MySQL
Category:Data Services
Wiki Replicas Special Interest Group

The Wiki Replicas Special Interest Group, is a small group of stakeholders from various teams and demographics, who directly work on issues related to the Wiki Replicas.
Scope: All aspects of running and maintaining the Wiki Replicas, and providing these data services to end-users.
Stakeholder criteria: Demonstrated stakeholders are people with administrative or maintenance responsibilities, and representative power-users.
Members: Chase Pettet, Madhumitha Viswanathan, Bryan Davis, Jaime Crespo, Manuel Arostegui, Aaron Halfaker. (All tentative until first meeting)
Meeting cadence: Monthly
Minutes published at: /Meetings
Communication venue: #wikimedia-cloud
Lead: Chase
Wikidata Percent Usage Dashboard

== Introduction ==
The Wikidata Percent Usage Dashboard tracks the number and the percent of pages in Wikimedia projects that make use of Wikidata.
The working definition of "a page that makes use of Wikidata" in this project is the following one:
* we consider only pages in namespace = 0, and no redirects;
* Wikidata (WD) usage upon which the reported data are based excludes Sitelinks (see: eu_aspect field in the wbc entity usage table of the Wikibase schema);
* for Commons, we extend the definition to encompass the following namespaces: 0, 6 ,14.
== Dashboard operation ==
The dashboard relies on the following two components:
* the back-end update engine, WD_percentUsage_PRODUCTION.R, an R script running on regular daily update from stat1004;
** this script relies on the extraction of Wikidata re-use statistics from the wdcm_clients_wb_entity_usage table (by orchestrating a HiveQL call from within an R environment) produced in the data generation process from the Big Data component of the Wikidata Concepts Monitor system;
** the re-use data, identifying the page IDs of those pages that make use of Wikidata, are compared against page IDs on the per project basis to determine how many pages use Wikidata in any particular WMF project;
** the results are published daily on https://analytics.wikimedia.org/datasets/wdUsagePercentArticle/ (wdUsage_ProjectStatistics.csv) and used on the front-end.
* the front-end (ui.R, server.R), relying on a simple RStudio Shiny page to present the results and visualizations.
== Dashboard functionality ==
The dashboard provides:
* a table encompassing Wikidata usage statistics, in particular:
** the project (e.g. enwiki, itwikivoyage),
** total number of articles in the respective project,
** number of articles that make use of Wikidata (c.f. the definition given above),
** percent of articles that make use of Wikidata, and 
** project type (e.g. Wikipedia, Wikivoyage, Wikiquote, etc).
* several visualizations to help get a glimpse of a big picture:
** the distribution of Wikidata usage across project types (e.g. Wikipedia, Wikivoyage, Wikiquote, etc),
** the top 20 WMF projects in terms of the absolute number of pages using Wikidata, and
** the top 20 WMF projects in terms of the percent of pages within the project making use of Wikidata.
== Notes ==
The Wikidata Percent Usage Dashboard is developed by WMDE, 2019. 
GitHub: https://github.com/wikimedia/analytics-wmde-WD-WD_percentUsageDashboard
Wikifunctions

This page is the primary entry-point for documentation related to Wikifunctions technical infrastructure at Wikimedia.
== Sub-pages ==
=== Create a subpage ===
prefix=Wikifunctions/
type=create
placeholder=Name of subpage
useve=true
== Guides ==
== See also ==
* Abstract Wikipedia team, on mediawiki.org
__NOTOC__
Category:Wikimedia infrastructure
Wikifunctions/Beta Cluster

The Beta Cluster houses a functional test instance of Wikifunctions and associated services, with all software components running at HEAD.
The URL for Wikifunctions wiki is https://wikifunctions.beta.wmflabs.org/
== function-orchestrator and function-evaluator ==
The function-{orchestrator,evaluator} services run under Docker, on the deployment-docker-wikifunctions01 cloud instance.
* To view the configuration for the services, see the .
* To make changes to the configuration, use Horizon to manage the instance.
* To access the instance via SSH, see Help:Accessing Cloud VPS instances.
You need to be a member of the deployment-prep project to access the instance via SSH, and a project admin to change the instance configuration.
Both services are exposed via a web proxy. The evaluator is exposed to facilitate testing.
Service 	Public address 	Internal address
Orchestrator	wikifunctions-orchestrator-beta.wmflabs.org:443	deployment-docker-wikifunctions01:6254
Evaluator 	wikifunctions-evaluator-beta.wmflabs.org:443 	deployment-docker-wikifunctions01:6927
===Debugging===
The function-* services log to Logstash (see #Debug logs below for details).
The units are configured to automatically restart on failure. If deployment-docker-wikifunctions01 is up but one or both of the services are down, they are probably crash-looping. To debug, SSH into the machine and run:
# Check service status:
$ sudo systemctl status mediawiki-services-function-orchestrator
# Get additional logs:
$ sudo journalctl --unit=mediawiki-services-function-orchestrator
# See which Docker image is running:
$ docker inspect --format='{{.Name}} {{.Image}}' $(docker ps -aq)
# Get an interactive shell inside the Docker container:
$ docker exec -u root -it mediawiki-services-function-orchestrator.service bash
# Once you're inside the container you can run `apt update` followed by `apt install` to install additional debugging tools.
(Replace 'orchestrator' with 'evaluator' in the commands above for debugging the function-evaluator.)
==Running MediaWiki maintenance scripts==
To run maintenance scripts, you need to SSH into the Beta Cluster deployment host (currently deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud). See Help:Accessing Cloud VPS instances for access instructions.
Once you've logged in, you need to wrap invocations of maintenance scripts with mwscript, specifying --wiki=wikifunctionswiki to ensure the code runs in the context of Wikifunctions. 
For example: 
:mwscript extensions/WikiLambda/maintenance/loadPreDefinedObject.php --zid Z4 --wiki=wikifunctionswiki
*To run raw SQL against the database (don't), run: mwscript maintenance/sql.php --wiki=wikifunctionswiki
*To run code in an interactive MediaWiki PHP REPL, run: mwscript maintenance/shell.php --wiki=wikifunctionswiki
If you do anything potentially destructive, remember to log it to the RelEng server action log via !log message in .
==Debug logs==
Logs from both MediaWiki and the function-* services are shipped to Logstash. See for access instructions.
*Logstash dashboard for function-* services
*Logstash dashboard for PHP
Category:Abstract Wikipedia team
Wikimedia Cloud Services team/Administrator meeting template

Notes are taken in Etherpad pages named "WMCS-YYY-MM-DD". Following the meeting, the contents of the etherpad are archived to officewiki and a new pad is started for the following week. See team calendar in google calendar for schedule.
We used to keep a template for populating a new weekly etherpad here, but keeping that template up to date with changes in quarterly goals and other structural changes has not worked well. Current best practice is to copy the prior week's notes forward to the next week and remove content such as week summaries that are not applicable to the following meeting.
See also Wikimedia_Cloud_Services_team/Clinic_duties.