nyuuzyou/wikis huggingface dataset:
PKI/Policy

== ROOT CA ==
The root CA is managed on a dedicated server. New intermediated certificates need to be created on this server and copied into puppet manually as described on the root CA page
=== Certificate Parameters ===
The root Ca is configured with the following parameters 
* Algorithm: ecdsa-with-SHA512
* Size: 521
* Key Usage: Certificate Sign, CRL Sign
* pathlen: N/A
* Expiry: 10 years
== Intermediate CA == 
The intermediate CA's are managed via puppet with the private key distributed via the puppet private repo. Hosts with a puppet agent certificate are able to requests certificates via the cfssl api at https://pki.discover.wmnet:8888 or by using the puppet profile profile::pki::client 
=== When to create a new intermediate CA ===
For systems or services which just need a certificate to expose an https end point internally such as services set up with DNS/Discovery records. Then the advice is to just use the discovery intermediate CA.
If you want to set up a service which requires mutual TLS via client auth then you should set up a new intermediate CA for that specific service. This means that a compromise to one mutual TLS client certificate doesn't give access to all client auth protected services. We currently have intermediates for:
* DebMonitor
* kafka
* cloud RabbitMQ
* etcd
=== certificate parameters (defaults) ===
* Algorithm:ecdsa-with-SHA512
* Size: 521
* Key Usage: Certificate Sign, CRL Sign
* pathlen: 1
* Expiry: 5 Years
=== Default signing policies ===
By default intermidiate CA's are use the following defaults for all signing requests.
The defaults can also be overridden by specifying a profile to use. by default we configure additional ocsp and server profiles
* Expiry: 4 weeks 
* Key Usage: digital signature, key encipherment, server auth
* OCSP URL: "http://pki.descover.wmnet/ocsp/$CA_CN",
* CRL URL: "http://pki.descover.wmnet/crl/$CA_CN",
Overtime we hope to reduce the expiry down to 24 hours however we would like to get more operational experience first
==== OCSP profile ====
This policy is only used for creating the ocsp signing certificate for the specific intermediate CA
* Key Usage: digital signature, ocsp signing
* Expiry: 43800h
==== Server profile ====
This policy is only used for creating the ocsp signing certificate for the specific intermediate CA
* Key Usage: digital signature, key encipherment, server auth
* Expiry: 4 weeks
== OCSP Responder ==
Currently the OCSP responder runs on the same host as intermediate signing server. We currently maintain a patch so that the ocsp refresh services is able to work with the same databased used but cfssl-multirootca
== CA Bundles ==
Ca bundles are maintained and available via http://pki.discover.wmnet/bundles/$ca.pem. Please note the shortcomings we currently have around bundles and intermediated certificates documented on the Root ca page
Category:SRE Infrastructure Foundations
Packet-loss

==Packet-loss program==
The packet-loss program sends aggregated data to the log at a fixed
interval, currently every 10 minutes. First it shows totals: the number
of "invalid" packets, which didn't have a valid sequence number, the
number of out-of-order packets, and the number of lost packets. These
totals are averaged over the 10 minute period. Then it shows a breakdown
of lost packets by squid proxy server.
==Output==
Counting is done by sampling the log stream, currently taking 1 in every
10 packets. All the figures are shown as percentages, with standard
errors attached. These errors arise from sampling, and are calculated as
a Poisson standard deviation, sqrt(N) suitably scaled. The error is
larger when there are very few packets. So a line like this:
[2010-09-14T02:00:47] knsq1.knams.wikimedia.org lost: (-25.00000 ±
31.25000)%
is normal, and indicates that of the very few packets that knsq1 sent in
this time period, more packets were received than would have been
expected based on the gap between sequence numbers.
The log is configured in the ud2log config and currently points to
 /a/squid/packet-loss.log
==Link to UDP2LOG==
Udp2log is
single-threaded, we're still waiting for someone to rewrite it to use
multiple threads. Every output target (pipe processor, file, etc.)
receives the same log stream. If something is too slow, packets are
dropped on the input side. So measuring packet loss at any output is
sufficient to calculate packet loss at the other outputs.
To minimise packet loss, it's important that the udp2log thread is as
fast as possible and does as little processing as possible. The pipe
processors can be slower, because thanks to the 64KB pipe buffer, they
can run in parallel, on separate cores. That's why the packet loss
analyser had to run as a separate process, not as an integrated part of
udp2log.
Category:Bot and monitoring
Parser cache

Our implementation of the MediaWiki parser cache primarily uses MariaDB, with hot keys also residing in Memcached. 
The configuration is in CommonSettings.php () as part of a MultiWrite wgObjectCaches key. Although the mapping between parser cache sections and DB hosts used to be configured in mediawiki-config as well, this is now managed with dbctl.
File:Wikipedia Memcached flow 2022.png|ParserCache in Memcached (tier 1 and tier 2).
File:Wikimedia-relational-databases-2022.png|alt=|ParserCache DB (tier 3). MediaWiki uses consistent hashing to distribute entries over the pc1, pc2, and pc3 sections.
== Current configuration ==
* Cache retention period is set to (as of 2021-06-07 - see T282761):
** 21 days for wiki pages
** 10 days for talk pages
* Semi-random tasks from 2021 containing various current knowledge about parser cache: T282761. 
* Grafana: Parser Cache
* Each data center has its own parser cache; since multi-DC MediaWiki, it is possible for eqiad and codfw to consistently respond with different parser cache contents (can be tested with WikimediaDebug). A purge appears to correctly purge the parser cache in both data centers. Lucas Werkmeister (WMDE) (talk) 14:11, 13 October 2023 (UTC) 
== History ==
Prior to 2011, the parser cache was solely memcached-based.
In 2011, we introduced the MySQL-based parser cache. See Parser cache expansion and Disk-backed object cache for details of the 2011 project. 
In 2020, an additional local on-host Memcached tier was enabled for parser cache (T264604).
Category:Caching
Category:MediaWiki production
Category:MySQL
PartMan/AutoRaid

Back to PartMan
This allows preseeding of a fully configured RAIDed system by setting
partman-auto/method to "raid" and using partman-auto-raid/recipe to tell
it how to construct the RAID array(s).
The preseedable parameter partman-auto-raid/recipe should consist of
recipes separated by periods ("."). The recipes are of the form:
<raidtype> <devcount> <sparecount> <fstype> <mountpoint> <devices> <sparedevices>
where <devices> is a list of the devices to make up the RAID and
<sparedevices> is a list of the spare devices in the array; devices are
separated by hashes ("#").
For example:
d-i partman-auto-raid recipe string \
 1 2 0 ext3 / /dev/sda1#/dev/sdb1 . \
 1 2 0 swap - /dev/sda5#/dev/sdb5 .
This must be written all on one line; if you write it across multiple lines
you must use continuation characters (as in the example).
This makes the first RAID device that was created by this udeb be a RAID1 of
two disks disc0/part1 and disc1/part1 and be formatted as ext3 and mounted
on /. The second is also a RAID1 and used as swap (and the mountpoint
ignored).
-- Simon Huggins <huggie@earth.li>
sponsored by Black Cat Networks http://www.blackcatnetworks.co.uk/
It is also possible to preseed LVM over RAID. The following example will
create a RAID 1 setup on two hard drives with:
 /dev/md0 as /boot (ext3)
 /dev/md1 as a LVM physical volume
 /dev/mapper/<hostname>-root as / (ext3)
 /dev/mapper/<hostname>-swap_1 as swap
 /dev/mapper/<hostname>-home as /home (ext3)
--- 8< ---
d-i partman-auto/method string raid
d-i partman-auto/disk string /dev/sda /dev/sdb
d-i partman-auto-raid/recipe string			\
	1 2 0 ext3 /boot				\
		/dev/sda1#/dev/sdb1			\
	.						\
	1 2 0 lvm -					\
		/dev/sda5#/dev/sdb5			\
	.
# In the expert_recipe, there is a stanza for each RAID partition
# and each LVM partition.
# The RAID partitions are tagged as "lvmignore", while
# the LVM logical volumes as "defaultignore" and "lvmok".
# The RAID partition containing the LVM volumes must be made big
# enough to hold them all.
# Note that in the example shown here the mountpoint for /boot
# (which is not encapsulated within LVM) is specified in the
# partman-auto-raid recipe, rather than in the corresponding 
# raid partition definition below.
#
d-i partman-auto/expert_recipe string			\
	multiraid ::					\
		100 512 256 raid			\
			$lvmignore{ }			\
			$primary{ }			\
			method{ raid }			\
		.					\
		900 5000 4000 raid			\
			$lvmignore{ }			\
			method{ raid }			\
		.					\
		700 5000 4000 ext3			\
			$defaultignore{ }		\
			$lvmok{ }			\
			method{ format }		\
			format{ }			\
			use_filesystem{ }		\
			filesystem{ ext3 }		\
			mountpoint{ / }			\
		.					\
		64 512 300% linux-swap			\
			$defaultignore{ }		\
			$lvmok{ }			\
			method{ swap }			\
			format{ }			\
		.					\
		100 1000 1000000000 ext3		\
			$defaultignore{ }		\
			$lvmok{ }			\
			method{ format }		\
			format{ }			\
			use_filesystem{ }		\
			filesystem{ ext3 }		\
			mountpoint{ /home }		\
		.
--- >8 ---
Category:How-To
Partial page caching

See the RFC at mw.org for current-day information.
----
Both Varnish and Squid support edge side includes (ESI), although allegedly Varnish's support is limited.
If implemented, Mediawiki could send out sections of a page like so:
<esi:include src="http://en.wikipedia.org/wiki/Special:CentralNotice?action=gimmestuff" onerror="continue"/>
This would be interesting to have as a parser feature, so that dynamic content in a page can be included without invalidating the cache of the entire page. This would let us do lots of cool things like:
* Dynamically changing content based on the user's location
* Instant banner loading for CentralNotice
* Partially cache pages for logged in users to optimize page loading
* SVG client-side rendering
Gabriel Wicke may have experimented with this at one time. Artur Bergman/Wikia has played with it too.
Pascal

Pascal is the management server of the Kennisnet cluster. It is a Sun V40z dual CPU, 2 GB memory and 2 disks.
== Services ==
* pdns_recursor, as a primary resolver for knams hosts
== Failure ==
On October 24, 2005, Pascal crashed and didn't come up. Power cycling didn't help, console didn't show any sign of life.
:its network seems to have broken. i reconfigured it without vlan and it's up again.
...
Dead again November 13, 2008.
Password and 2FA reset/Confirming identities

== How to confirm identities ==
In the best case, the user has a committed identity, and can prove their ownership of the account. After the user confirms their identity by presenting the string/file used to generate their hash, they should regenerate their committed identity (since they are no longer the only person who knows it).
In other cases, you will need to do some research to find out if the person who is making the request really is the owner of the account. Here are some strategies for confirming account ownership:
* A user's identity and request can be confirmed by a trusted user. If a Steward or other trusted functionary knows the user in real life, and can verify the situation with the user (via a phone call, or other difficult-to-spoof mechanism) then we can trust these references and reset the user's password.
* Other ways that we have verified user's identities. If more than one of these can show a user's identity, we can be more sure that the request is legitimate:
** Other accounts known to be controlled by the user (bot accounts) can make a requesting edit
** Accounts with known relationships (UserAccount1 specified that they were creating an account UserAccount2 for a family member. Both accounts have the same email address. UserAccount2 makes an edit vouching for the change request of UserAccount1).
** If the user has a static IP, and a checkuser can verify that the user and the request come from the same IP.
** A user's old (pre-SUL) email address can sometimes be found in a local wiki's user.user_email column. That is sometimes useful as an additional voucher in case an old email should be restored.
== Process to re-claim accounts ==
* Lock the account for a few days so if the request isn't legitimate, the real owner will likely notice that something is going on.
* Change / set the email address on the account.
* Have the user initiate a password reset, or initialise the reset for the user.
* Notify someone like Trust and Safety?
Paypal

CiviCRM recieves contributions from two pipelines
Users enter the donation pipeline at a banner, gateway, email, etc and are directed at donate.wikimedia.org. There they are given the option of donating by paypal or by credit card.
== Web Checkout ==
Once a user selects paypal they are redirected to paypal servers who will serve the rest of the request. Upon successful completion the donor is redirected to a Wikimedia thank you page.
Under the hood the following things happen:
# PayPal internally processes the transaction and pushes it to their IPN service.
# IPN pushes the contribution to our gateway at the following configured url.
# We check the structure of the contribution and if everything looks ok we resubmit it back to paypal as a verification.
# Upon a successful verification paypal responds with the the 'VERIFIED' string and we insert the contribution into civi.
All relevant code for this is present in the fundcore paypal module.
== Payflow Credit Cards ==
Payflow payments have been disabled since 2011.
Category:Fundraising
Pentaho

This page details some initial investigation into what it would take to install the Pentaho business intelligence platform for collecting stats about the Wikimedia community.
== Context ==
A few open source projects have recently started creating dashboards that show their community's activities. The most common is commits to source control, but also interesting is traffic in other areas such as the mailing list, bug reports, IRC, and so on. There are two flagship implementations of these sorts of statistics: meego (http://wiki.meego.com/Metrics/Dashboard) and Mozilla. Mozilla's dashboard is sadly inaccessible to the general public. 
As a first step towards developing one of these things, I (Bhartshorne 23:37, 11 October 2011 (UTC)) spent some time digging into Pentaho and taking a first stab at setting it up.
== Pentaho structure ==
Pentaho is actually an umbrella project that has a number of components. From our perspective, the interesting ones are the "Pentaho data integration" aka PDI and the "business intelligence server" aka biserver. So far as I understand it, the pdi manages the data from collection through transformation and storage. The biserver does reporting and is the front end with which users interact.
I tried to install the biserver. Most of it was a pretty straight forward path following the meego installation instructions with two exceptions. The download link is wrong - it points to a different part of the Pentaho software stack. What you actually want is the biserver. Once I got that sorted, things went well until I tried to load the admin console; it claimed the configuration was not valid but gave no indications on what to change.
== Links ==
* Commercial Pentaho web page: http://www.pentaho.com/
* Open source Pentaho "Community Edition": http://community.pentaho.com/
* Software download: http://wiki.pentaho.com/display/COM/Latest+Stable+Builds
** Note - sourceforge.net/projects/pentaho/ only downloads the PDI. You need to get the other projects (eg biserver) too.
* Meego's dashboard: http://wiki.meego.com/Metrics/Dashboard
** meego's installation instructions (useful): http://wiki.meego.com/Metrics/Installing_Pentaho
*** note that these instructions are for biserver 3.7. Current version is 3.9. There are some bits that don't match up.
*** download link for the biserver is wrong - it links to the pdi install
* random google doc with pentaho install instructions (not so useful): https://docs.google.com/a/wikimedia.org/Doc?docid=0AdJmocc0fj_EZDJ3YmZiZF83M2RtaHhwcmRk&hl=en
* Pentaho sandbox intro http://sandbox.pentaho.com/what-you-need-to-know/
* The company that's building the Mozilla dashboard (might be interesting to talk to): http://www.webdetails.pt/
Perf

= Observe $pid behavior =
 $ sudo perf record -F 99 -a -g -p $pid -- sleep 5
 $ sudo perf report -n [--stdio]
= Probes Cheatsheet =
List available functions:
 $ sudo perf probe -F -x /usr/sbin/varnishd
Define a probe event for each function named "smf_*":
 
 $ sudo perf probe -x /usr/sbin/varnishd --add 'smf_*'
 
List defined probes:
 $ sudo perf probe -l
 probe_varnishd:smf_alloc (on smf_alloc@storage/storage_file.c in /usr/sbin/varnishd)
 probe_varnishd:smf_free (on smf_free@storage/storage_file.c in /usr/sbin/varnishd)
 probe_varnishd:smf_init (on smf_init@storage/storage_file.c in /usr/sbin/varnishd)
 probe_varnishd:smf_open (on smf_open@storage/storage_file.c in /usr/sbin/varnishd)
 probe_varnishd:smf_open_chunk (on smf_open_chunk@storage/storage_file.c in /usr/sbin/varnishd)
 probe_varnishd:smf_trim (on smf_trim@storage/storage_file.c in /usr/sbin/varnishd)
Trace smf_* function calls:
 $ sudo perf trace --no-syscalls --ev 'probe_varnishd:smf_*'
 0.000 probe_varnishd:smf_alloc:(461af0))
 0.038 probe_varnishd:smf_alloc:(461af0))
 0.194 probe_varnishd:smf_free:(461630))
 [...]
Trace smf_alloc duration:
 $ sudo perf probe -x /usr/sbin/varnishd --add 'smf_alloc_exit=smf_alloc%return'
 $ sudo perf trace --no-syscalls --ev 'probe_varnishd:smf_alloc*'
 0.000 probe_varnishd:smf_alloc:(461af0))
 0.022 probe_varnishd:smf_alloc_exit:(461af0 <- 45fc36))
Print function parameters:
 $ sudo perf probe -x /usr/sbin/varnishd --add 'smf_alloc_params=smf_alloc $params'
 $ sudo perf trace --no-syscalls --ev 'probe_varnishd:smf_alloc_params'
 0.000 probe_varnishd:smf_alloc_params:(461af0) st=0x7f24910b43c0 size=0x118)
 0.024 probe_varnishd:smf_alloc_params:(461af0) st=0x7f24910b43c0 size=0x11e)
Performance.wikimedia.org

performance.wikimedia.org is the portal for Wikimedia Peformance, currenlty maintained by MediaWiki Engineering and SRE/Observability.
== Service ==
Currently hosted on webperf003.
=== perfsite ===
See also Performance/Runbook/Webperf-processor services#perfsite.
=== Arc Lamp ===
:https://performance.wikimedia.org/php-profiling/
MediaWiki collects stack traces using php-excimer and sends them to the Arc Lamp service, which then stores them as log files and also generates interactive SVG flame graphs using the FlameGraph tool. The perf site exposes these files from the Arc Lamp server via an Apache proxy.
=== XHGui ===
:https://performance.wikimedia.org/xhgui/
The XHGui web service is exposed via an Apache proxy.
== See also ==
*Performance
== External link ==
* https://performance.wikimedia.org/
Category:Services
Category:Wikimedia Performance
Performance/Essay/Performance survey (2019)

We started the Performance survey with the intention to identify which metrics best represent a satisfying experience, as defined by real people, with the assumption that generally "faster is better" (which we confirmed, no surprises there!) and with the hope that there is or are metrics available (or computable) in modern browsers that would correlate with a positive experience. Specifically, that we could find whether "page load time" (loadEventEnd) is still a good indicator and what better indicates might be available in browsers today.
== How ==
To give this hypothesis the best chance, we analyzed both the full range of our long-term RUM metrics (such as Time to first byte, DNS latency, redirect count, First paint time, and page load time) as well as temporarily several new metrics. These new metrics included: RUM SpeedIndex, Element Timing for "time to first paragraph", and Resource Timing for the bandwidth size and download time of the first image.
We also included several aspects of metadata that while not directly under our control could help explain or counteract certain biases in the data. For example, a particularly negatively perceived article, or a certain account experience-level or geography that correlated particularly positively or negatively to permit relative analysis within a certain group (e.g. analyze categorically slower experiences separately). Meta data included: page weight (transferSize), CPU benchmark score, wiki hostname, account metadata like edit count (<10 edits, 10-100, etc.), article metadata like page ID and image count, device metadata like country, time of day, connection type (3G/WiFi), and whether a CentralNotice banner was presented on this pageview.
The research project revolved mainly around training a machine learning model with all of the above data as input, with the task to output (or "predict") what a real user's would respond on the perf survey (positive or negative exp), and to identify which metric(s) are most predictive of a good or bad perceived experience.
== Learned ==
In a nut shell, we learned that (blog post):
# the new and upcoming browser metrics did not contain a new metric that represented user experience better than page load time (in fact, most were significantly worse).
# all metrics, including our existing ones, correlate relatively poorly (on the Pearson coefficient scale from 0 to 1, our "best" metric scored a mere 0.14).
# domComplete (page load time) is that "best" one.
# firstContentfulPaint (paint timing) was a very close second.
== Further reading ==
* https://meta.wikimedia.org/wiki/Research:Study_of_performance_perception_on_Wikimedia_projects
* https://techblog.wikimedia.org/2019/06/17/performance-perception-correlation-to-rum-metrics/
== See also ==
* Performance/Essay/Perceived Performance (2018)
* Performance/Essay/Google Web Vitals (2021)