nyuuzyou/wikis huggingface dataset:
Npm registry

The @wikimedia org on npmjs.com is used to publish Node.js packages for use outside production, such as dev packages used in CI and local development, and as one of the distribution methods of our general purpose libraries.
== Access control ==
To help with access control, contact one of the group owners, which are as of January 2024:
# catrope
# edg2s
# jdforrester
# krinkle
# santhosh.thottingal
# volker_e
The preferred default group for newly invited members is developers.
Security admin as of January 2024: sbassett29.
== Packages ==
To add an existing unprefixed package to the wikimedia org, log-in to npmjs.com as someone who is both a co-owner of that package and a member of the wikimedia org.
Via https://www.npmjs.com/settings/wikimedia/teams/, select a team, then under "Packages" for that team, you can "Add Existing Package". 
== See also ==
* mw:Developers/Maintainers#Libraries - list of JS and PHP libraries with their maintainers.
* https://doc.wikimedia.org/ - documentation of libraries
== External links ==
* https://www.npmjs.com/org/wikimedia - package list
* https://www.npmjs.com/settings/wikimedia/members - acces control (restricted)
Nutcracker

Nutcracker (also known as twemproxy) is a lightweight proxy for memcached and Redis. It was primarily built to reduce the connection count between web servers and backend hosts. 
== Service ==
* Provisioned by Puppet (nutcracker module, and mediawiki::nutcracker).
* Monitored by Icinga.
* Logs are not stored. (?)
== Usage ==
Installed on Application servers, Changeprop pods, Thumbor pods and API Gateway pods.
== See also ==
* Memcached
* Redis
== External links ==
* https://github.com/twitter/twemproxy
Category:Caching
Category:MediaWiki production
Category:SRE Service Operations
ONFIRE/Incident doc review workflow

Process for grooming our Phabricator board:
Review both wikitech and gdrive for new documents:
* Incident documentation
* https://drive.google.com/drive/folders/1NaGMm8KzCWkVtQm4NbnS49ifbbbTO9bM (you're only concerned with 'proper' incident docs, not status documents)
Any new incidents, create a task or tag the task with #SRE-OnFire. Task title is the title of the document, body is a link to the document & the names of authors. Put it in the "Pending Review & Scorecard" column. Assigned tasks should be moved to the Phabricator work-board column with the corresponding quarter when the incident occurred (example: #SRE-OnFIRE-FY2021-2022-Q2) 
=== General Guidelines ===
* Incidents should be documented and scored within two weeks of the incident occurring.
* Any documents that have been unassigned for longer than a week after it happened should be assigned promptly, work with the IC for that incident to determine who is the best person to document and score the incident.
* For documents that are in-review, ask on IRC to find reviewers for them.
* If a reviewer hasn't made progress on a doc in a week, feel free to ping them.
* If a document author hasn't replied to review comments in a week, feel free to ping them.
* Once the document and review process is complete, ensure the pertinent documentation is updated in Wikitech as applicable.
* Once review comments have been addressed, close the task in the same column pertaining to the quarter when the incident occurred.
ORES/Prepare deployment

== Push LFS data to gerrit ==
There's a known issue with LFS data not being mirrored via phabricator's diffusion to gerrit (Phab:T212818). So we need to manually do that ourselves for the time being. 
Using a local version of the repository with a github.com origin, you need to pull down all the LFS content and then push it to gerrit directly. 
=== Add a `gerrit` remote to your github.com clone ===
For example, for the editquality repo, we have the following lines in ./.git/config.
[remote "origin"]
 url = https://github.com/wikimedia/editquality
 fetch = +refs/heads/*:refs/remotes/origin/*
[remote "gerrit"]
 url = https://gerrit.wikimedia.org/r/scoring/ores/editquality
 fetch = +refs/heads/*:refs/remotes/gerrit/*
=== Pull and push LFS data ===
For each repository where changes were made in LFS and the repos main origin is on github, you'll need to pull and push LFS data. 
$ git lfs fetch --all
$ git lfs push gerrit master
== Update the deploy repository ==
# Update all of the submodules
#* In the deploy repo, cd submodules/<submodule> and run git pull origin master
# If necessary, make any updates to requirements.txt in the deploy repo
# If necessary, make any changes to config/00-main.yaml -- e.g. add new models
# Update frozen-requirements.txt
## In the deploy repo, run make -- this will re-generate the complete set of dependencies for running ORES
## Review frozen-requirements.txt with git diff frozen-requirements.txt to see if any strange new packages appeared or any package was downgraded. 
##* Note that sometimes, strange versions appear when requirements.txt was not updated or the submodules have requirements.txt that disagree on versions. Double-check each of those requirements.txt files if you see anything weird.
== Update wheels == 
# Copy the new wheels to the wheels repo
#* From a clone of the `wheels` repo, run cp ../deploy/wheels/*.whl .
# Update the wheels repo
#* run `git status`
#** for modified files, we're going to just re-check them out git checkout <filename>
#** for "untracked" files, we're going to remove the old version of the same package and add the new one
#*** `git rm <old-file> && git add <new-file>` -- e.g., `git rm revscoring-2.5.2-py2.py3-none-any.whl && git add revscoring-2.6.1-py2.py3-none-any.whl`
#** if no old version exists, this is a new dependency so you can just `git add <new-file>`
# Submit a new patchset with git review -R
== After the change to wheels is merged: Submit the deploy repo changes for review ==
# Update the wheels submodule
#* From the clone of the deploy repo, cd submodules/wheels and run git pull origin master
#* Then cd ../..
#* And git commit
# Submit a new patchset 
#* From the clone of the deploy repo, run git review -R
Office virtualization

== Creating virtual machines ==
# Connect to sfo-virtual1.corp.wikimedia.org either using VNC, or using SSH and X11 forwarding
# Launch the virtual machine manager:
#: virt-manager
# Connect (by double clicking) to "localhost (QEMU)", not to "localhost (QEMU Usermode)". The latter installs VMs as your account using usermode linux, which is slow and ties the VM to your account. The former uses KVM, and runs the VM with hardware acceleration at the system level.
# Click "New", while ensuring you have "localhost (QEMU)" selected
## Give the VM a name, and select "Local install media"
## Select "Browse" under "Use ISO image"
## Either select an already used ISO, or click "Browse Local" to use a new ISO; the location of the media is /var/opt/iso
## Select the OS type and version; don't use generic unless you have no other choice
## Enter the desired amount of RAM. Generally leave the number of CPUs at 1, unless you have a really good reason to do otherwise.
## Give the desired amount of disk space. Try to be reasonable here. 30GB is usually more than enough. You can always expand disks or attach more later.
## Uncheck "Allocate entire disk now"
## Select "Customize configuration before install"
## On the last step in the installer, expand "Advanced options"; for networking select "Specify shared device name", and type in br0 for "Bridge name"; click "Finish"
# In the details view of the virtual machine, under "Boot options", ensure "Start virtual machine on host boot up" is selected.
== Troubleshooting ==
=== Keyboard mapping issues ===
# virsh list
#* Note domain you wish to edit
# virsh edit
#* Ensure the graphics section has "keymap='en-us'" set, for instance:
#*: <graphics type='vnc' port='-1' listen='127.0.0.1' keymap='en-us'/>
# shutdown and start the guest
# rdesktop works well if you are using an Ubuntu client to connect to the Windows VMs.
Offsite Backups

Planning notes for offsite backups. This is a draft, edit heavily please.
== Overview ==
Our backup strategy has several components:
* replication, for immediate failover in case of hardware or power failure
* snapshots, for fast restoration after data corruption
* off site backups, for protection against undiscovered long-term corruption, malicious destruction of data, and for historical archiving
== Requirements ==
* Old backups are kept around forever (every six months? every year?)
* Backups are off-line and inaccessible to WMF employees, volunteers, hackers, etc.
* Media is durable for decades (or provision is made for periodic copying)
* Archive format will be readable for decades (or provision is made for periodic conversion)
* ...?
== Notes == 
*See also Obsolete:Offsite Backups (old)
*Random discussions
**[Foundation-l] Long-term archiving of Wikimedia content (2009)
**Nanoscale printing of Wikipedia (2012)
Category:Risk management
Ops Onboarding Chats

Onboarding chats are a series of sessions to help new hires understand our not complex infrastructure. 
== Session 1: Requests Flows Overview ==
* Datacentres
* GeoDNS
* Caching (briefly)
* Active/Active and Active/Passive Services
* MediaWiki architecture 
* MediaWiki requests flow
===== Related =====
* Global traffic routing
== Session 2: Dive into MediaWiki and Restbase ==
== Session 3: Monitoring ==
== Session 4: Puppet ==
== See Also ==
*https://office.wikimedia.org/wiki/Operations/Ops_sessions
Category:Onboarding
Ops notifications

The operations team provides notification of maintenance or upgrades being done, emergency outages, and day to day work. Below is a summary of the different means in use by ops.
== IRC ==
;Primary channel -- 
:Use: day to day work and discussions go on here. Updates to the sysadmin log are done from here.
;Secondary channel -- 
:Use: during an outage, someone is usually monitoring and providing updates to this channel, while watching for reports of problems.
;Other:
:For sensitive matters that involve possible security or privacy issues, discussion is carried out elsewhere, for obvious reasons.
== Server Admin Log (SAL) ==
Use: powercycling machines, restart of services, upgrades or configuration changes etc. are logged to the Server Admin Log. During long outages brief status updates will be provided here as well.
== Mastodon ==
Everything posted to the Server Admin Log by stashbot is also automatically mirrored to the Mastodon account @sal@wikimedia.social.
== Wiki ==
Outage post-mortems are published on-wiki as incident reports.
== Phabricator ==
When bugs involving operational issues are opened, followup will be done here (publicly available at Phabricator)
== Email ==
There is no single public email mechanism currently in use for updates during site outages or after recovery. (Should there be, or do the above cover us?)
== Gerrit ==
You can look at Gerrit changes and see what has been merged in ops-related projects, f.e. operations/puppet and other operations/* repositories.
Category:Wikimedia infrastructure
Osmium

osmium is a server in eqiad for testing HHVM.
=== Fingerprints ===
[osmium:~] $ gen_fingerprints
+---------+---------+-------------------------------------------------+
| Cipher | Algo | Fingerprint |
+---------+---------+-------------------------------------------------+
| RSA | MD5 | 44:a3:fc:0d:44:c7:e7:f2:b6:49:3f:57:0e:cd:8b:66 |
| RSA | SHA-256 | VO2NgV+neaI+0knmR6Pn7Kk/rwRvjjHvIqnnPnkkGWc= |
+---------+---------+-------------------------------------------------+
| DSA | MD5 | 23:f7:eb:4c:a7:2e:4c:86:bb:6b:1f:d7:c6:c4:cd:94 |
| DSA | SHA-256 | GyMzI1lodQQauqtXDYep5JfTF44Cs0Ttf9w/TNMDXGw= |
+---------+---------+-------------------------------------------------+
| ECDSA | MD5 | 19:a3:a5:89:84:32:0a:87:16:ca:44:ff:45:69:40:be |
| ECDSA | SHA-256 | kkhQHsTdXiRaiUrLHFUw6feZBWT25jV47y3BQ91ndeI= |
+---------+---------+-------------------------------------------------+
| ED25519 | MD5 | 56:10:8c:7f:3e:ec:e5:05:e7:1f:86:40:01:34:58:d7 |
| ED25519 | SHA-256 | CMF/OTQX9I4UMnLK8jbuGUcy9/BZL0I4OKK8UWvjcqA= |
+---------+---------+-------------------------------------------------+
Out-of-band network

All of our sites have a dedicated Out Of Band (OOB) network, consisting of internet access (DIA), routers and switches physically separated from our production network.
This OOB access is our last resort solution in case of a site becoming partially (eg. bastions) or fully (eg. transport/transits) unreachable by normal means.
Note that in the event of a site losing connectivity to the internet, it's still reachable through its transport links from any other bastion host.
== Prerequisites ==
To use our OOB network you need to have:
# An account on the network devices, see the users list in common.yaml
# The password store checked out on your local machine
== Step by step ==
The entry points to SSH to our management routers over their DIA are:
Sites OOB
site 	hostname
eqiad	mr1-eqiad.oob.wikimedia.org
codfw	mr1-codfw.oob.wikimedia.org
esams	mr1-esams.oob.wikimedia.org
ulsfo	mr1-ulsfo.oob.wikimedia.org
eqsin	mr1-eqsin.oob.wikimedia.org
drmrs	mr1-drmrs.oob.wikimedia.org
magru	mr1-magru.oob.wikimedia.org
NOTE: You should make sure that your SSH config does not have any entries that will try to connect to the above hostnames via one of our bastion hosts. To ensure your connection cannot be affected by any in-band changes to the network you need to ensure it goes directly from your machine to the OOB link over the internet.
From there you can SSH directly to relevant devices on the OOB network, for example a console server, eg:
mr1-eqiad> ssh root@scs-a8-eqiad.mgmt.eqiad.wmnet
Where you will need the matching password from the password store.
Note that it's also possible do to agent forwarding and port forwarding on the management routers to reach devices with password authentication turned off.
==== SSH Algorithms ====
On some of the more recent JunOS versions you may find the management router cannot connect to older devices, due to the support algorithms/ciphers supported on the destination device not being allowed by default:
 cmooney@mr1-eqiad> ssh root@scs-f8-eqiad.mgmt.eqiad.wmnet 
 Unable to negotiate with 10.65.0.4 port 22: no matching MAC found. Their offer: umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
To work around this you can drop to a csh shell and use the regular ssh client, specifying the correct parameters manually:
 cmooney@mr1-eqiad> start shell 
 % 
 % ssh -m hmac-sha2-512-etm@openssh.com root@scs-f8-eqiad.mgmt.eqiad.wmnet
 Password: 
 #
=== No DNS ===
If DNS resolution doesn't work (for internal or external hosts), you can replace the FQDNs with the IPs listed in our Puppet configuration (OOB, scs, routers, etc)
To protect against the eventuality of being unable to resolve wikitech-static.wikimedia.org to reach this page, you can save a copy of this page locally or look it up in the Wayback Machine. In both cases, there is of course a risk that the last saved version is outdated; also, you would have to know about these workarounds on your own, being unable to look them up here on Wikitech.
== See also ==
* tunnelencabulator - alternate-PoP IP address ssh tunnel management tool
Category:Wikimedia infrastructure
PAWS

PAWS: A Web Shell (PAWS) is a Jupyter notebook deployment hosted by Wikimedia. 
A Jupyter notebook is a popular Open Source tool that allows users to create and share documents that contain live code. You can use Jupyter notebooks to run scripts that help you perform essential technical tasks on wikis, use data to create visualizations, graphs, and more, and to write technical documentation and tutorials that help others work on Wikimedia projects. 
Jupyter notebooks are used both by experienced programmers and technically curious newcomers. There's no need to download software or set up a development environment. All of your work is done in your browser. 
Our deployment is using JupyterHub, which can serve live Jupyter notebooks to multiple users.
Get started now!
 
PAWS Documentation
PAWS/Getting started with PAWS

This page contains information about how to get started with PAWS.
== Overview ==
The page contains a link to a comprehensive notebook based tutorial that you can follow to get started with PAWS and navigate the control panel. This page also includes a Quickstart guide for those who want to get started right away.
== Prerequisites ==
Since PAWS is hosted in the cloud and requires no installation, you can use PAWS without installing any software. Some programming knowledge can be useful but is not necessary to use the service for many purposes.
== Notebook based tutorial ==
* Get Started with PAWS - This notebook based tutorial provides a comprehensive walkthrough of the PAWS service.
== Quickstart ==
The following steps will help you get logged in and perform some very basic tasks with PAWS.
# Open PAWS
# Click the Sign in with MediaWiki button.
#* This is your Wikimedia Single User Login (SUL). If you don't have a Wikimedia SUL account, you'll need to create one first.
# Allow permission for PAWS to perform some actions on your behalf.
===Create===
Create a new or upload an existing notebook.
# Click the New button and choose your notebook type. You can choose Python 3, Bash and R or folder, text file or terminal.
===Share===
Share your notebook using a public link.
# Alter the URL manually to make your notebook public: https://public-paws.wmcloud.org/User:YOURUSERNAME/YOURNOTEBOOK.ipynb
===Fork===
Fork an existing notebook
# Get the URL of another public PAWS notebook. Example: https://public-paws.wmcloud.org/YOURUSERNAME/YOURNOTEBOOK.ipynb 
# Add ?format=raw to the end of the URL to download a raw .ipynb file. Example: https://public-paws.wmcloud.org/YOURUSERNAME/YOURNOTEBOOK.ipynb?format=raw 
# Log into your PAWS account and use "upload" to upload this copy into your own directory.
PAWS/PAWS and Pywikibot

Learn how to use PAWS to run Pywikibot scripts to perform automated tasks on wikis.
== Overview ==
Pywikibot is a Python library and collection of tools that automate work on wikis. Originally designed for Wikipedia, Pywikibot is now used throughout the Wikimedia Foundation's projects and on many other MediaWiki wikis. 
This page will help you understand what you can do with Pywikibot in PAWS and provides a link to a notebook based tutorial that you can follow to learn how to perform some basic tasks on wikis.
=== What can you do with Pywikibot? ===
If you want to perform maintenance and tasks on wikis that would be difficult to do manually or learn more about wikis you are working with, Pywikibot can be very useful to you. Some examples include giving you the ability to create multiple pages at once, add text and categories, add labels to Wikidata, and much more. 
Because Pywikibot is very versatile and makes it possible to perform many different types of tasks on wikis, you may find it useful to explore existing notebooks. 
* This page of example notebooks, how-tos, and recipes provides a diverse selection of notebooks that have been created by PAWS users. 
* The public index of all PAWS notebooks is also a good way to see how others are using PAWS. Note: This list is not organized by notebook type, so searching it is truly a matter of exploration.
====scripts====
You can find an extensive list of scripts in the Pywikibot manual.
== Notebook based tutorial ==
=== General ===
PAWS & Pywikibot tutorial - This tutorial provides information about Pywikibot with PAWS and a walkthrough that will get you started using Pywikibot with Python 3 notebooks and the terminal in PAWS.
=== Wikidata ===
Wikidata Pywikibot Tutorial - This handbook is a summary of d:Wikidata:Pywikibot - Python 3 Tutorial with some parts taken to other tutorials. It is built for quick reference. It is recommended you read the tutorials before using it.
PAWS/PAWS maintenance and administration

Learn more about how PAWS is administered and maintained.
== Overview ==
This page contains links to pages related to the maintenance and administration of the PAWS service.
== PAWS source code ==
* PAWS source code - View the PAWS source code on GitHub.
== Administration ==
* PAWS/Admin - This page contains detailed notes and diagrams about the administration and architecture of the PAWS service. 
== Nova Resource==
* Paws
* Paws/SAL
* Tools.paws/SAL
* Tools.paws-beta/SAL
==Icinga==
* Icinga - (access required) This indicates the current health status of the PAWS service.
== Historical notes ==
These are older, potentially deprecated notes about the administration of the PAWS service.
* Chico's notes - A set of historical notes from 2018 about the administration of PAWS
* Historical Admin notes - Notes about PAWS on the old cluster
PAWS/Python with Pip

PAWS works great with default Python out-of-the-box, but what if you need to import packages from PyPi?
At the start of your file, add code like:
import sys
!{sys.executable} -m pip install [PACKAGE]
Replace "[PACKAGE]" with the name of the package you want to import. For example, to use mwclient, a lightweight wrapper for the MediaWiki API:
import sys
!{sys.executable} -m pip install mwclient
The next time you run this cell of the notebook, the specified package will be automatically downloaded and installed. On subsequent runs, Pip will detect if a package is already up-to-date and installed, and skip downloading.
PHP packaging

We currently use modified Debian packages to install PHP on appservers and in Docker images. These packages are available from apt.wikimedia.org in a dedicated component for others to use, like CI and MediaWiki-Vagrant.
==Differences from Debian==
Aside from custom patches we want to apply, the packages are slightly different. Starting with PHP 7.4, all packages are named php7.4-foo (instead of the versionless php-foo). This is implemented by patching dh-php:
In an extension package, the binary package name also needs to be changed, including any potential dependencies. Any php-foo.* files also need to be renamed to php7.4-foo.*
==Initial forking==
If you want to fork a new package from Debian, clone the Git repository from Salsa, adjust the packaging as explained above, and build the package:
== Modifying a package ==
If you need to patch or modify a package already in our repository, use the latest .dsc to create a new Git repo. You can find the URL of the .dsc by looking in the pool.
Category:Package management