nyuuzyou/wikis huggingface dataset:
Thumbor/DjVu

Our custom DjVu engine leverages the ddjvu utility. Like all multipage engines we have, it is used in combination with the page filter, which simply passes the requested page number to the engine.
The DjVu engine uses a PPM file as an intermediary file to pass to our Imagemagick engine, which is uncompressed bitmap. The reason for that is to ensure the best quality possible. If memory consumption becomes an issue for some files, we could potentially use the compressed TIFF option in ddjvu which uses JPEG compression inside a TIFF container.
Thumbor/GIF

Our Thumbor GIF engine reuses the built-in GIF engine, with minor additions.
== MAX_ANIMATED_GIF_AREA ==
We have some huge animated GIFs on our wikis, and rendering their corresponding animated thumbnails would generate very heavy thumbnails. Which is why, when the source animated GIF is over a certain threshold, we render its thumbnail as a static GIF.
The MAX_ANIMATED_GIF_AREA limit, defined in Thumbor's Deployment Charts configuration, is compared against the surface of the GIF multiplied by its frame count.
== Color palettes and resizing ==
Thumbor's GIF engine uses gifsicle, with default parameters. Which means that the color palette of the original image is used as-is for the thumbnail. While this works fine for animated GIFs that are videos in nature, it doesn't work well with schematics. We have a lot of those, and thin lines against plain backgrounds resize poorly if the palette isn't adapted to incorporate new shades of the line and background's colors mixed in. This is solved by our use of the resize-colors gifsicle parameter.
Thumbor/Runbook

== Testing a specific thumbnail ==
In Thumbor staging, the pods are left in debug mode so there will be details of every step of the query logged. This means that staging is the best place to test library/image upgrades and also to debug images that are failing for whatever reason. Reading the logs for thumbor-8080 on staging will generally make it clear whether an image is somehow invalid (incorrect ICC profile, bad metadata, not meeting specifications of the format etc) or if there is an issue with thumbor or swift. 
It is best to test a variety of formats if possible, especially if major changes have happened to a format that required the integration tests to be altered. In order to ensure we generate a new image, it is recommended to change the size of the image in question by specifying an arbitrary odd number for the px value in the path. 
Some sample paths: 
 /wikipedia/commons/thumb/e/e5/NASA_logo.svg/574px-NASA_logo.svg.png
 /wikipedia/commons/thumb/a/af/The_pedigree_of_Birkbeck_of_Mallerstang_and_Settle%2C_Braithwaite_of_Kendal%2C_Benson_of_Stang_End_(IA_cu31924032192811).pdf/page1-220px-The_pedigree_of_Birkbeck_of_Mallerstang_and_Settle%2C_Braithwaite_of_Kendal%2C_Benson_of_Stang_End_(IA_cu31924032192811).pdf.jpg
 
 /wikipedia/commons/thumb/1/10/Star-Forming_%22Bubble%22_RCW_79.jpg/250px-Star-Forming_%22Bubble%22_RCW_79.jpg
 
 /wikipedia/commons/thumb/8/83/3D_model_of_the_human_mouth.stl/1283px-3D_model_of_the_human_mouth.stl.png
These can be tested against staging using
 curl http://staging.svc.eqiad.wmnet:8800/${IMG_PATH}
In production, appending these paths to https://upload.wikimedia.org will hit Thumbor directly. 
Category:Runbooks
Thumbor/STL

== Rendering ==
STL rendering is done by our custom utility called 3d2png, written in node. It shares rendering code with our client-side STL viewer. Since STL files contain 3D content, they don't define an aspect ratio for a 2D rendering of that content. This is currently hardcoded in the STL Thumbor engine, but could be easily made configurable.
Rendering STL requires xvfb and is done in a buffer specific to the current Thumbor instance, to avoid concurrent Thumbor instances on the same machine having race conditions rendering things in xvfb at the same time.
== Format detection ==
Usually, a media file type can be identified via a magic string at the beginning of the file. Unfortunately that best practice wasn't included in STL, and binary STL files cannot be identified using a magic string.
This required a special workaround for the STL engine, because for memory efficiency purposes, Thumbor only deals with the beginning of the file, to avoid loading in memory the original media in a non-streaming fashion, which can sometimes take gigabytes. As a result, STL is the only format handled by Thumbor in the Wikipedia setup where the file extension is looked at to identify binary STL files.
We look at the extension in our custom Thumbor loaders, and modify the excerpt passed to the STL engine, to make it look like a STL ASCII file. Only the excerpt is modified to "fool" the engine, the download original on disk which 3d2png will consume remains intact.
Thumbor/SVG

Our SVG engine relies on rsvg.
Something critical for SVG support is fonts, which is handled at the OS level. The fonts setup that ensures we have good multilingual SVG support is thus defined in Deployment Charts. 
The SVG engine can received information about the "lang" special thumbnail parameter, which allows the client to specify the language to use in the thumbnail, when an SVG is multilingual. This special parameter is passed via our lang custom Thumbor filter.
Our SVG detection is very lax, because the wide variety of SVGs we host has shown that syntax can vary a lot between SVGs and that valid XML isn't a requirement to render an SVG.
Thumbor/TIFF

TIFFs has 2 custom engines. The vips engine is leveraged when the TIFF is over a size limit, defined by the VIPS_ENGINE_MIN_PIXELS Thumbor configuration value. Otherwise the tiff engine is used (which is really just a thin wrapper for our default custom Imagemagick engine).
Both engines handle multipage support via the "page" filter. If the requested page fails (eg. is out of bounds), we retry using the first page of the document.
We determine the size of the image that will select engine or the other via exiftool, running inside the vips engine. Note that this is a bit of an oversimplified approach, as in a multipage situation, this would look at the size of the first page of the document and not the page actually being requested.
Thumbor/Video

As far as video is concerned Thumbor is only used for thumbnailling, not for video conversion - see Videoscaling. Unlike other file formats that are solely handled by a custom engine, video is handled in Thumbor via a custom loader. This allows us to leverage ffmpeg's seeking capabilities, only fetching the frame we're interested in over the network, rather than the entire video file. Once the frame is extracted, we pass it on to our default (custom) imagemagick engine.
The custom loader reuses the multipage "page" filter to support the seek parameter from the Mediawiki thumbnail URL format. If no explicit seek point is requested, the default is the video's midpoint, just like Mediawiki. If the desired frame fails (explicit seek point or default midpoint), we attempt to fetch the first frame of the video as a last resort.
Ogv and Webm are handled the same. The custom loader decides to trigger or not based on the file extension. This is to avoid having a separate network call just to sniff the content.
Torrus

Torrus is an SNMP monitoring/graphing system. It's like a much more advanced version of Cricket.
== Configuration and setup ==
It's installed on netmon1001. The configuration directory is /etc/torrus/, with the configuration of data sources and graphing configuration in XML files under /etc/torrus/xmlconfig/. Some global Torrus options are set in /etc/torrus/conf/torrus-siteconfig.pl. Torrus display style has been modified via files in /etc/torrus/templates; these files are needed for it to work as they are referenced from the data sources XML files.
=== Discovery ===
A daily cron job is run to update the list of ports on switches and routers, using the files in /etc/torrus/discovery/. The cron job is in /etc/cron.daily and looks like this:
#!/bin/sh
DDXFILES="coreswitches.ddx accessswitches.ddx"
TREES="Network"
for ddx in $DDXFILES
do
	torrus devdiscover --in=$ddx
done
for tree in $TREES
do
	torrus compile --tree=$tree
done
== Common problems ==
=== Deadlock problem ===
When Torrus stops working in any way, that's usually because it ends up deadlocked. The web processes and the collector processes work on the same DBM files in /var/lib/torrus/db. Sometimes they get into a deadlock, presumably because some process died and didn't reduce a lock. Here's the steps to resolve the issue:
 service torrus-common stop
 db5.1_recover -h /var/lib/torrus/db
 sudo -u Debian-torrus torrus compilexml --all --verbose
 service torrus-common start
More detailed information on how to resolve this can be found at torrus.org's site.
== See also ==
* torrus.wikimedia.org
Category:Services
Category:Bot and monitoring
Tw cli

Some of our hardware raid controllers are controlled by the tw_cli command line tool. Most machines that have hardware arrays will have it installed but if not then you can grab it from the vendor 3ware.
Once on a machine you can start interacting with the controller by doing the following
 root@amane:~# tw_cli 
 //amane> ?
 Copyright(c) 2004 Applied Micro Circuits Corporation(AMCC). All rights reserved.
 Copyright(c) 2002, 2003, 2004 3ware, Inc. All rights reserved.
 AMCC/3ware CLI (version 2.00.02.009)
 Commands Description
 -------------------------------------------------------------------
 info Displays information about controller(s), unit(s) and port(s).
 maint Performs maintenance operations on controller(s), unit(s) and ports.
 alarms Displays current AENs.
 set Displays or modifies controller and unit settings.
 sched Schedules bachground tasks on controller(s) (9000 only)
 quit Exits the CLI.
 ---- New Command Syntax ----
 focus Changes from one object to another. For Interactive Mode Only!
 show Displays information about controller(s), unit(s) and port(s).
 flush Flush write cache data to units in the system.
 rescan Rescan all empty ports for new unit(s) and disk(s).
 commit Commit dirty DCB to storage on controller(s). (Windows only)
 /cx Controller specific commands.
 /cx/ux Unit specific commands.
 /cx/px Port specific commands.
 /cx/bbu BBU specific commands. (9000 only)
To check up on the controller use the info command
 //amane> info
 Ctl Model Ports Drives Units NotOpt RRate VRate BBU
 ------------------------------------------------------------------------
 c0 9500S-8 8 7 1 1 2 4 Fault 
 c1 7006-2 2 2 1 0 2 - - 
 c2 9500S-8 8 8 1 1 3 4 OK 
In this case c0 is showing us a serious fault. To find the details of the failure take a look at
//amane> info c0 
 Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC
 ------------------------------------------------------------------------------
 u0 RAID-10 INOPERABLE - 64K 1490.07 OFF OFF OFF 
 Port Status Unit Size Blocks Serial
 ---------------------------------------------------------------
 p0 DEGRADED u0 372.61 GB 781422768 WD-WMAMY12171 
 p1 OK u0 372.61 GB 781422768 WD-WMAMY12171 
 p2 OK u0 372.61 GB 781422768 WD-WMAMY17829 
 p3 OK u0 372.61 GB 781422768 WD-WMAMY11772 
 p4 SMART-FAILURE u0 372.61 GB 781422768 WD-WMAMY12171 
 p5 NOT-PRESENT - - - -
 p6 OK u0 465.76 GB 976773168 WD-WCAS877686 
 p7 OK u0 372.61 GB 781422768 WD-WMAMY12170 
 Name OnlineState BBUReady Status Volt Temp Hours LastCapTest
 ---------------------------------------------------------------------------
 bbu On No Fault TooLow TooLow 0 xx-xxx-xxxx
Once the hardware fault has been identified, please add the repair to the Data Center tasks page.
Two-factor authentication

Two-factor authentication (TFA or 2FA) is a method of confirming a user's claimed identity by utilizing a combination of two different components. Two-factor authentication is a type of multi-factor authentication.
You may be looking for:
* Help:Two-factor authentication
* Yubikey-2FA
* Password and 2FA reset#Reset two factor authentication
* Phabricator#Removing Two Factor Authentication
== See also ==
* Help:Two-factor authentication on mediawiki.org
* Help:Two-factor authentication on metawiki
Category:Disambiguation pages
UID

==UID ranges==
Used UID ranges on Wikimedia systems
UIDs 	Usage 	Notes
0 	root 	
1-999 	System users 	Also includes a few human users for legacy reasons
1000-49999 	Humans 	LDAP is the authoritative source for UID assignments; do not use an UID for a production shell account that is not assigned to the same user in LDAP (LDAP uid = shell username)
50000-59999 	Toolforge tools 	Stored in LDAP, created by toolsadmin.wikimedia.org.
60000-64999 	System users 	Assigned by Debian (see policy § 9.2.2)
65000-65535 	Reserved by Debian	This includes:
* 65000-65533 unused at the time
* 65534 is the user "nobody"
* 65535 must not be used due to compatibility issues with legacy systems (65535 = 2^16 - 1)
65536-4294967293	Unused 	Unused for now. Potential future use includes extra IDs for users/toolforge tools if the current allocations aren't enough
==Reserved UIDs & GIDs==
Althrough we try to keep this up-to-date it the authoritative source is admin.yaml
Make sure to add reservation entry to admin.yaml
If you want the account to be created every where you must reserve the UID in the puppet admin module and use systemd-sysuser. like in this example.
If the user will just exist on a few machines then you should reserve the account in the admin module with a commented block. like in this example, then create the user with a normal user block e.g.
 systemd::sysuser { 'phd':
 ensure => present,
 id => '920:920',
 description => 'Phabricator daemon user',
 home_dir => '/var/run/phd',
 }
*(table columns are sortable)
UID	GID	user name
33 	33 	www-data
48 	48 	apache
107	112	puppet
110	115	nagios
111	116	mwdeploy
130	130	(swift on some old systems, to be phased out by late 2024 cf. T123918)
445	445	rancid
497	498	phd (phabricator) (outdated! on newer servers replaced with 920)
499	499	trebuchet
901	901	reprepro
902	902	swift
903	903	hdfs (previously jenkins)
904	904	yarn
905	905	mapred
906	906	analytics
906	906	analytics
907	907	druid
908	908	hadoop
909	909	analytics-privatedata
910	910	analytics-product
911	911	analytics-search
912	912	analytics-research
913	913	analytics-platform-eng
914	914	minio-user
915	915	git
916	916	kafka
917	917	bgpalerter
918	918	stats
919	919	scap
920	920	phd (phabricator, was 497:498)
921	921	librenms
922	922	doc-uploader
923	923	zuul
924	924	jenkins
925	925	gerrit
926	926	spark
927	927	analytics-wmde
928	928	authdns
929	929	librenms
==Permission/security hierarchy==
the security hierarchy looks as follows as decribed by TimStarling: 
*root > wikidev > mwdeploy > www-data
**root can own wikidev but wikidev can't own root
**wikidev can own mwdeploy but mwdeploy can't own wikidev
**scripts owned by mwdeploy can only be run by www-data
**everything has to su to www-data before running maintenance scripts
also see:
Ubuntu todo

* new tidy
* wikidiff2
* FSS
* /etc/sudoers
* Apache configuration
We can defer these by using the fedora scaling cluster for now:
* ImageMagick is 6.2.4 Q16, should be 6.2.8+, Q8
* librsvg is stock, needs security patches. See /h/w/src/packages/install-librsvg
* Font consistency:
** We have DejaVu fonts 2.14, needs to be upgraded. Debian unstable has 2.19, we could upgrade to that and also upgrade the FC package, for consistency.
** Lots of fonts missing, e.g. Japanese miscellaneous fonts. We probably would benefit from just installing all ~120 font packages
*** Ubuntu 7.04 wasn't much good, but Ubuntu 8.04 is better than what we're using now.
External storage:
* It's possible to use stock MySQL 5.0, but it shouldn't be slaving from 4.1 or use a data directory copied from 4.1 if possible. Lots of migration work to do.
Category:Ubuntu
Udprec

Simple UDP receiver, outputs on stdout. Probably written by River.
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
void 
read_and_echo(int sfd, struct sockaddr *cliaddr, int clilen)
{
	int n;
 socklen_t len;
	char msg[BUFSIZ];
	for (;;) {
		len = clilen;
		if ((n = recvfrom(sfd, msg, BUFSIZ, 0, cliaddr, &len)) < 0) {
			perror("recvfrom");
			exit(8);
		}
		write(STDOUT_FILENO, msg, n);
	}
}
int
main(int argc, char *argv[])
{
	if (argc < 2) {
		fprintf(stderr, "usage: %s <port> [host]\n", argv[0]);
		exit(8);
	}
	int sfd;
	struct sockaddr_in servaddr, cliaddr;
	if ((sfd = socket(AF_INET, SOCK_DGRAM, 0)) < 0) {
		perror("socket");
		exit(8);
	}
	memset(&servaddr, 0, sizeof(servaddr));
	servaddr.sin_family = AF_INET;
 if (argv[2])
 servaddr.sin_addr.s_addr = inet_addr(argv[2]);
 else
 servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
	servaddr.sin_port = htons(atoi(argv[1]));
	if (bind(sfd, (struct sockaddr *) &servaddr, sizeof(servaddr)) < 0) {
		perror("bind");
		exit(8);
	}
	read_and_echo(sfd, (struct sockaddr *) &cliaddr, sizeof(cliaddr));
	exit(0);
}
Ulsfo buildout

Please link to RT tickets for any/all tasks!
If there are subtasks that you would like help/volunteering on, please denote this with Help Needed
==Current blockers==
==Deadlines==
January 7th
: All miscellaneous hardware must be ordered by this time
* https://rt.wikimedia.org/Ticket/Display.html?id=3491
* https://rt.wikimedia.org/Ticket/Display.html?id=3490
* https://rt.wikimedia.org/Ticket/Display.html?id=3735
* https://rt.wikimedia.org/Ticket/Display.html?id=4275
* https://rt.wikimedia.org/Ticket/Display.html?id=4274
January 21st 
: All servers must be ordered by this time.
* https://rt.wikimedia.org/Ticket/Display.html?id=2799
* https://rt.wikimedia.org/Ticket/Display.html?id=3492
* https://rt.wikimedia.org/Ticket/Display.html?id=3493
* https://rt.wikimedia.org/Ticket/Display.html?id=3494
* https://rt.wikimedia.org/Ticket/Display.html?id=3488
February 18
: All network gear correctly cabled and labeled
February 19
: All network gear configured
February 20
: New DNS system completed
: All gear physically racked and attached
February 21
: All DRAC's and remote access configured
February 25th - March 1st
: Hackathon! We will put this cluster into production and serving traffic at this time.
==What is done==
Category:Ulsfo data center
Ulsfo data center

ulsfo (pronounced U-L-S-F-O, or the San Francisco DC) is located at United Layer's 200 Paul Ave data center in San Francisco. It provides edge caching for West USA and East Asia.
== Photo gallery ==
File:Wikimedia racks ULSFO 2013 06 06 02.jpg|New servers arriving
File:Wikimedia racks ULSFO 2013 06 06 07.jpg|Server rack
File:Wikimedia racks ULSFO 2013 06 06 10.jpg|Server rack
File:Wikimedia racks ULSFO 2013 06 20 02 network gear.jpg|Network gear
File:Wikimedia racks ULSFO 2012 10 25 06.jpg|Terminal
File:Wikimedia racks ULSFO 2013 06 20 03 servers.jpg|Server rack
== See also ==
* Pages linking to ""
 
Category:Data centers
Upload.wikimedia.org

upload.wikimedia.org is the hostname used for access to original and scaled-down media files for the Wikimedia projects.
The domain name is separated from wikipedia.org to aid in browser security (around cross-origin, cookies, and JavaScript especially), as well as web performance (through the use of parallel connections).
== MediaWiki ==
When uploading new files from MediaWiki, storage is handled by MediaWiki's FileBackend component (see wmf-config/filebackend). Each wiki has its own Swift bucket, and these are exposed through an Nginx proxy at upload.wikimedia.org. For example, files from English Wikipedia (en.wikipedia.org) are served from upload.wikimedia.org/wikipedia/en/*.
Users mainly upload files through the UploadWizard feature on commons.wikimedia.org. For read-access, the Special:Filepath shortcut redirects to the appropiate path on upload.wikimedia.org.
When articles are edited and saved by MediaWiki, the creation of newly referenced thumbnails or thumbnail sizes is deferred. In production, we turn off $wgGenerateThumbnailOnParse. Instead, it generates a deterministic URL only. We then use the equivalent a 404 handler to have thumbnail servers generate these on-the-fly. On smaller wikis and during local development this can be done with MediaWiki's thumb_handler.php. In production, we route cache misses from the CDN for upload.wikimedia.org to Swift through a rewrite proxy that does the same thing. In case of thumbnail not (yet) existing, it forwards the requests to Thumbor which then responds with the new thumbnail for the CDN cache and the currently-requesting client, and (also) stores it in Swift for future use after it falls out of the CDN cache.
On private wikis, media files are not served from "upload.wikimedia.org", but instead served through the img_auth.php MediaWiki entry point from the private wiki's own domain name (which enforces authentication and access control).
== Thumbor ==
== Media storage ==
== History ==
=== 2008 ===
upload.wikimedia.org has its own set of Squid proxy caches separate from the "text" squids (for HTML/CSS responses). This avoids contention between the two data sets, which have different characteristics for object size, update rate, etc.
=== 2009 ===
IPv6 support was partially added. AAAA records are currently sent to a list of participating resolvers only. This works using the PowerDNS pipebackend, running selective_answer.py. Currently this is Europe (knams) region only, due to lack of sufficient IPv6 connectivity elsewhere! The IPv6 address is served by an IPv6-to-IPv4 proxy (ha-proxy), which simply proxies the HTTP request to the IPv4 LVS cluster, including an X-Forwarded-For header. Unfortunately it does not support doing HTTP processing along with persistent connections.
== External link ==
* https://upload.wikimedia.org
Category:Services
Uploading large files

== Requirement ==
You need:
* the URL of the media file to upload
* a text file with the first revision content
* the name of the user account for this first revision and upload
Our swift backend storage is limited to files smaller than 5 Gb. See phab:T191802 for discussion to extend this limit beyond 5 GB.
== Step 1: download files ==
Download the files to mwmaint1002 (or if there's not enough space, deploy1001).
wget <URL>
At this stage, it could be pertinent to check the hash of the file if known.
Requestors are advised to provide direct link to the to-upload-file. However, ocasionally, they do not do that, and use a public cloud service instead, which usually don't provide direct download links (like Google Drive).
From Google Drive, it is possible to download a file using it's unique ID via rclone:
urbanecm@titanium /nas/urbanecm/wmf-rehosting
$ rclone -P backend copyid <config>: '<fileid>' '<filename>'
where:
* <config> refers to the name of the rclone config entry (you can use rclone config to see/edit the config entries)
* <fileid> is the ID of the file at Google (for https://drive.google.com/file/d/1K9QrMXyhPqlvc-vQRjVmT8YrbgYjfelC/view, the ID is 1K9QrMXyhPqlvc-vQRjVmT8YrbgYjfelC
* <filename> is the name you want to store the file under
Since rclone is not installed at production servers, this requires copying the file first to a temporary location first and then transfering to the maintenance server, however, it does not mean downloading the file to administrator's own laptop (which might have capacity or connection speed issues).
== Step 2: import image to Commons ==
Server-side uploads run much faster because of minimal network overhead, and as a result can cause extra strain on the job queue, especially videos which require transcoding. It's recommended to add some delay in between each upload with the --sleep parameter. Because videos have various factors (resolution, fps, length) that would affect how long transcodes would take, it might be worth uploading one video, seeing how long the median transcode takes, and then sleeping for that length to avoid queuing up a large number of transcodes.
mwscript importImages.php --wiki=commonswiki --sleep=SECONDS --comment-ext=txt --user=USERNAME /tmp/uploads
If you receive errors about files not existing, try making the file world-readable. 
Category:How-To