nyuuzyou/wikis huggingface dataset:
Data Platform/Transform data

__NOTOC__
This page describes the process and internal tools for creating datasets and reports based on private/internal data sources. For info about publicly-accessible resources and data, see meta:Research:Data.
== Before you start ==
This page assumes you have already identified datasets relevant for your analysis, and that you can access and query that data using internal analysis tools.
Before you create a new table or dataset, check the existing data sources and datasets in DataHub to see if the data you need is already there. If not, is there a similar table that could be updated to meet your needs?
== Plan data lifecyle ==
* Dataset creation process
* Data lifecycle management process 
=== Get approval for new data collection ===
If you intend to collect a new type of data or design a new instrument for experimentation or product analysis, follow these data collection policies and procedures to submit and get approval for your data collection activity:
* Data Collection Guidelines (draft, currently internal only, succeeds Instrumentation DACI; will eventually be posted to Foundation wiki)
* WMF staff should use the Legal, Safety and Security Service Center (L3SC) to submit a request to have data collection plans reviewed and approved.
* (draft) guide on measurement plans and instrumentation specifications and Instrumentation process and spec template(Google sheet)
== Model and document your data ==
== Build your table or dataset ==
=== Table formats and storage === 
You can store data in private namespaces in Hive or Iceberg, but product data should be in Iceberg (for exceptions, contact the team).
== Share data and dashboards ==
=== Before you publish any data ===
=== Share queries and visualizations ===
=== Tools and platforms for publishing data externally ===
== Manage published data ==
=== Maintainance and monitoring ===
* Data Incident management
* Data Issue reporting
TODO: are there dashboards where people can check the status of canonical data pipeline generation runs on which their datasets depend?
=== Retention and deletion ===
* Data Retention Guidelines
* Event data retention: data retention practices for events, and privacy best practices for creating or modifying event schemas
* Event Sanitization: processes used with Event Platform data for retaining event data in Hive beyond the standard 90 day retention period.
* Dataset archiving and deletion
Category:Landing page
Category:Data platform
Data Platform/Web publication

This page describes how to make Tier 3: Low Risk (or Tier 2: Medium Risk and sanitized) datasets, notebooks, or other research products public on the web in the analytics.wikimedia.org/published directory – adhering to the data publication guidelines. For guidelines on how to formally release an open dataset (with metadata and persistent identifiers), please refer to Data releases. For regular, structured, and maintained datasets, please see Analytics#Datasets.
If you're looking for data here, some of it may not be maintained or documented. If possible, please reach out to the authors of the data for help, or to Data Platform Engineering team. If you're publishing data here, there are some guidelines in the README on the server.
== Instructions ==
# Double-check that the dataset or notebook you want to publish is a Tier 3: Low Risk (or Tier 2: Medium Risk and sanitized) release according to the official data publication guidelines.
# Decide where you want to publish it. There are separate folders for notebooks and datasets; within those, you should browse the existing subfolders and decide where your code fits. For example, if you have my-data-2020-01.tsv, you may want to publish it as datasets/one-off/my-data/my-data-2020-01.tsv. Please try to use names that the complete strangers viewing the website will understand!
# Make sure you're using on one of the Analytics clients (AKA stat boxes).
# Copy our file to the corresponding location within the /srv/published/ folder on that machine, or in the /wmf/data/published/ folder on HDFS. Create the intermediate folders if necessary. If you're using Analytics/Systems/Jupyter, for security reasons you will not be able to access this file from the terminal in your browser. You'll need to SSH directly into the server and move the file using the command line.
# Log it via data publication log form
Once you do this, it will be automatically synced to the website by a script that runs automatically every 15 minutes. If you want to run the sync immediately, you can do it manually with the published-sync command.
Category:Data platform
Data Platform Engineering/Contact

For most requests, use https://www.mediawiki.org/wiki/Data_Platform_Engineering/Intake_Process.
== Other ways to contact the Data Platform Engineering teams ==
If you have questions about our work or the infrastructure we provide, you can contact us:
*by e-mail: data-engineering@wikimedia.org 
*on our public mailing list, analytics@lists.wikimedia.org (subscribe, archives)
*in our public IRC channel, . You can use the keyword a-team to ping us, so we notice your question.
WMF staff:
* Office Hours: Every two weeks on Tuesday at 15:00 UTC (to find the meeting details please, see the WMF staff calendar)
* Slack: #data-engineering. For urgent notifications use the @data-engineering-team handle in your message
==== Reporting issues and making requests ====
For most requests, use https://www.mediawiki.org/wiki/Data_Platform_Engineering/Intake_Process.
Please use the following steps to report an issue with one of the systems, datasets, or pipelines  maintained by the data engineering team:
*File a Phabricator bug report and tag it as data-engineering
*In case this is an urgent blocking issue, post a note with the Phabricator link on the #data-engineering slack channel using the @data-engineering-team handle, indicating that it is a blocking issue.
*If you don't have access to slack you can do so on the public IRC 
*To escalate further please use the contact information on office wiki to reach out to Emil Chetty or Olja Dimitrijevic
*The team will aim to respond to all reported incidents within 2 hours of posting, between the hours of 9am and 9pm UTC, Mon-Fri. If you encounter an incident that requires urgent attention outside these hours, please use the escalation protocol.  At the time we will assess the urgency of the incident to determine if immediate action is required. 
*For new features and data requests, please use the data engineering request form.
===Work organization===
The Data Engineering team uses Phabricator to track its projects. The following tags are used to classify work within phabricator.
*#data-engineering/ for top-level classification of open tasks and epics.
*#data-engineering-planning/ for sprint planning purposes.
In addition to these regular tags we work across a number of different value streams, each with its own tag.
* #data-pipelines
* #shared-data-infrastructure
* #event-platform value stream
Plus a number of other tags related to major projects, such as
* #data-catalog to track work related to the DataHub implementation
Data Platform Engineering/Decision log

The goal of this page is to encourage Data Platform Engineering teams to make use of quick decision logs to document decisions they are making.
When to log a decision is subjective. Generally, if you might need to refer to a decision in the future, it is a good idea to log it.
Complex decisions should link to documents with more context, ideally in a wiki or in Phabricator. Google docs, Slack, or other links may be helpful too.
If a you think a more focused project or team specific Decision Log will be useful (e.g. Event Platform/Decision log), feel free to make one. Make sure you add a [[Category:Decision log]] category link to your page, so it will show up in the global list of Decision logs on wikitech.
= Data Platform Engineering Decision Log =
ID	Category	Decision Description	Alternatives Considered	Responsible Parties	Decision Date	Comments/Links
 	 	 	 	 	 	
Category:Decision log
Data Platform Engineering/Learning Materials

== Talks and Presentations ==
=== Data Practices and Privacy Engineering ===
* Wikipedia and the Lean Data Diet, Nuria Ruiz
=== General Overview: What is analytics doing back there? ===
Our most recent talk that gives you an overview of the stack:
* What is Analytics Doing Back There?, Dan Andreescu
=== Talks recommended by other members of the Data Engineering team: ===
* Wikipedia’s Event Data Platform, Or: JSON Is Okay Too, Andrew Otto, Confluent Current 2022 ()
* Modern Event platform, Andrew Otto
* The Paramecium Talk, Aaron Halfaker
* Kafka @ Wikimedia foundation, Andrew Otto
* Hadoop and Beyond. An overview of Analytics infrastructure
* What happens when you type el.wikipedia.org (overview of the setup of wikipedia by our SRE team)
Data centers

These are data centers used by the Wikimedia Foundation, in which it has colo space where its operates server clusters.
To read about the history behind current and previous hosting facilities and technology choices, see "Wikimedia Foundation#Technology" on Wikipedia, and Wikimedia servers on Meta-Wiki.
== Present ==
"Caching" data centers serve as points of presence (PoP) for the Wikimedia CDN.
"Application" data centers host MediaWiki servers, databases, and other services. 
Nr	Name 	Location 	Role 	Since
#1	Eqiad	Ashburn, Virginia, USA 	Application + Caching	2010
#2	Codfw	Carrollton, Texas, USA 	Application + Caching	2014
#3	Esams	Amsterdam, Netherlands 	Caching 	2005?https://wikitech.wikimedia.org/w/index.php?title=Kennisnet_rackspace&oldid=13519
#4	Ulsfo	San Francisco, California, USA	Caching 	2014"ULSFO going online", Deployments calendar, Week of 3 February 2014.
#5	Eqsin	Singapore 	Caching 	2017T156028: Name Asia cache DC, Brandon Black, 26 September 2017.
#6	Drmrs	Marseille, France 	Caching 	2022T282787: Configuration for new drmrs datacenter, Brandon Black, 13 May 2021.
#7	Magru	São Paulo, Brazil 	Caching 	2024
Networking
Name 	Location 	Role 	Since
eqord	Chicago, Illinois, United States (North America)	Networking	2015
eqdfw	Dallas, Texas, United States (North America) 	Networking	2015WMF Infrastructure Quarterly Review 2015 FQ1, page 66 "TechOps - Appendix", 15 October 2015T91077: EQDFW/EQORD Deployment, Wikimedia Phabricator, 27 February 2015
== Past ==
Name 	Location 	Usage 	Years
Pmtpa 	Tampa, Florida, United States 	Core services 	2004 – 2014
Yaseo 	Seoul, South Korea 	Core services (zhwiki, kowiki)	2005 – 2009Started on September 2005. Oldest mention in Server admin log, 9 September 2005."Decommissioned all yaseo servers", Server admin log, 23 July 2009.
Lopar 	Paris, France 	Edge caching 	2005
San Diego	San Diego, California, United States	"The server" 	2001 – 2004
Networking
Name 	Location 	Role 	Since
sdtpa	Tampa, Florida, United States	Networking (for pmtpa)	2004 – 2014
== See also ==
* SRE/Infrastructure naming conventions#Data centers
*Infographics - server layout diagrams
== References ==
__NOTOC__ __NOTALK__
Datasets.wikimedia.org

:This repository is now DEPRECATED. The files have been moved to https://analytics.wikimedia.org/datasets/archive and redirects have been set up so no links are broken.
http://datasets.wikimedia.org is a repository for compiled analytics data. These datasets are often used by Limn dashboards such as the EE Dashboard. The site is hosted on thorium, although there are also virtual directories on stat1006 that are copied over to throrium regularly, for example, '/a/public-datasets/' and '/a/limn-public-data/'.
Db1008

db1008 is the master database of the fundraisingdb cluster.
db1008 hosts these databases which are used for fundraising:
* civicrm
* drupal
* dev_civicrm
* dev_drupal
* dev_fundraising
* faulkner
db1008 has two slave databases:
* db1025 (eqiad host): does lvm snapshots
* storage3 (pmtpa host): does lvm snapshots, mysql dumps, also does archival proxy log storage
Category:Fundraising
Db9

Dell PowerEdge 1950
= Active Databases =
== CiviCRM ==
* civicrm - migrated to db1008 and replicated to db1025 and storage 3 in Nov 2012
* drupal - migrated to db1008 and replicated to db1025 and storage 3 in Nov 2012
* dev_civicrm - migrated to db1008 and replicated to db1025 and storage 3 in Nov 2012
* dev_drupal - migrated to db1008 and replicated to db1025 and storage 3 in Nov 2012
== Blogs ==
* blog
* techblog
== Others ==
* otrs - [migrated Dec 2011, db48 --> db49]
* racktables
* scholarships
* bugzilla3
== Disk space issues as of 7/29/2011 ==
The following was obtained by converting a slave to innodb_file_per_table,
then a simple df -k within /a/sqldata:
91752	blog
853972	bugzilla3
798264	bugzilla4
466416	bugzilla_testing
3996536	civicrm [migrated]
3052	communicate
17388	communicate_civicrm
76440	contacts_civicrm
7076	contacts_drupal
3851328	dev_civicrm
2097040	dev_drupal
6604	dev_fundraising
63380	donateblog
2542624	drupal
657032	etherpad
28	faulkner [migrated]
37180	limesurvey
63492	newsblog
46132	observium
10992	ocs_wikimania2009
197330256	otrs [migrated in Dec 2011]
19556	outreach_civicrm
2136	outreach_drupal
58228	puppet
47376	racktables
337664	rt
5388	scholarships
9292	techblog
66380	testblog
Dbtree.wikimedia.org

dbtree.wikimedia.org provides a visualisation of master-slave relationships inside each of the MediaWiki database clusters. Originally written by Asher Feldman.
== Service ==
* Currently runs from mw1152 (check for the role role::noc::site in site.pp).
* Source code: operations/software/dbtree.git
* Uses data from Tendril.
* Configured in puppet:/modules/noc/manifests/init.pp.
== Deploy ==
To deploy a new version, once merged into the repository, ssh into the host where it runs, with the user mwdeploy go to /srv/dbtree and do a git fetch and git rebase.
== See also ==
* tendril.wikimedia.org, a tool for analyzing slow query logs
Category:Services
Category:MySQL
DebMonitor

DebMonitor is a Debian package tracker website and tool developed at the Wikimedia Foundation and used to track installed and upgradable packages across the fleet. It has multiple components.
== DebMonitor website ==
The DebMonitor website is a Django-based application installed in an active/passive setup with uwsgi and nginx on two dedicated Ganeti VMs and deployed via the debmonitor Debian package. The application has two different virtual hosts, one on port 80 for Varnish and one on port 443 for the DebMonitor client (see below). The data is stored on a MySQL database hosted by the m2 cluster.
There is a weekly systemd timer on each host to run a garbage collection script twice a week to remove orphan objects from the database (e.g. package versions not installed anymore in any host).
== DebMonitor client ==
The debmonitor-client Debian package is installed on all the clients and reports the installed packages to the DebMonitor active server. It reports them in four different ways:
* A dpkg hook triggered on Dpkg::Pre-Install-Pkgs to report any change to packages. It doesn't block package actions on failure.
* An APT hook triggered on APT::Update::Post-Invoke to report any upgradable packages. It doesn't block apt-get update on failure.
* A daily crontab to report all installed and upgradable packages to reconcile the data in case any of the above failed.
* A separate component called docker-reporter traverses the Docker registry and submits information on packages present in container images
The client authenticate with the DebMonitor server via mutual authentication using certificate issued by our PKI, and the server authorize the connecting host to modify only its own data.
== Common commands ==
=== Manually remove a host from DebMonitor ===
From one of the cluster::management hosts () run the sre.debmonitor.remove-hosts cookbook. See also Spicerack/Cookbooks#Run_a_single_Cookbook.
Alternatively it can be done manually running:
sudo curl -X DELETE "https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN}" --cert "/etc/debmonitor/ssl/debmonitor_$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor_$(hostname -f | tr '.' '_')-key.pem"
=== Manually remove an image from DebMonitor ===
This will remove all tags of this image from DebMonitor.
From one of the builder hosts run:
# IMAGE_NAME=docker-registry.wikimedia.org/foo-bar-image
sudo curl -X DELETE "https://debmonitor.discovery.wmnet/images/${IMAGE_NAME}" --cert "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_')-key.pem"
Category:Deployment
Category:SRE Infrastructure Foundations
Debian

Debian is the GNU/Linux distribution used on Wikimedia production servers and Cloud VPS instances. It replaced our previous usage of Ubuntu.
* OS Upgrade Policy
* DebMonitor tracking
== Documentation ==
* https://www.debian.org/doc/user-manuals#debian-handbook - Debian administrator manual (multi-language)
* https://www.debian.org/doc/manuals/debian-reference/ - Debian complete reference (multi-language)
== See also ==
* wikipedia:Debian
* https://www.debian.org/doc/manuals/debian-faq/ - Why Debian is so stable, etc.
* https://wiki.debian.org/DontBreakDebian - How to not destroy your system stability
Debian installer rescue mode

Some notes on having access to Debian installer in rescue mode in physical servers of WMF infrastructure.
Accessing the rescue mode can help in restoring a bad grub configuration, getting into a given offile file-system, running fsck or other disk maintenance tasks.
== Option 1: interrupting autoinstall ==
This option consist on interrupting the standard debian installer autoinstall mechanism (with preseed) and then escaping the install process by means of the debian installer menu, and then operating the underlying operating system.
== Option 2: pxe-bootable rescue image ==
This option is under development. See phab:T78135.
After https://gerrit.wikimedia.org/r/c/operations/puppet/+/654192 it is now possible to use the rescue image during PXE boot. A typical workflow could be:
# Powercycle the host.
# Force PXE (via F12 for example on Dells).
# When the "boot: " prompt pops up, type "rescue" and hit enter.
Debianize python package

----
You really want to look at Git-buildpackage which has a section to build a debian package
----
Create an instance in labs.
 sudo -s
 gem install fpm
 mkdir /mnt/workdir
 cd /mnt/workdir
== Official way ==
http://wiki.debian.org/Python/Packaging
== Using fpm ==
Pass to fpm the source type (-s python) and the target package system (-t deb) then the package name.
Quick example:
 # fpm -s python -t deb voluptuous
 Created deb package {:path=>"/mnt/workdir/take2/python-voluptuous_0.6_all.deb"}
 # ls *.deb
 python-voluptuous_0.6_all.deb
Long example:
 fpm -s python -t deb \
 --maintainer "First Last <user@host.tld>" \
 --iteration wmf1 \
 --license BSD \
 --description 'Voluptuous, despite the name, is a Python data validation library. It is primarily intended for validating data coming into Python as JSON, YAML, etc.' voluptuous
You have now a very basic package.
== Using debianize.sh ==
debianiaze.sh is a wrapper around fpm that will take care of fetching any dependencies for you and build them as well.
 git://github.com/specialunderwear/debianize.sh.git
Lookup your package at http://pypi.python.org/ . We will use voluptuous as an example.
 # cd /mnt/workdir
 // download:
 # wget http://pypi.python.org/packages/source/v/voluptuous/voluptuous-0.6.tar.gz
 // uncompress
 # tar -xvzf voluptuous-0.6.tar.gz
 # cd /mnt/workdir/voluptuous-0.6
 // build!
 # ../debianize.sh/debianize.sh -m "Antoine Musso <hashar@free.fr>"
If all goes well, you will end up with some deb packages:
 $ ls -1 *.deb
 python-distribute_0.6.34_all.deb
 python-voluptuous_0.6_all.deb
 $
Category:Package management
Deployment-prep/deployment-pdf

There are two varieties of PDF server, only one of which is in labs.
* mwlib -- The pediapress 'legacy' configuration. This is in labs, but cannot be puppetized
* ocg -- The new configuration. This can be puppetized (see the ocg role), but needs ubuntu 14.04 to run properly
== mwlib configuration ==
Since mwlib boxes have to be set up by hand; these are the steps to recreate one (on ubuntu 12.04):
mkdir /srv/mwlib
chmod 775 /srv/mwlib
sudo apt-get install build-essential python python-dev python-virtualenv \
 libjpeg-dev libz-dev libfreetype6-dev liblcms-dev libxml2-dev libxslt-dev \
 ocaml-nox git-core python-imaging python-lxml texlive-full ploticus dvipng \
 imagemagick pdftk
virtualenv --distribute --no-site-packages /srv/mwlib
sudo ugo+x /srv/mwlib/bin/activate*
export PATH=/srv/mwlib:$PATH
hash -r
export PIP_INDEX_URL=http://pypi.pediapress.com/simple/
pip install pyfribidi mwlib mwlib.rl mwlib.zim
git clone https://github.com/pediapress/texvc
cd texvc; make
sudo make install
mkdir /srv/mwlib-tmp
chown www-data:www-data /srv/mwlib-tmp
=== Upstart configurations ===
You will need an upstart configuration for nserve, nslave, mw-qserve, and postman. nslave and postman need a --cachedir=/srv/mwlib-tmp argument
start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [!2345]
setuid www-data
setgid www-data
script
 /srv/mwlib/bin/activate
 nserve
end script
respawn
Deployment pipeline

The Deployment Pipeline is modernizing Wikimedia's deployment processes.
The Deployment Pipeline is moving all Wikimedia-infrastructure-deployed software to a continuous delivery model with the goal of reducing cycle time — the time from idea to deployment in production.
The pipeline makes use of PipelineLib to quickly build images with Blubber, integrate those images with Helm, and deploy to Kubernetes with Helmfile.
For a deeper-dive into how exactly the pipeline works, check out the Wikimedia Tech Talk about it on YouTube.
Documentation
For more information on the background and design of the Deployment Pipeline, checkout the Streamlined Service Delivery Design document.
Category:Kubernetes
Category:Deployment
Deployment pipeline/Components

== Blubber ==
Blubber creates Dockerfiles from a higher level description expressed as YAML. The Deployment pipeline relies on an application's Blubberfile(s) to build images for testing and production.
== PipelineLib ==
PipelineLib is a library of code that consumes an application's .pipeline/config.yaml. This configuration directs the pipeline through integration telling it:
* What blubber.yaml
* What unit tests to run
* What integration tests to run
* Which tests can be run in parallel
* What container image is to be used in production
== Helm ==
Helm uses Helm charts to define the resources needed to deploy an application to production. Helm manages things like:
* Config files
* Resource limits
* What ports your application uses
* Smoke tests for your application
In short, Helm allows the easy management of mapping deployment resources to an environment. If you need a different configuration for local development and production, chances are it'll be easy to configure with Helm.
== Helmfile ==
Helmfile is what you will use to deploy your application in production. Helmfile uses the environments and values in the deployment charts repository to define a deployment to a specific environment. A deployment of an existing production service looks like:
$ cd /srv/deployment-charts/helmfile.d/services/blubberoid
$ helmfile -e staging -i apply
== Kubernetes ==
Kubernetes is the execution framework that the pipeline works on. See Kubernetes
== Application Metrics monitoring ==
Monitoring in the pipeline happens via prometheus and exposing via Grafana.
==== Prometheus ====
Applications need to either provide Prometheus native endpoints (recommended), or emit Statsd (deprecated) compatible metrics to a local prometheus-statsd-exporter instance. In the latter case, SRE provides a ready to use component as part of any Helm chart in the deployment-charts repo.
In all cases, applications are discovered automatically and scraped by Prometheus. 
==== Grafana ====
Deployment pipeline applications are currently exposed under the Service folder in grafana.wikimedia.org. See Grafana.wikimedia.org#Pipeline for more