nyuuzyou/wikis huggingface dataset:
Confd

Confd is a configuration management system that can actively watch a consistent kv store like etcd or zookeeper and change config files based on templates whenever those key change; it can also tell the service depending on said configs to reload its own configuration. It is ideal to coordinate changes across a cluster almost atomically whenever your application doesn't support the kv store as a config source. It will probably also speed up our integration of more and more services easier.
== How to add a configuration/service to confd ==
== How do you write a template? ==
Templating in confd is based on the (quite awkward) go text/template system, to which confd adds a few of specific functions that you can use in pipelines. They are all listed in the confd docs
== What to expect from confd ==
Confd will watch the keys you tell it to watch in the service file, and re-apply your template whenever they change. If the resulting file has changed compared to what you precedingly obtained, confd will change the file on disk, and - if you have given it such information - will first validate the new config file and in case it's valid, it will issue the reload command you indicated. Please note that confd will not watch its own templates directory for changes, so if you change a template, you need to restart confd. No interruption of service will come from confd not running for a short while, and any changes it didn't get earlier will be accounted for. If you are using our own confd puppet module, it will take care of it
== What should I configure via confd, and what should I use puppet for instead? ==
Roughly speaking, puppet holds configuration, and etcd holds the state of resources. So, using f.e. varnish, the whole VCL logic should be configured via puppet, but the backends list should probably be generated via confd. Which brings us to the next topic
=== Making puppet and confd work together ===
The main strategy, whenever the same config file holds configuration and state, is to make puppet generate just the confd template files, and let confd create the actual config files and take care of the service.
== Monitoring ==
=== Compilation is broken ===
Investigate /var/log/confd.log (syslog) on the alerting host for errors.
Possible causes:
* A service with no backends weighted/pooled
=== Stale template error files present ===
This means compilation is not currently broken, but was broken at some point in the past. This commonly occurs when adding a new LVS service, for instance.
If you are aware of the condition, then you can simply remove the error files. For instance, if you get this alert for the cloudceph service on puppetmaster2001's expansion of pybal state, you can:
 ssh puppetmaster2001.codfw.wmnet sudo rm /var/run/confd-template/.cloudceph*.err
Config-master.wikimedia.org

config-master.wikimedia.org is a public web service for browsing configuration data from etcd.
== Service ==
The service is hosted on two Ganeti VMs in eqiad/codfw and is powered by Apache. The VMs are active/active via discovery records.
To reboot one of the nodes or enforce a service restart, the sre.misc-clusters.restart-reboot-config-master cookbook can be used.
== See also ==
* Conftool
* etcd
== External link ==
* https://config-master.wikimedia.org 
Category:Services
Conftool2git

Conftool2git is a service that listens for HTTP POST requests of conftool audit log events, and combines them with querying the etcd backend to create changes to a git repository that is, to all effects, recording all history of what was changed by whom in conftool. Conftool will only send such audit log events if the conftool2git_address configuration key is set. The service runs on one of the puppetservers, indicated by profile::conftool2git::active_host, and saves the data it collects to /srv/git/conftool/auditlog, and that repository is then replicated to all puppetservers.
==== Some details ====
The service is written in async python, and spawns an aiohttp web server to receive POST requests at the root url. 
It listens on the address indicated by profile::conftool2git::address, and once it receives a message it json-decodes it, and sends it to a queue. Another coroutine will consume from this queue, fetch the relevant data from etcd, dump it to the appropriate file on disk, then commit the changes. Once the commit is done, the post-commit hooks are run, which are installed by the git::replicated_local_repo puppet class, pushing the new commit to all puppetservers.
On startup, the service will try to perform a full sync between what is on the backend and what's currently on disk, so that any lost messages or other issues (like a server/service downtime) are amended.
===== What to do if it breaks =====
Just restart the service, that should solve most issues that are inherent to conftool2git. If the problem comes from the git commit process, that's another kind of issue and you should go look at the postcommit hook in the repository and try to debug that.
If any attempt at debugging the problem fails, you should just force push from the active host to the others.
Container

In the context of Wikitech container usually means a Linux Container as in LXC.
== Glossary ==
=== LXC ===
Linux Containers
* en-wp:LXC (Linux Containers)
* https://linuxcontainers.org/
* https://linuxcontainers.org/lxc/introduction/
=== LXCFS ===
"Userspace (FUSE) filesystem offering two main things:
*Overlay files for cpuinfo, meminfo, stat and uptime.
*A cgroupfs compatible tree allowing unprivileged writes.
It's designed to workaround the shortcomings of procfs, sysfs and cgroupfs by exporting files which match what a system container user would expect."
* https://linuxcontainers.org/lxcfs/introduction/
=== LXD ===
LXD is a container manager.
"LXD is the new LXC experience. It offers a completely fresh and intuitive user experience with a single command line tool to manage your containers. Containers can be managed over the network in a transparent way through a REST API. It also works with large scale deployments by integrating with cloud platforms like OpenNebula and OpenStack."
* https://linuxcontainers.org/lxd/introduction/
== See also ==
Containers
Category:Containers
Contint1001

contint1001 runs the continuous integration tools such as Jenkins and Zuul.
== Alias ==
The contint.wikimedia.org public hostname is an alias to the currently active CI host.
As of 2016, this is usually contint1001 in Eqiad. As of Dec 2021, contint2001.codfw is primary.
== Services ==
* Jenkins
* Zuul
* integration.wikimedia.org
== Misc ==
* Gallium/Upgrade to Precise
Contint1001/Upgrade to Precise

This page is about upgrading the gallium host from Ubuntu Lucid to Ubuntu Precise.
== Process ==
* send an email to wikitech-l about host being upgraded and service under maintenance
* Stop Apache
* Stop Jenkins
Backup!!!
* Backup the following directories (and maybe do a full backup of the 46GB as well)
** /home 
** /var/lib/jenkins (roughly 40GB)
** /srv 
* Need to make sure dotfiles are included ;-)
* Upgrade
* Get puppet to run. Will reinistall a half working copy of jenkins.
* Stop Jenkins
* Drop /var/lib/jenkins and replaces it with the /var/lib/jenkins backup
* Start Jenkins
Jenkins should be available at https://integration.mediawiki.org/ci/
== Stuff that need packaging ==
=== Android SDK ===
Android SDK needs to be packaged. It is, as of Oct 2012, in /home/hashar which is not really ideal :-]
Should probably install android on a local machine then build a dumb package that will simply install all files under /opt/androidsdk
=== PHPUnit upgrade ===
PHPUnit is installed from PEAR repository. The Ubuntu provided packages are too old. There is a watch file in the Debian package, so we could probably easily create a PHPUnit 3.7.8 packages using uscan and uupdate.
=== PHP 5.3.2 ===
optional would be nice to get PHP 5.3.2 instead of PHP 5.3.10. 5.3.2 is the minimal version supported by MediaWiki.
Contint2001

contint2001 runs the continuous integration tools such as Jenkins and Zuul.
== Alias ==
The contint.wikimedia.org public hostname is an alias to the currently active CI host.
As of Dec 2021, contint2001.codfw is primary.
== Services ==
* Jenkins
* Zuul
* integration.wikimedia.org
== Misc ==
* Gallium/Upgrade to Precise
* contint1001
* contint2002
Contint2002

contint2002 will run the continuous integration tools such as Jenkins and Zuul and replace contint2001 hardware.
== Alias ==
The contint.wikimedia.org public hostname is an alias to the currently active CI host.
As of Dec 2021, contint2001.codfw is primary. Soon it will be replaced by contint2002.
== Services ==
* Jenkins
* Zuul
* integration.wikimedia.org
== Misc ==
* Gallium/Upgrade to Precise
* contint1001
* contint2001
Cookies

This page contains hard-won advice on the use of cookies in MediaWiki code deployed to Wikimedia wikis.
First of all: if you can, don't use cookies. They bloat request sizes, can lead to concurrency issues when the user has multiple tabs open, and increase our privacy footprint.
Alternatives include:
* If you only care about logged-in users and need cross-device consistency, use MediaWiki user preferences (not though that they are included in every response so, like cookies, they bloat the HTTP payload size even when your feature is not being used).
* If you only need the data on the client side, use Local Storage with expiry set by using the mw.storage API.
** You can also use Session Storage via mw.storage.session, but beware that web browsers scope Session Storage to the life of a specific browser tab. This means when a tab is closed the data is effectively lost. This is usually not what you want, but sometimes useful. On the other hand, it also means each tab can have its own unique, and each one does survive restoration from back-forward history cache, "Re-open closed tab", and browser/device restarts if open tabs are restored.
== But if you must… ==
Only use cookies if you need to make the stored data available to PHP code too. Avoid cookies which have session or token in their name (case-insensitively) unless you understand how those interact with custom logic in the caching infrastructure.
Before deploying code that adds a new cookie, email the Privacy team at privacy@ so that they can verify that the new cookie complies with the cookie statement. Include the name of the cookie, its purpose, and its expiry time.
Use WebResponse::setCookie() / WebRequest::getCookie() in PHP, mw.cookie.set() / .get() in JS.
When setting a cookie on the server side, consider whether the response can get cached (together with the Set-Cookie header). MediaWiki tries to automatically make responses with Set-Cookie uncacheable, but the process is not always reliable and you should make sure to also do that in the code which sets the cookie (usually via OutputPage::disableClientCache()).
If the output for some page is cacheable and depends on the cookie value or presence, you need to add a Vary: Cookie header so caches are split (on Wikimedia wikis the header will probably be present anyway because authentication-related code almost always adds it, but in general you can't rely on that!). If you set the Vary header on an URL sometimes, you must set it on that URL always (i.e. not just when the cookie is present).
Be aware of the SameSite flag if you need to read cookies in cross-site requests.
== See also ==
* Incidents/20150527-Cookie
* Caching overview
* MediaWiki at WMF
* The cache-cookies log channel, which reports responses where someone has set a cookie but has not disabled caching.
Core

Core in Wikimedia technology may refer to:
== Networking ==
A core router is a type of Internet router,
* see "Core router" on Wikipedia, for information about the router in general. 
* see Network design, for how core routers are used at Wikimedia Foundation.
* see Common Datacenter Specifications, for how core routers are installed at Wikimedia Foundation.
* see Network monitoring, for an operational manual relating to core routers at WMF.
== Databases ==
Core databases refers to the MySQL clusters at Wikimedia Foundation that contain the databases of its MediaWiki websites (Wikipedia, Wikimedia Commons, Wikidata etc.)
* See MariaDB#Core MediaWiki databases. 
== Software ==
The MediaWiki software is often referred to as "core" or "MediaWiki core", to distinguish it from the various MediaWiki extensions that add to the "core" functionality.
== See also ==
* Core (disambiguation) on Meta-Wiki, for concepts, teams, or organisations in the Wikimedia movement
__NOEDITSECTION__ __NOTOC__
Category:Disambiguation pages
Create a Racktables server

Moving/copying Racktables from one server to another:
* Add the misc::racktables group to server
* Copy the directory /srv/org/wikimedia/racktables from the old server to the new server
* Reload apache
* Look at the inc/secret.php file - it contains the password and database backend information. Securely store this information. 
* Backup the current database, in case of mishap.
* If you are not naming the new host racktables.wikimedia.org , got to the apache sites-available directory and edit the racktables.wikimedia.org file to have the correct host name
* go to https://newhost/install.php - click through the steps and put the correct backend database information in.
If you want to upgrade
* If you want to upgrade to the (current as of 2011/12) .19 branch, you must first upgrade to .18.7 branch.
* Grab the .tar.gz from the racktables repository. Untar this. 
* Copy the files over to the current directory. By default this is /srv/org/wikimedia/racktables
* Go to https://newhost/upgrade.php - click through the steps again. 
Upgrading to the .19 branch
The .19 branch is special. When you unpack the tarball, you only need the wwwroot directory. 
* Copy the wwwroot directory to /srv/org/wikimedia/racktables
* Go to https://newhost/index.php 
* Click on the upgrade link and through all of the steps.
Category:How-To
Category:Operations
DNS/PowerDNS

PowerDNS is the DNS server we use for geographic DNS balancing, and possibly "normal" authoritative DNS in future as well. This page explains configuration. More can be found on DNS.
== Configuration ==
It's best to bind PowerDNS to a specific IP, instead of INADDR_ANY. To do that, put:
 local-address=IP
 query-local-address=IP
A backend like geobackend can't make use of caching, since returned results will be different depending on "who's asking". Therefore, disable caching:
 query-cache-ttl=0
 cache-ttl=0
 negquery-cache-ttl=300
If just geobackend is used, multithreading is unnecessary, and may even impact performance. Make pdns single-threaded with:
 distributor-threads=1
In geobackend itself, there are no wildcard records, so pdns shouldn't have to check for them:
 wildcards=no
Running privileged is unnecessary (of course make sure these user/group exist):
 setuid=pdns
 setgid=pdns
Launch the backend that will be used, in the specified order, seperated by commas:
 launch=geo
=== Geobackend configuration ===
Specify the zone that will contain the "georecords":
 geo-zone=wikimedia.org
Every zone needs a SOA record, and so does a geo-zone. Only the SOA name and hostmaster fields are important. Specify them comma separated:
 geo-soa-values=ns0.wikimedia.org,hostmaster@wikimedia.org
Every zone needs to have NS records as well, to specify the authoritative servers:
 geo-ns-records=ns0.wikimedia.org,ns1.wikimedia.org,ns2.wikimedia.org
Geobackend reads in an rbldnsd-style zonefile, to map specific IPs to countries. Specify the path to this file:
 geo-ip-map-zonefile=/etc/powerdns/zz.countries.nerd.dk.rbldnsd
Every record in the "geo zone" has its own "map file", mapping every country to some CNAME. Geobackend will read all files in the directory specified here, and use the filename as the recordname:
 geo-maps=/etc/powerdns/geomaps/
Every DNS record has a TTL, and geobackend currently only allows to specify these globally. Default is 1 hour for normal (CNAME) records, 1 day for NS records:
 geo-ttl=3600
 geo-ns-ttl=86400
== Provisioning ==
:This is not the way Wikimedia's setup works. For that, see DNS
To maintain an identical configuration of geomaps, the following rsync cronjob can be used:
 */15 * * * * pdns NR=$(rsync -rt --delete \
 rsync://rsync-path /etc/powerdns/directormaps | \
 awk '/Number of files transferred/ { print $5 }'); \
 [ $NR != "0" ] && /usr/local/bin/pdns_control rediscover > /dev/null
This rsyncs the geo map files from a central repository, and sends PowerDNS a signal to reread them if they've actually been changed.
The DNSBL should be regularly updated as well, for example daily:
 07 04 * * * pdns rsync -qt rsync://rsync.blitzed.org/countries/zz.countries.nerd.dk.rbldnsd \
 /usr/local/etc/powerdns/zz.countries.nerd.dk.rbldnsd && \
 /usr/local/bin/pdns_control rediscover > /dev/null
== External links ==
* PowerDNS documentation
* Geobackend README
Category:Network
Data Gateway

The Data Gateway Service is an abstraction that sits between consumers of published datasets, and the underlying database (currently Cassandra, though other databases are possible as well). It decouples consumers from the database(s), providing them a contract consisting of an HTTP interface and JSON-encoded results, (as opposed to an implementation-specific driver and/or idiomatic client library).
== Datasets ==
* Image suggestions
* Commons Impact Metrics
== Service endpoints ==
https://data-gateway.discovery.wmnet:30443 (production)
https://data-gateway.k8s-staging.discovery.wmnet:30443 (staging)
== Deployment ==
See: Kubernetes/Deployments
== Monitoring & debugging ==
=== /healthz ===
The /healthz endpoint is used for k8s readiness testing, but returns a JSON-encoded object with useful meta-data.
deploy2002:~$ curl https://data-gateway.discovery.wmnet:30443/healthz; echo
{
 "version": "v1.0.10",
 "build_date": "2024-07-23T00:13:01:UTC",
 "build_host": "buildkitsandbox",
 "go_version": "go1.21.8"
}
deploy2002:~$
=== /metrics ===
The /metrics endpoint returns Prometheus metrics.
eevans@deploy2002:~$ curl https://data-gateway.discovery.wmnet:30443/metrics 2>/dev/null
[ ... ]
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 7350
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
eevans@deploy2002:~$
=== Logging ===
OpenSearch Dashboard
== See also ==
* Data Gateway
Category:Data Persistence
Data Platform

__NOTOC__
Wikimedia's Data Platform is a collection of systems and services that enable data producers and consumers to discover, use, and collect data to derive insights, conduct research, and build new data products. The Data Platform is primarily maintained by the Data Platform Engineering team. To contact the team, use the intake process. 
== Get started ==
The Data Platform provides access to private data and internal WMF resources, so you must have specialized data access to use it. For public, open access Wikimedia data and tools, see meta:Research:Data.
== Data platform infrastructure ==
Data platform systems and infrastructure include the data lake, ingestion and processing pipelines, and production search and query services.
=== Data pipelines ===
Information about data pipelines is currently at:
* Data Platform/Systems/Cluster
* Airflow
* Hadoop Event Ingestion
* :Category:Data_pipelines
=== Search data and services ===
* Using search for new features 
* Search Platform 
* Wikidata Query Service (WDQS) 
=== Overview of data platform systems ===
File:WMF_Data_Platform_Technical_Overview_2023_V1.jpg |Data Platform Technical Overview 2023
File:WMF_Analytics_Data_Platform_2021_v1.png |Analytics Data Platform 2021
The following list highlights some major Data Platform systems. For more details and a full list of Data Platform system documentation pages on this wiki, see Data_Platform/Systems.
System name and link 	Type 	Accessibility
Airflow 	Workflow Job Scheduler 	Private
Archiva 	Repository for Java archives 	Private
AQS - Analytics Query Service 	REST API for analytics data 	Public
Ceph 	Software defined storage, serving block and object storage 	Private
Clients (stat100X) 	Analytics client nodes to access Hadoop and various services 	Private
Cluster (Hadoop, Gobblin, Hive, Spark...)	Hadoop 	Private
Datahub 	Data Catalog 	Private
Dashiki 	Framework for building dashboards 	Public
Druid 	Data storage engine optimized for exploratory analytics 	Private
EventLogging 	Ad-hoc streaming pipeline 	Private
EventStreams 	Mediawiki events streams 	Public
Kafka 	Data transport and streaming system 	Private
MariaDB 	Data storage for MediaWiki replicas and EventLogging 	Private
Matomo (formerly known as Piwik) 	Small-scale web analytics platform 	Private
Presto 	Big data high performance sql query engine 	Private
ReportUpdater 	Job Scheduler 	Private
Superset 	Web interface for data visualization and exploration 	Private
Jupyter 	Hosted notebooks for data analysis 	Private
Turnilo 	Web interface for exploring data stored in Druid 	Private
Wikistats (1 and 2) 	Community Dashboard with high-level metrics 	Public
Wmfdata-Python 	Python package for streamlined data access on the analytics clients	Private
 
== Data platform operations ==
Find ops week and other process documentation at Data Platform Engineering on Wikitech and the project pages on MediaWiki.org.
The list of scheduled manual maintenance tasks are documented at Data Platform/Systems/Manual maintenance
Data Platform
Category:Documentation
Category:Data platform
Data Platform/AQS

Analytics Query Service (AQS) is a set of services that provide public-facing APIs that serve analytics data about Wikimedia projects, such as page views, unique devices, and editor counts. AQS APIs provide the data behind Wikistats (stats.wikimedia.org).
== User documentation ==
To use the AQS APIs, see the Wikimedia Analytics API documentation.
== Maintainer documentation ==
For information about the AQS services, see Data Platform/Systems/AQS.
=== Studies and data analyses ===
* Commons Impact Metrics
* Media metrics
* Media requests limitations
* Vetting media request metrics
* Pageviews per project
* Vetting data per project
* Vetting data per project family
Category:Analytics Query Service
Category:Landing page
Category:Data platform