nyuuzyou/wikis huggingface dataset:
Check bacula.py

Check bacula is a wrapper for bconsole that is able to produce output to be used by icinga and prometheus, as well as used interactively to debug backup job errors/misses, useful for monitoring of backups generated by Bacula in the WMF production.
== Usage ==
 check_bacula.py --help
 usage: check_bacula.py [-h] [--list-jobs] [--prometheus] [--port PORT]
 [--icinga] [--backup_config_path BACKUP_CONFIG_PATH]
 [--bconsole_path BCONSOLE_PATH]
 [job]
 
 Checks bacula backup freshness status and prints it on standard output (or
 starts the bacula prometheus exporter).
 
 positional arguments:
 job If set, check only the status of this job. Otherwise,
 check all jobs.
 
 optional arguments:
 -h, --help show this help message and exit
 --list-jobs When used, it just prints the list of configured
 backup jobs and returns without any check.
 --prometheus If set, instead of outputing information to the
 command line, it waits in a loop listening for an HTTP
 request and returns metrics in the typical prometheus
 exporter format.
 --port PORT When using the prometheus mode, it binds and listens
 on this port. If no one is given, 9133 is used by
 default.
 --icinga If set, it prints just a summary for icinga.
 Otherwise, it prints a detailed output to the command
 line.
 --backup_config_path BACKUP_CONFIG_PATH
 Path of the directory with the files where the bacula
 job configuration is, by default:
 "/etc/bacula/jobs.d".
 --bconsole_path BCONSOLE_PATH
 Full path of the "bconsole" executable, by default:
 "/usr/sbin/bconsole".
== Examples of interactive usage ==
* No parameters run will do the full list of configured backups, classified by freshness (See Bacula#Monitoring for the categories). Similar to the icinga output but without sumarizing
 check_bacula.py
* To list jobs, one per line, without checking or classifying:
 check_bacula.py --list-jobs
* To give the available list of past jobs for a particular job (including its id, timestamp, level, finishing status and total size), one can run:
 check_bacula.py <full-job-name>
== See also ==
* More info on how it works and why it was created on Bacula#Monitoring
* Source code: phab:/source/operations-puppet/browse/production/modules/bacula/files/check_bacula.py
Category: Backups
Check ganglia

check_ganglia is a nagios/icinga plugin script that can be used to generate alerts based on metric values in Ganglia. It query gmond instances or the main gmetad via its xml query interface.
check_ganglia has been imported into our gerrit and debianized: https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/check_ganglia. It was originally imported from https://github.com/larsks/check_ganglia.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6602
== Puppet Usage ==
To install a new icinga alert based on a Ganglia metric, use the monitor_ganglia define.
 # Set up icinga monitoring of Kafka broker per second.
 # If this drops too low, trigger an alert.
 # These thresholds have to be manually set.
 # adjust them if you add or remove data from Kafka topics.
 monitor_ganglia { 'kafka-broker-MessagesIn':
 description => 'Kafka Broker Messages In',
 metric => 'kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate',
 warning => ':1500.0',
 critical => ':1000.0',
 require => Class['::kafka::server::jmxtrans'],
 }
This is a just wrapper for our monitor_service define. See the monitor_ganglia documentation for up to date usage.
== CLI usage ==
In addition to nagios/icinga alerts, check_ganglia is also a useful CLI interface for querying Ganglia directly. It can query either gmond aggregators or gmetad itself.
=== Querying gmond ===
Querying gmond aggregator instances is slightly faster than querying gmetad, but requires knowledge of which aggregator instance has data for which nodes.
# analytics1010.eqiad.wmnet is a gmond aggregator.
check_ganglia -g analytics1010.eqiad.wmnet -H analytics1010.eqiad.wmnet -m Hadoop.NameNode.FSNamesystem.CapacityRemainingGB
# or you can even list all ganglia metrics for a given node
check_ganglia -g analytics1010.eqiad.wmnet -H analytics1010.eqiad.wmnet --list
=== Query gmetad ===
Querying gmetad is slightly easier, because all of the metrics for all nodes should be available here.
# -q flag tells ganglia to use gmetad (which is on port 8654 for us).
check_ganglia -q -g nickel.wikimedia.org -p 8654 -H analytics1010.eqiad.wmnet --list
Check graphite

check_graphite is a nagios/icinga plugin script that can be used to generate alerts based on metric values in Graphite. It simply queries graphite to fetch data in JSON format through the /render endpoint of the graphite server. Our code is an (almost complete) rewrite of the check_graphite plugin from disquis. 
== Puppet Usage ==
We have two types of checks that can be performed on graphite-collected metrics:
* check_graphite_threshold for checking thresholds
* check_graphite_anomaly which performs some form of anomaly detection on the metric.
Both are a just wrapper for our monitor_service define. See their respective documentation for up to date usage docs.
=== Define monitor_graphite_threshold ===
A simple threshold checking is supported -this simply checks if a
given percentage (by default, 1%) of the data points in the interested interval
exceeds a threshold.
So, for instance, if you want to ensure that less than 5% of the checks in the last hour for the number of 5xx responses is above 500, you can do as follows:
 # Alert if the same metric exceeds an absolute threshold 5% of
 # times.
 monitor_graphite_threshold { 'reqstats-5xx':
 description => 'Number of 5xx responses',
 metric => 'reqstats.5xx',
 warning => 250,
 critical => 500,
 from => '1hours',
 percentage => 5,
 }
=== Define monitor_graphite_anomaly ===
A very simple predictive checking is also
supported - it will check if more than N points in a given
range of datapoints are outside of the Holt-Winters confidence
bands, as calculated by graphite (see
http://bit.ly/graphiteHoltWinters), at 3 delta confidence level (99.7%) - which should be good in most cases). The obvious advantage of this method is that we don't need to pre-define thresholds at all.
This kind of monitoring always requires at least a week of data to graphite, which is needed to have decent predictions, so it's pretty computationally-expensive. you can define the interval of datapoints on which you wish to check for anomalies via the check_window parameter.
Let's see how we could try to detect an anomaly in the same metric as before: we will raise an alarm if 5 (or 10) measured datapoints out of the last 200 have an anomaly.
 # Alert if an anomaly is found in the number of 5xx responses
 monitor_graphite_anomaly { 'reqstats-5xx-anomaly':
 description => 'Anomaly in number of 5xx responses',
 metric => 'reqstats.5xx',
 warning => 5,
 critical => 10,
 check_window => 200,
 }
A few more checks are in the work, this page will be updated then.
Checking MySQL replication

==Checking replication delay==
Look for the two system user accounts in the processlist. One is the control process, the other is the SQL worker process. The Time: field of the SQL worker process is the difference between the local system clock and the timestamp of the replication log entry that it is processing. If the clocks are synchronized, this is the replication delay in seconds.
 mysql> show processlist \G
 *************************** 2. row ***************************
 Id: 18736
 User: system user
 Host:
 db: NULL
 Command: Connect
 Time: 5
 State: Waiting for master to send event
 Info: NULL
 *************************** 3. row ***************************
 Id: 18737
 User: system user
 Host:
 db: jawiki
 Command: Connect
 Time: 29
 State: Updating
 Info: UPDATE cur SET cur_counter = cur_counter + 1 WHERE cur_id = 97735
Category:How-To
Category:MySQL
Cinder

Cinder is a Block Storage service for OpenStack. It virtualizes the management of block storage devices and provides end users with a self service API to request and consume those resources without requiring any knowledge of where their storage is actually deployed or on what type of device.
The Cloud VPS OpenStack cluster has Cinder installed with Ceph as the underlying storage system. 
== See also ==
* https://wiki.openstack.org/wiki/Cinder
* https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/
* [Cloud-announce] new Cloud VPS feature: attachable block storage
* Help:Adding Disk Space to Cloud VPS instances#Cinder
* Portal:Cloud VPS/Admin/Ceph
Cindy The Browser Test Bot

Cindy is the browser test bot which runs browser tests for CirrusSearch extension. It is run on any patch against mediawiki/extensions/CirrusSearch repository and V+1 or V-1 the patch in Gerrit depending on the test results. It is based on Barry The Browser Test Bot.
== The tests ==
The tests are located in tests/integration subdirectory in the CirrusSearch source. They can also be run manually by cloning the integration test runner repository and running:
./create-env.sh && ./run-integration.sh
== The runner ==
The runner is found in search-platform/cirrus-integration-test-runner. This polls gerrit for patches to review, sets up a multi-wiki environment for each, and runs the integration test.
== The bot ==
The bot is running on cirrus-integ03.search.eqiad1.wikimedia.cloud. In order to control it, after logging in, run:
 sudo -u integration-bot tmux attach
Then you can run the bot with:
 ~/cirrus-integration-test-runner/run-cindy.sh
The bot should always be run from within the tmux session, this simplifies administration and ensures it keeps running after disconnect.
TODO: Could this be more reliable as a systemd unit?
== Logs ==
Most logs are destroyed when the next test run starts. A record of the console output of the last integration test run for each tested patch is found in </code>/tmp/cindy_run.<gerrit id>.log</code>
== Web Access ==
Access to the http site under test is done through SSH port forwarding. You may need to stop any local service using port 8080.
Connect like so:
ssh -L 8080:localhost:8080 cirrus-integ03.search.eqiad1.wikimedia.cloud
And then visit cirrustestwiki.mediawiki.mwdd.localhost:8080.
Category:Discovery
Citoid

Citoid is a stateless Node.js service running in Kubernetes. It processes HTTP requests containing a URL, ISBN number or other identifiers/search terms by retrieving citation information from external sources with help from a local nodejs Zotero server. It responds with a JSON structure containing the citation information.
==Updating==
Citoid runs behind service-runner, and should be regularly updated in line with service-template-node
See: mw:Service-template-node/Updating
== Deployment ==
== Monitoring ==
* Icinga: Search for 'citoid'
* Grafana shows probe history for all swagger probes, including for citoid: https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?orgId=1. Logs for swagger endpoint alerts: https://logstash.wikimedia.org/app/dashboards#/view/8b1907c0-2062-11ec-85b7-9d1831ce7631?_g=h@faedd66&_a=h@c160d35
== Logs ==
Kibana dashboard: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631
Use kibana to look at logs. However, if necessary, you can look at the raw logs from the cluster directory, 
 kube_env citoid eqiad # or codfw 
 kubectl logs -l app=citoid -c citoid-production
== Stats ==
Stats are done currently with both Prometheus and statsd- in progress of converting statsd. Citoid/Prometheus for more. 
== See also ==
* Developer documentation on mediawiki.org
* Puppet module
* Source code
Category:Services
Citoid/Prometheus

You can test citoid locally with Prometheus. 
== Download Prometheus ==
https://prometheus.io/docs/prometheus/latest/getting_started/
== Local config ==
=== For citoid ===
In your config.dev.yaml 
# Metrics reporters
metrics:
 - type: prometheus
 port: 9100
=== For Prometheus ===
Your prometheus.yml should look like
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's citoid.
scrape_configs:
 # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
 - job_name: 'prometheus'
 static_configs:
 - targets: ['localhost:9100']
 - job_name: 'citoid'
 static_configs:
 - targets: ['localhost:9102']
Then run both citoid and prometheus./prometheus --config.file=prometheus.ymlAnd from your citoid directorynpm start
== Testing ==
Make a few requests to citoid.
Navigate to http://localhost:9090/ to look for data collected from those queries, i.e. search citoid_input_type_total in the search bar.
Civicrm-dev.wikimedia.org

CRM and donations records.
== Hardware ==
* srv9 - web frontend (SSL)
* db1008 - (master of fundraising db cluster) backend
== Software ==
* Apache+PHP
* Drupal: Version 5.7
** CiviCRM: Version 2.2.8
** Customization of CiviCRM by David Strauss, as listed below:
== Usage ==
* This version went live August 2008 as the development server
== Installation ==
dev.civicrm is a clone of civicrm circa August 2008 and retains the same confiugration. All application files are stored under 
/srv/org.wikimedia.civicrm.dev 
The only changes that exist between the two installations are present in sites/default/civicrm.settings.php
They includle
 $civicrm_root
 CIVICRM_TEMPLATE_COMPILEDIR
 CIVICRM_UF_BASEURL
 CIVICRM_UF_DSN
and within sites/default/settings.php
for
 $db_url 
 
New Database names are drupal5_dev & civicrm_dev
== Databases ==
To access the databases, ssh to the dev server:
 ssh -A civicrm-dev.wikimedia.org
The dev databases are hosted on db1008.eqiad.wmnet. The databases are dev_drupal (for the drupal installation) and dev_civicrm (for the CiviCRM installation).
 mysql -h db1008.eqiad.wmnet -p
 show databases;
 use dev_civicrm;
 show tables;
To dump the current database:
 mysqldump -h db1008.eqiad.wmnet -u <username> -p --single-transaction civicrm > civicrmdump.sql
=== Updating development copy paths following a database clone ===
After syncing the database with the live data, follow the following instructions:
* Visit: https://dev.civicrm.wikimedia.org/civicrm/admin/setting/updateConfigBackend?reset=1
* Update the file path to append a ".dev"
* Update the URL path to prepend a "dev."
* Save
* Rebuild menus: https://dev.civicrm.wikimedia.org/civicrm/menu/rebuild?reset=1
* Update the URLs to point to the development copy: https://dev.civicrm.wikimedia.org/civicrm/admin/setting/url?reset=1
The non-appearance of the search box on the development site is a result of a Drupal bug (maybe?) in block handling that doesn't like block types to change without a module re-install. This happens because we clone over the CiviCRM database but not the Drupal one.
== CiviCRM code ==
To edit or update the CiviCRM code used by the dev server:
 ssh -A civicrm.wikimedia.org
 cd /srv/org.wikimedia.civicrm.dev
Most of the modules can be found at:
 cd /srv/org.wikimedia.civicrm.dev/sites/all/modules
Category:Services
Category:Fundraising
Category:Fundraising - Needs Updated
Civicrm.wikimedia.org

CRM and donations records.
= Mail =
There is a lightweight Exim4 set up running on the server (see Grosley#Mail_setup for more) which handles sending mail for CiviCRM (via SMTP).
When mail is generated by CiviCRM, Civi stores the mail in some internal queue and does not send the mail until https://civicrm.wikimedia.org/sites/all/modules/civicrm/bin/civimail.cronjob.php is hit. We have scripts living in /home/civicmail/ that handle hitting the appropriate URLs to send and receive mail:
* civimail_send.sh - sends mail for the live site
* dev_civimail_send.sh - send mail for the dev site
* civimail_receive.sh - receives mail for the live site
* dev_civimail_receive.sh - receives mail for the dev site.
Currently, there are two separate Hudson jobs configured to send and receive mail for CiviCRM every 10 minutes.
= Periodic Automated Tasks =
Hudson handles the periodic automated tasks that need to happen for our CiviCRM set up. This includes:
* Drupal cron.php runs
* Verifying pending PayflowPro transactions
* Consuming donations in the 'donations' queue into CiviCRM (this is handled by drush - note that this particular component needs to be run via drush as an admin user [eg drush --user=1 qc])
* Sending mail for CiviMail
* Receiving mail for CiviMail
= Upgrades =
See the internal CiviCRM upgrade docs for the latest upgrade information
= Public Data Dumps =
* Public data dumps are run from srv9 with cron under the reporter user. public_reporting_dump.sh runs fundraising.pl which pulls civicrm data for a given range. These dumps are then sent out to download.wikimedia.org and publicly consumed.
Relevant files are under the private svn depot under
 svn.wikimedia.org/svn-private/wmf/donation/fundraising.pl
 svn.wikimedia.org/svn-private/wmf/donation/crons/public_reporting_dump.sh
=Issues=
* Maintained on the CiviCRM wiki
Category: Fundraising
Category:Fundraising - Needs Updated
Close a wiki

In order to lock a wiki, do the following.
== Wikimedia configuration ==
* Read How to do a configuration change
* Add the wiki to closed.dblist
* Remove the wiki from groupOverrides and groupOverrides2 in wmf-config/core-Permissions.php
* Search touch wmf-config/InitialiseSettings.php for the language code and the wiki db name, and consider removing it.
* If you didn't modify InitialiseSettings.php in the previous steps, run touch wmf-config/InitialiseSettings.php on the deployment server to trigger the update of the config cache
* Remove the wiki from the wikimedia/portal repository.
== Parsoid ==
Parsoid has its own copy of SiteMatrix, which needs updating using tools/fetch-sitematrix.js in mediawiki/services/parsoid.git.
Then, once merged, it must be deployed; if not, switch to visual editor will trigger Failed to load resource: the server responded with a status of 404 () errors.
== Cxserver ==
* Remove wiki from cxserver/config/languages.yaml
== CentralAuth ==
* Remove wiki from all wiki sets via meta:Special:WikiSets (or ask on meta:SRM).
== See also ==
* Add a wiki
* Delete a wiki
Category:How-To
Cloud VPS Server Admin Log

The Server Admin Log (known as SAL) is a public collection of short messages, published from server administrators (in Toolforge, Cloud VPS projects, etc.) in order to track and share server activities (related to maintenance, upgrades, fixes, etc.) and better collaborate.
== Add from IRC ==
To add a new message you can join this IRC channel: . Then send a message using this format:
 !log <project> <message>
Example:
 !log write-here-the-project-code My cat destroyed the database. Sorry. Now fixed.
Note that the project code of a Toolforge tool is "tools.<toolname>".
A bot will read your message and automatically add it to a Wikitech page (Nova Resource:<project>/SAL).
== Add from Toolforge ==
If you are in a Toolforge bastion shell you can also send a message using this command (the project name is automatically included):
 dologmsg <message>
== Exploring ==
The SAL tool is useful to explore log entries from projects:
* toolforge:sal/projects
== See also ==
* :Category:SAL
* Project list on the SAL tool
* [ recent changes in Nova Resource namespace] ([ atom])
Cloudweb1003

cloudweb1003 and cloudweb1004 host Horizon and toolsadmin (Striker).
== Connecting ==
Note that the domain name is .wikimedia.org, not .eqiad.wmnet.
== Maintenance ==
cloudweb1003 and cloudweb1004 are load-balanced and redundant, using LVS. To take one of them out of service for maintenance, log in and run
cloudweb1003:~$ sudo depool
After maintenance, put the host back into service:
cloudweb1003:~$ sudo pool
Codfw data center

codfw (pronounced code-ef-double-you, or the Dallas DC) is located at CyrusOne's DFW1 data center in Carrollton, Texas, United States.
== History ==
The name Codfw is based on CyrusOne (the vendor), and DFW being the large nearby airport. Technically, Love Field airport is closer but less well-known.
== Cabinet name mapping ==
We label our cabinets in lettered rows, and numbered cabinets (example: A1-codfw). CyrusOne's reports have info for cabinets like CAR-DH7-WIKI-7010-66.
WMF Cabinet Name	CyrusOne Cabinet Name
a1 	EZ-DH7-7010-66
a4 	EZH-DH7-7010-69
a7 	EZ-DH7-7010-72
b1 	EZ-DH7-7013-66
b4 	EZ-DH7-7013-69
b7 	EZ-DH7-7013-72
c1 	EZ-DH7-7018-66
c4 	EZH-DH7-7018-69
c7 	EZ-DH7-7018-72
d1 	EZ-DH7-7021-66
d4 	EZ-DH7-7021-69
d7 	EZ-DH7-7021-72
== Media gallery ==
File:Wikimedia Foundation Servers 2015-90.jpg
File:Wikimedia Foundation Servers 2015-92.jpg
File:Wikimedia_Foundation_servers_-_CyrusOne_(Carrollton,_Texas)_(43).jpg
File:Wikimedia Foundation Servers 2015-63.jpg
File:Wikimedia Foundation Servers 2015-88.jpg
File:Wikimedia Foundation Servers 2015-54.jpg
==See also==
*Wikimedia Foundation selects CyrusOne in Dallas as new data center (May 2014), blog.wikimedia.org.
*Phabricator project to track on-site Codfw work
*Historical Phabricator project "Codfw rollout" from 2014
*Pages linking to ""
 
Category:Data centers
Collaborative tmux sessions

Sometimes you want another pair of eyes when troubleshooting or working in production machines. The tools screen and tmux are being used to create collaborative terminal sessions.
=== tldr; ===
Create tmux session:
tmux new -s session_name
Join tmux session:
tmux attach -t session_name
You can use the -r flag for read-only connections:
tmux attach -rt session_name
=== Create sessions with fixed size ===
It could happen that different terminal sizes are used (old hardware, mobile, big fonts for better readability). By default, tmux will fall back to the window size of the smallest connected client. The behavior can be changed by setting the the window-size option. Possible options are largest, smallest, manual, latest. When creating a new session, this option can be set to manual to prevent unwanted changes of the window size:
tmux new -s session_name \; setw window-size manual \; attach
Join tmux session is the same:
tmux attach -t session_name
=== Share session between users ===
It is possible to share tmux session between different users. 
==== Share session using sockets ====
By default tmux will create a socket in /tmp/tmux-<user_id>/default. It is also possible to explicitly use a different socket using the -S option:
tmux -S /path/to/socket new -s session_name
To connect to this session, grant the user which wants to connect read and write access to the socket and then use:
tmux -S /path/to/socket attach
This method is useful when you don't want to use root (see below). Use this method with caution if you are in a shared environment with different user privileges. Users may use this to escalate privileges.
==== "Share" session using root/sudo ====
It is also possible to create a tmux session as root and let others join by using sudo/root:
sudo -i tmux new -s session_name
Join the tmux session:
sudo -i tmux attach -t session_name
Keep in mind that you are using the root user here. So for read-only operations and debugging you may want to use your personal user and share the session using sockets or su back to a user with less privileges.
Commons Impact Metrics/Algorithm

This page describes the algorithm used by the Commons Impact Metrics data product to traverse the commons category graph. These slides are from a presentation given by Marcel Ruiz Forns at the 2024 Wikimedia Hackathon. 
=== Because of these computational challenges, we set a couple of boundaries to the calculation. ===
Instead of calculating metrics for all Commons categories, we do it only for a curated list of categories related to commons mass upload, usually GLAM. Also, we define a maximum “depth” that our algorithm will compute, to avoid going down an infinite rabbit hole of unrelated subcategories. And finally, we aggregate metrics at the coarsest granularity (monthly), to keep the size of the output data small enough for community members to handle without the need of a distributed computation cluster.