nyuuzyou/wikis huggingface dataset:
Monitoring/Netbox DNS uncommitted changes

== What it does ==
This Icinga check runs the Python script that generates all the DNS records from Netbox (the same script run by the sre.dns.netbox cookbook) and checks if there are any uncommitted changes in the Netbox#DNS exported repository. This ensures that changes in Netbox are reflected into the DNS repository and hence in production in a timely manner and that there are no leftover changes that will show up to the next person running the sre.dns.netbox cookbook.
== Statuses ==
* OK: there are no uncommitted changes, the autogenerated DNS repository is in sync with Netbox data.
* WARNING: there are uncommitted DNS changes but the last edit in Netbox was made few minutes ago (at most 30 minutes as of Sep. 2020) hence it's not yet considered critical because most likely someone is still editing Netbox and will propagate the changes soon.
* CRITICAL: there are uncommitted DNS changes and the last edit in Netbox was a while ago (at least 30 minutes as of Sep. 2020).
== What to do ==
To see what are the pending DNS changes that are uncommitted, run in DRY-RUN mode the sre.dns.netbox cookbook as follows (see Spicerack/Cookbooks#Run a single Cookbook for more context on how to run a cookbook):
sudo cookbook -d sre.dns.netbox "test"
It will show all the pending DNS changes not yet committed.
If you are missing context get in touch with the people/team most likely related to those changes asking in the various SRE IRC channels.
You can also look at the Netbox Change Log.
Only if you are sure that those changes can be committed, follow DNS/Netbox#Update generated records.
This will propagate DNS changes to production, hence care must be taken to make sure they are correct and would not cause an outage.
Category:SRE Infrastructure Foundations
Monitoring/atftpd

atftpd is a TFTP server. It is running on install servers.
The Icinga check that links here is checking whether the tftp process is running.
The command line it uses is:
/usr/lib/nagios/plugins/check_procs -c 1:1 -u nobody --ereg-argument-array=\'.*/usr/sbin/atftpd .*\'
It checks if exactly one process (no more and no less) are matching the regular expression in the list of running processes.
== What you can do ==
•ssh to the relevant server and check yourself for processes matching atftp (ps aux | grep atftpd ..)
•Use the exact command line from above that icinga uses
•If it's because there are 0 procs, check the status and try to start the service (systemctl status atftpd; systemctl start atftpd)
•If it's because there are 2 or more processes, stop it, make sure nothing is running anymore, start it again once
•If it still fails to start try to find out why (journalctl -xn ; view /var/log/syslog )
==== Where are files? ====
•The config file on the server is in /etc/default/atftpd. See the parameters there.
•The puppet class that sets it up is class install_server::tftp_server.
•The files it serves are in /srv/tftpboot, config files are in puppet, larger files are in volatile/tftpboot.
== How bad is it if this is down? ==
A: Not immediately critical but until it's fixed nobody can install new servers. It should be treated with High but not UBN priority.
Category:Runbooks
Monitoring/bad directory owner

The Icinga check called 'bad directory owner' checks if certain files are owned by the wrong user and need manual cleanup.
A defined type in puppet is used and located at modules/monitoring/manifests/icinga/bad_directory_owner.pp.
The most common case is files owned by root because somebody ran scripts as root when they should have used a deployment user.
As of May 2019 there is only one practical use of it on deployment servers where we check the mediawiki-staging directory.
The fix is to change ownership to the correct user.
This can be achieved by using find to both detect the affected files and run chown on them.
example: find /srv/mediawiki-staging/ -user root -exec chown mwdeploy:mwdeploy {} \;
also see: Find All The Files Owned By a Particular User / Group
Category:Runbooks
Monitoring/check conntrack

check_conntrack is an Icinga check that ensures that the nf_conntrack table is not full.
=== Things to do ===
* Check if the sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait is correctly set to 65, there's a race connection which sometimes make it apply the 120 seconds default:
https://phabricator.wikimedia.org/T136094. If that's the case, reset it to 65: "sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65"
* check the relevant "max" value: cat /proc/sys/net/netfilter/nf_conntrack_max
* check the current count: cat /proc/sys/net/netfilter/nf_conntrack_count
* raise the max value: echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
* permanently change the max value in puppet using sysctl::parameters (example)
* example ticket handling this alert: https://phabricator.wikimedia.org/T131028
Category:Runbooks
Monitoring/check cpufreq

This Icinga checks tests if CPUs are running at an expected frequency.
It uses this plugin to parse /proc/cpuinfo.
It has been added in this ticket as a reaction to CPU throttling issues on Dell R320.
Therefore it is currently only used on PowerEdge R320.
What you can do:
* Read and make a comment on T162850. Possibly reopen it because we have not seen these alert since firmware upgrades.
* rmmod acpid_pad
* https://phabricator.wikimedia.org/T162850
* https://gerrit.wikimedia.org/r/c/operations/puppet/+/348966
* https://gerrit.wikimedia.org/r/c/operations/puppet/+/348976
Category:Runbooks
Monitoring/check dhclient

This Icinga alert says it found a running DHCP client (dhclient) process.
We don't want any running DHCP clients. You can read here about the reason to add this check. 
It uses the command NRPE command /usr/lib/nagios/plugins/check_procs -w 0:0 -c 0:0 -C dhclient' on the affected host which you can run yourself.
So it expects exactly 0 processes called "dhclient" and any other number will create a CRIT.
You can manually check with "ps aux | grep dhclient" to look for the culprit. Kill the process.
* https://gerrit.wikimedia.org/r/c/operations/puppet/+/125989
Category:Monitoring
Category:Runbooks
Monitoring/check dsh groups

This Icinga alert checks if a MediaWiki appserver is a member of the mediawiki-installation DSH group.
The DSH groups control which hosts Scap deploys code to. In the Puppet repo you can see the list of servers now comes from Hiera from ./hieradata/common/scap/dsh.yaml where a reference is made to conftool.
 mediawiki-installation:
 conftool:
 - {'cluster': 'appserver', 'service': 'apache2'}
 - {'cluster': 'api_appserver', 'service': 'apache2'}
 - {'cluster': 'jobrunner', 'service': 'apache2'}
 - {'cluster': 'testserver', 'service': 'apache2'}
The conftool data is in ./conftool-data/node in the puppet repo as well. Check if the affected host name shows up in there. If not, you can add it.
Make sure first there is no existing hardware issue with this server by searching Phabricator for its host name.
If it is in there but you still get the alert, first run scap pull to fetch the latest code and then pool to add it to the pool. The Icinga alert should recover a little while later.
Alternatively you can pool the server from a management host such as cumin1001 using conftool commands.
== Inactive servers ==
There are two levels of depooling. It can be set as pooled=no (translates to enabled: False in pybal) which means it receives no public traffic, but still receives Scap deployments. Or it can be set as pooled=inactive which also removes it from the DSH group. This is generally only used if a host is unable to receive code updates, which then helpfully avoids Scap deployment errors due to unreachable servers.
If a server has come back online from maintenance or downtime and starts issuing Host not in mediawiki-installation dsh group alert, it is recommended to run scap pull to ensure it will not be serving outdated code to monitoring requests in production (T310225), and set pooled=no so that it receives Scap deployments going forward.
Once the maintenance or repair ticket is updated/resolved, and any other verification has taken place, it can also be repooled again. 
== History ==
Historically, before we had Salt or Cumin, we used DSH to run commands on multiple servers at once.
Server groups were text files in the Puppet repository and mediawiki-installation was one of them. Taking a server out of the pool meant making an edit to this text file. The use case of pooling severs is now managed via conftool/confctl. See also pool/depool app servers.
Category:Runbooks
Monitoring/check ferm

check_ferm is an Icinga check that ensures the ferm service is running.
ferm is a frontend for iptables. If this service fails it should not mean there are no more firewall rules but no changes can be applied until it is fixed.
=== Things to do ===
* Confirm there are still iptables rules loaded: iptables -L
* Check the status of systemd unit: systemctl status ferm
* Try starting the ferm service: systemctl start ferm
* This should show the error.
** A common one is that a DNS name appears in the config that can't be resolved. Especially happens when IPv4 records exist but IPv6 records are missing. In this case add the missing records.
* Look at the config files in /etc/ferm/conf.d/. If a specific one is causing the issue it's possible to stop puppet, move the file to another backup location and try starting the ferm service again.
Category:Runbooks
Monitoring/check rp filter disabled

Check rp_filter disabled is an Icinga check running on LVS servers via NRPE.
It is defined in modules/profile/manifests/lvs.pp.
It checks if rp filter is disabled in sysctl. The plugin it uses is /usr/lib/nagios/plugins/check_sysctl.
LVS servers require this setting in order to do their primary job of asymmetrically forwarding traffic.
"rp_filter" stands for "Reverse path filtering".
"when a machine with reverse path filtering enabled recieves a packet,
the machine will first check whether the source of the recived packet is reachable through the interface it came in.
If it is routable through the interface which it came, then the machine will accept the packet
If it is not routable through the interface, which it came, then the machine will drop that packet."
The command is executed on the LVS hosts and you can find the full command line in /etc/nagios/nrpe.d# view check_check_rp_filter_disabled.cfg.
== Why has this been added ==
The reason this check has been added is described Incident_documentation/20140203-LVS, which it was a follow-up for.
== What to do ==
* disable rp_filtering -> https://serverfault.com/questions/816393/disabling-rp-filter-on-one-interface
* also see: https://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html
* The check has been added in https://phabricator.wikimedia.org/rOPUP72bd9d36d75ef162223a378311e69cd3a7e8b3de and parameters have been tuned in https://phabricator.wikimedia.org/rOPUPbced02009ea59f7faf07c1608417024d6a5a693b
Category:Runbooks
Monitoring/dpkg

If Icinga reports a dpkg issue you should ssh to the affected host and run:
 dpkg -l | grep -v "^ii"
This shows all packages that are installed but in a "weird" (not 'ii') state. That's exactly what the Icinga check runs as well, so to get that back to OK you want to make sure all packages are in "ii" state.
What the other states mean exactly you can see at the very top of the output. One letter is the desired state and the other the current state.
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
Either remove broken packages (apt-get remove --purge <package>" or fix half installed packages (apt install <package>) or try to remove and reinstall it.
If it happens on a lot of servers at once or unsure open a ticket.
Category:Runbooks
Category:Monitoring
Monitoring/https unauthorized

check_https_unauthorized is used to ensure that a specific end point requires authentication. The icinga definition looks as follows
define command {
 command_name check_https_unauthorized
 command_line $USER1$/check_http --ssl --sni -I $HOSTADDRESS$ -H $ARG1$ -u $ARG2$ -e "HTTP/1.1 $ARG3$ "
}
So if we have a monitoring::services definition with the following `check_command` then the expectation is that visiting `https://www.example.org/foobar' will return a response containing `HTTP/1.1 401'
 "check_https_unauthorized!www.example.org!/foobar!401"
you can manually test this using curl
 curl --head -k https://www.example.org/foobar
Monitoring/puppet checkpuppetrun

This Icinga alert says a puppet agent failed to run.
Some things you can do are:
* ssh to the affected host and run Puppet manually with "sudo puppet agent -tv" to see verbose output
* check if you see any error, sometimes issues are transient or on the puppetmaster. If you do see an error try to find out which change caused it.
* Leave a comment on the change that broke it and/or ping the user who merged it on IRC and/or upload a fix yourself and merge it if appropriate.
* If that fails or you see in the Icinga web UI this alert has been triggering for a longer time, create a ticket about it.
if you are able to find a fix and only want to run it on the hosts with a failed puppet command you can use the following hack. however please 
pay attention to the cumin host selection and the command you are running to make sure you don't run the command on the wrong system. in the following example we just run hostname on all servers with a failed puppet run
$ sudo cumin '*' "bash -c '. /usr/local/share/bash/puppet-common.sh; last_run_success' || hostname"
Category:Monitoring
Category:Runbooks
Monitoring/root disk space

This Icinga alert says the disk space on the / file system is too low.
Some things you can do are:
* apt-get clean (deletes cached package files)
* find largest files recursively using find and examine if any can be deleted
* check /var/log/ and its sub directories for their sizes (du -hs *) and work your way down to find what is taking the most space
* gzip or tar some (older) log files
* move some files to /srv if that is mounted on a separate volume
* create a ticket and/or ping somebody on IRC who is familiar with the service to see what can be deleted
* create patches to add or improve logrotate setup
* create tickets to request more disk space / hardware if really needed
Category:Monitoring
Category:Runbooks
Monitoring/sre.puppet.sync-netbox-hiera.timer

== What it does ==
The systemd timer on the cumin hosts the cookbook that syncs data from Netbox data so its readable by puppet. It runs the cook book in check mode and checks if there are any uncommitted unsynced changes. This ensures that changes in Netbox are reflected in the puppet hiera data and hence in production in a timely manner and that there are no leftover changes that will show up to the next person running the sre.puppet.sync-netbox-hiera cookbook.
== What to do ==
To see what are the pending changes that are uncommitted, run in DRY-RUN mode the sre.puppet.sync-netbox-hiera cookbook as follows (see Spicerack/Cookbooks#Run a single Cookbook for more context on how to run a cookbook):
sudo cookbook -d sre.puppet.sync-netbox-hiera "test"
It will show all the pending DNS changes not yet committed.
If you are missing context get in touch with the people/team most likely related to those changes asking in the various SRE IRC channels.
You can also look at the Netbox Change Log.
Only if you are sure that those changes can be committed, run the cookbook again without the dry-run flag
This will propagate DNS changes to production, hence care must be taken to make sure they are correct and would not cause an outage.
Category:SRE Infrastructure Foundations
Monitoring/systemd unit state

The "systemd unit state" Icinga checks test if there are any failed systemd units. Units commonly include, but are not limited to, services (.service), mount points (.mount), devices (.device) and sockets (.socket). See for details on each of these.
For this type of alerts, you should ssh to the server in question and run systemctl list-units --state=failed (or the shortcut systemctl --failed) to check which unit is the one that has issues. 
You can use systemctl status unit, journalctl -u unit and journalctl -xn to see more details and logs to figure out why it failed. 
When using systemctl status unit take note of the Doc: entry which may provide further pointers to additional wikitech pages to further troubleshoot the issue.
Try manually starting it with systemctl start unit.
Sometimes the failure has been fixed already and you just need to clear the list of failed units with systemctl reset-failed unit.
== See also ==
* for an ongoing issue with "Systemd session creation fails under I/O load"
* Auditing systemd: solving failed units with systemctl
* How To Use Journalctl to View and Manipulate Systemd Logs
Category:Runbooks
Monitoring/unmerged changes

The "Unmerged changes on repository ..." alert is an Icinga check that tests whether there are pending unmerged changes on a git repository.
As of today it is used for the operations/puppet repo on various Puppet masters and for the mediawiki_config repo on deployment servers.
If the alert is about the puppet repo, the fix is to SSH to the relevant puppet master and run puppet-merge to check what is pending.
puppet-merge should tell you whose change it is and whether there are multiple ones pending and show you a diff. You probably want to ping the author
of the pending patch on IRC and ask them if they forgot and can merge. Or you find it trivial enough and are confident then you just merge it yourself.
If the alert is about the mediawiki_config repo you probably want to ping a Mediawiki deployer and ask what's up. You can also check the deployments calendar to see what happened last or ask jouncebot when the next deployment is happening.
Category:Runbooks
Monitoring procedure

Proposed monitoring procedure
Daily:
* Check nagios for new alerts.
* Fix simple issues such as daemons that need restarting or servers that can be rebooted remotely.
* Note any issues which need on-site attention at datacenter tasks.
* Pass responsibility for any more complex software issues to a competent staff member.
Weekly:
* Capacity check. Make sure key metrics such as application CPU utilisation and disk space usage are not approaching dangerous limits. 
* Publish a report detailing the times at which Nagios was checked, the issues noted, and any people notified. Or, make this information available continuously, for review on a weekly basis.
* Another team member should check the report and make sure that the monitoring done was of an appropriate standard.
One to two months:
* Capacity review. Analyse capacity metrics and report your findings. Notify the team of upcoming performance bottlenecks which might require hardware purchases.
* Report any long-term issues which have been left unfixed.
Category:Bot and monitoring
Monitoring sources

An enumeration of all sources of monitoring data
Origin 	Service 	Current 	Ideal 	Notes
Routers 	Netflow 	 	 	pmacct work in progress by Joel
Routers 	SNMP 	LibreNMS 	 	
Switches 	SNMP 	LibreNMS 	 	
Firewalls 	SNMP 	LibreNMS 	 	
Power Strips 	SNMP 	LibreNMS
Torrus 	 	
Netapp filers 	SNMP 	Torrus 	 	
Linux kernel 	CPU 	Ganglia 	 	
Linux kernel 	disk 	Ganglia
Icinga 	 	
Linux kernel 	memory 	Ganglia 	 	
Linux kernel 	network 	Ganglia 	 	
Linux kernel 	processes 	Ganglia 	 	
Linux kernel 	sysctls 	 	automated auditing to detect deviation 	
System services	dpkg database 	Icinga 	 	
System services	lldpd 	 	 	
System services	mdadm 	 	 	
System services	ntp 	Icinga 	 	
System services	puppet 	Icinga (clients) 	 	
System services	sshd 	Icinga: port 	Additional monitoring for awareness of security issues such as brute force attacks or odd logs	
Applications 	ActiveMQ 	Ganglia: perf
Icinga: perf 	 	
Applications 	Apache 	Ganglia: perf
Icinga: port 	 	
Applications 	Camus 	 	 	Not yet in production
Applications 	ElasticSearch 	Ganglia: perf
Icinga: health 	 	
Applications 	Exim 	Ganglia: perf stats on selected hosts
Icinga: port checks on selected hosts	Process stats emails
Monitor exim on every node 	
Applications 	(Fundraising components)	 	 	
Applications 	Gerrit 	Ganglia: perf
Icinga: proc 	 	
Applications 	Hadoop 	Ganglia: perf
Icinga: proc 	 	
Applications 	Jenkins 	built in email notifications?
Ganglia: perf
Icinga: proc	 	
Applications 	Kafka 	Ganglia: perf
Icinga: perf 	 	
Applications 	Lighttpd 	 	 	
Applications 	Logstash 	 	 	
Applications 	Lucene 	Ganglia: perf
Icinga: port 	 	
Applications 	LVS 	Icinga: implicit, via cluster service checks	dmesg
/proc/net/ip_vs_stats 	
Applications 	Mathoid? 	 	 	
Applications 	Mediawiki 	Graphite 	 	Is this superceeded by the Graphite work?
Applications 	Memcached 	Ganglia: perf
Icinga: port 	 	
Applications 	(Misc web services) 	 	 	
Applications 	Mysql / MariaDB 	Icinga 	 	
Applications 	Nginx 	Icinga (labelled HTTPS) 	 	
Applications 	Openstack (Labs) 	 	 	
Applications 	Parsoid 	Icinga 	 	
Applications 	PDF renderers? 	 	 	
Applications 	PowerDNS (pdns) 	Ganglia: perf 	 	
Applications 	Redis 	Ganglia: perf
Icinga: port 	 	
Applications 	Storoid? 	 	 	AFAIK this service doesn't exist yet
Applications 	Swift 	Ganglia: perf
Icinga: proc 	 	
Applications 	Squid 	Icinga: port 	 	
Applications 	Timed media handler 	 	 	
Applications 	Twemproxy (Nutcracker) 	Icinga: proc 	 	
Applications 	Varnish 	Ganglia: perf
Icinga: ports, procs 	 	
Applications 	Visual Editor 	 	 	
Applications 	Zuul 	Icinga: port, proc 	 	
Category:Bot and monitoring