nyuuzyou/wikis huggingface dataset:
Portal:Cloud VPS/Admin/Email

This page contains information on the email setup and how to manage it inside Cloud VPS.
== Servers ==
The general outgoing SMTP servers are hosted in the cloudinfra project. At the time of this writing the instances in use are:
* mx-out03.cloudinfra.eqiad1.wikimedia.cloud 
* mx-out04.cloudinfra.eqiad1.wikimedia.cloud
Each servers uses a floating IP so we can reference the servers in SPF records.
Additionally, each floating IP is referenced by a public FQDN:
* mx-out-a.wmcloud.org --- 185.15.56.18
* mx-out-b.wmcloud.org --- 185.15.56.19
Servers use the puppet role role::mail::smarthost::wmcs plus some hiera config:
profile::mail::smarthost::cert_name: mx-out-a.wmflabs.org
profile::mail::smarthost::cert_subjects: mx-out-b.wmflabs.org
profile::mail::smarthost::exim_primary_hostname: mx-out-a.wmflabs.org
profile::standard::has_default_mail_relay: false
standard::has_default_mail_relay: false
== SPF ==
TODO: put here some info.
== Toolforge ==
The Toolforge email setup also uses Exim, but a bit different setup than the general Cloud VPS servers.
Email server use the puppet role role::wmcs::toolforge::mailrelay and hiera config:
profile::standard::has_default_mail_relay: false
profile::toolforge::active_mail_relay: tools-mail-02.tools.eqiad1.wikimedia.cloud
profile::toolforge::base::is_mail_relay: true
profile::toolforge::mailrelay::external_hostname: mail.tools.wmflabs.org
One of the features offered to Toolforge tools is $HOME/.forward processing via exim to route incoming emails to other addresses. This functionality relies on the tool and user $HOME directories being mounted on the mail servers (NFS).
== Operations ==
Some information on administrative tasks, specifically related to how to handle Exim.
A few pointers:
* Exim is hard. This documentation is not: http://bradthemad.org/tech/notes/exim_cheatsheet.php
* sudo -i exim -q -v to manually run over the queue once
* Log files are in /var/log/exim4/*
=== Total count for queue ===
$ sudo -i exim -bpc
286
=== Count for the queue per user ===
$ sudo -i exim -bp | grep '<' | awk '{print $4}' | sort | uniq -c |sort -n
 1 <someuser@some.tld>
 285 <>
=== Stats ===
$ sudo -i exim -bp | exiqsumm
Count Volume Oldest Newest Domain
----- ------ ------ ------ ------
 1 106KB 28h 28h A.tld
 1 4198 2h 2h B.tld
 1 106KB 7h 7h C.tld
 1 106KB 30h 30h D.tld
 1 106KB 27h 27h E.tld
 1 106KB 29h 29h F.tld
 1 106KB 25h 25h G.tld
 1 106KB 26h 26h H.tld
 1 106KB 31h 31h I.tld
 1 106KB 4h 4h J.tld
 2 4300 31h 13h K.tld
 271 629KB 22h 8h tools.wmflabs.org
 1 87KB 5h 5h L.tld
 1 57KB 12h 12h M.tld
 1 106KB 46m 46m N.tld
---------------------------------------------------------------
 286 1842KB 31h 46m TOTAL
=== Blackhole an in-bound address ===
in /etc/aliases add a new line: <address>: :blackhole:
If this was for Tools exim to blacklist the user 'wiktcapt' we could do:
wiktcapt: :blackhole:
== See also ==
* Portal:Cloud_VPS/Admin/DNS -- our DNS setups, our domains, etc (admin docs).
* Help:Toolforge/Email -- email in Toolforge (end user docs)
* Help:Email in Cloud VPS -- email in Cloud VPS (end user docs)
Portal:Cloud VPS/Admin/Galera

The databases for OpenStack services are stored on a Galera cluster hosted on the cloudcontrol nodes. 
== Deployment ==
This cluster does not use custom WMF packages; it runs the standard packages from upstream Debian.
The cluster is active/active/active which means that write actions can be taken on any of the cloudcontrol nodes. The only thing preventing to do so is the haproxy in front of them, sending requests just to one of them.
Directions for standing up a new cluster are included in puppet/modules/galera/manifests/init.pp.
== DB setup ==
OpenStack services tend to use connection pooling, opening many long-lived connections to each database. For this reason, our Galera config has extremely long connection timeouts and very high connection limits.
== General Operations ==
=== Restarting the local mariadb process ===
Don't configure puppet to restart this. Only restart one node at a time to avoid split-brain.
* Tell haproxy the database is down just to be safe with sudo touch /tmp/galera.disabled
* To begin, disable puppet so it can't mess with things.
* sudo systemctl stop mariadb
* In another shell run sudo journalctl -u mariadb.service -f to verify it cleanly exits. It can take a few moments or quite a while.
* sudo systemctl start mariadb
* Again, watch journalctl to see that it comes up alright.
* Once it is up, access the mysql shell with sudo -i mysql -u root
* Run SHOW STATUS LIKE "wsrep_local_state_comment"; and SHOW STATUS LIKE "wsrep_ready". If the first isn't "joined" or "synced" the node isn't ready. The second needs to be "ON" or almost all queries against it will fail. If you are manually handling haproxy to keep it out of the cluster, don't include it until you get one of those statuses.
* Repool the database in haproxy by removing the /tmp/galera.disabled file and re-enable puppet
=== Depool a single node ===
* Disable Puppet on the node
* sudo touch /tmp/galera.disabled
== Troubleshooting ==
Please read Portal:Cloud_VPS/Admin/Troubleshooting#Galera.
== See also ==
TODO.
Portal:Cloud VPS/Admin/Haproxy

This page contains information about our HAProxy configuration in the CloudVPS environment.
= OpenStack API Endpoints =
== Backend HA Status ==
Backend services that are full active / active will automatically be pooled and depooled by HAProxy. As long as there is one instance available there will be no interruption in service.
=== EQIAD1 ===
Service 	active backends 	standby backends	notes
Designate 	cloudservices1003cloudservices1004	NA 	
Glance API 	cloudcontrol1003 	cloudcontrol1004	Waiting on shared storage (CEPH) for full active/active
Glance Registry	cloudcontrol1003 	cloudcontrol1004	Waiting on shared storage (CEPH) for full active/active
Keystone 	cloudcontrol1003cloudcontrol1004 	NA 	
Neutron 	cloudcontrol1003cloudcontrol1004 	NA 	
Nova 	cloudcontrol1003cloudcontrol1004 	NA 	
== Port Mappings ==
Service 	Backend Port	Frontend Port
Designate API 	9001 	9001
Glance API 	19292 	9292
Glance Registry	19191 	9191
Keystone Admin 	15000 	5000
Keystone Public	45357 	35357
Neutron 	19696 	9696
Nova API 	18774 	8774
Nova Metadata 	18775 	8775
= Load Balancing and Session Handling =
Backend connections for each service are dynamically directed to the server with the least amount of connections using the `leastconn` load balancing algorithm. 
Session persistence is maintained with HAProxy stick tables using the IP address of the client. This will ensure new client connections get forwarded to the same server every time.
= Monitoring =
== Icinga ==
Icinga is configured with the following HAProxy process checks:
* nrpe_check!check_haproxy_alive!10
* nrpe_check!check_haproxy!10
== Prometheus ==
The prometheus haproxy exporter is collecting haproxy backend and frontend metrics directly from haproxy through the URL http://localhost:9900/stats;csv. This URL is read-only and only provides statistic information through the loopback interface.
Grafana dashboard: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats
= Troubleshooting =
== Monitoring ==
HAProxy statistics and metrics can be queried through the "stats socket" at `/run/haproxy/haproxy.sock`.
Process information:
 echo "show info" | sudo socat /run/haproxy/haproxy.sock stdio
Statistics:
 echo "show stat" | sudo socat /run/haproxy/haproxy.sock stdio
== Log files ==
* /var/log/haproxy/haproxy.log
= Primary Failover =
== Design ==
Each controller is running HAProxy with no cross dependencies on other controllers or HAProxy instances. Each load balancer is operating on its own with no shared information like client sessions between them. 
== DNS ==
Each region has a dedicated FQDN (openstack.<region>.wikimediacloud.org) which maps to one of the OpenStack controllers running HAProxy. In the event of failure or maintenance this DNS entry will need to be remapped to the other controller in the operations/dns git repository at https://gerrit.wikimedia.org/r/admin/projects/operations/dns
Portal:Cloud VPS/Admin/Horizon

= deploying updates to Horizon =
Horizon is deployed in a docker container using branch that corresponds to OpenStack upstream release branches. Horizon is generally backwards-compatible with older OpenStack APIs, so its release cadence does not need to be perfectly in sync with the rest of Cloud-VPS; generally Horizon is several releases newer.
Currently (2023-09-07) the individual horizon components are hosted in gerrit and the top-level deployment repo on gitlab. 
= Refreshing upstream code base =
Most repos are forks of upstream code. Repos starting with 'wmcs-' are in-house UIs which don't have upstreams to sync with.
Ideally each forked repo has a branch named after an upstream version (a release name like zed or antelope, or an LTS release like 2023.01); this branch should reflect the upstream branch of the same name with local wmf patches applied on top. Not all upstream projects branch with every version, so we aren't 100% consistent about this.
To update (using designate-dashboard as an example):
On your local development system clone the WMCS and upstream git repositories
$ git clone "ssh://<username>@gerrit.wikimedia.org:29418/openstack/horizon/designate-dashboard" && \
 scp -p -P 29418 <username>@gerrit.wikimedia.org:hooks/commit-msg "designate-dashboard/.git/hooks/"
$ cd designate-dashboard
$ git remote add upstream https://opendev.org/openstack/designate-dashboard.git
$ git checkout -b train origin/train
$ git pull upstream stable/train --rebase --stat
If there are no conflicts you can rebase the upstream changes on top of any local customizations
$ git rebase origin/train
Requires the labs-striker group https://gerrit.wikimedia.org/r/admin/groups/3db64b50b88a848f800e95c946fab6a84b8fec2b
$ git push origin HEAD:refs/for/train%topic=upstream_refresh
= Deploying =
The docker image is built by gitlab whenever a new tag is pushed to the deploy repo.
To apply a specific patch:
* Apply patch to the master branch of <submodule>
* Cherry-pick that patch to <release branch> (e.g. 'train') of <submodule> and merge
* make a submodule update patch on <release branch> in the deploy repo and push
* tag the latest release, typically with a date stamp:
$ git tag 2023-09-06-225100
$ git push origin 2023-09-06-225100 # this starts an image build on gitlab
* Once gitlab completes the build (and, assuming the build succeeds) this new image can be deployed by updating hiera settings, like this example.
Portal:Cloud VPS/Admin/Infrastructure projects

A number of projects should be considered infrastructure for Toolforge and Cloud VPS. These projects are open to participation by technical volunteers, but ultimately the Cloud VPS admin team and the Foundation staff on that team are responsible for making sure that they work (or are properly removed from use).
== Projects ==
=== admin ===
; purpose
: magic superuser project for OpenStack
=== admin-monitoring ===
; purpose
: used by automated nova-fullstack tests
=== bastion ===
; purpose
: ssh jump hosts for accessing instances without public ips
=== clouddb-services ===
; purpose
: Hosting for database as a service instances (ToolsDB, postgres for Maps & wikilabels)
=== cloudinfra ===
; purpose
: Shared services for Cloud VPS
:* Puppetmasters
:* Outbound mail relay servers
:* NTP servers
:* Cumin servers
=== cloudstore ===
; purpose
: Testing NFS things. Someday maybe hosting NFS servers that face other projects.
=== cloudvirt-canary ===
; purpose
: canary instances that we run on all hypervisor servers
=== metricsinfra ===
; purpose
: Instance level monitoring and alerting. Intended to replace #shinken.
: See also Portal:Cloud VPS/Admin/Monitoring#Monitoring for Cloud VPS
=== project-proxy ===
; purpose
: HTTP/TLS front proxies for *.wmcloud.org and *.wmflabs.org. Includes custom Horizon integration to create and remove proxied hostnames.
=== paws ===
; purpose
: PAWS
=== quarry ===
; purpose
: m:Research:Quarry
=== redirects ===
; purpose
: Nova Resource:Redirects
=== striker ===
; purpose
: Testing project for Striker
=== testlabs ===
; purpose
: Misc testing
=== tools ===
; purpose
: Portal:Toolforge
=== toolsbeta ===
; purpose
: Testing for Portal:Toolforge infrastructure changes
=== trove ===
; purpose
: Service instances created by the Openstack Trove component (DBaaS).
== See also ==
* Portal:Toolforge/Admin/Infrastructure tools
* Portal:Cloud VPS/Admin/Projects
Portal:Cloud VPS/Admin/Interest matrix

This is an attempt to identify who knows what about which Cloud Services subject areas. Started: October 2018.
On a scale of 0-5 where 0 means "no idea" and 5 means "expert", please rate your knowledge of the below skills. All prolific (incl past) contributors to these areas, please add yourselves freely!
Category 	Skill 	Staff 	Staff 	Staff	Staff	Staff 	Staff	Staff 	Staff 	Contractors	Volunteers	Volunteers	Volunteers
Category 	Skill 	Andrew	Arturo	Bryan	David	Francesco	Rook 	Slavina	Nicholas	Raymond 	KunalKunal	Chico 	Taavi
VM Infrastructure	NFS, system-level 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
VM Infrastructure	NFS integration e.g. nfs-exportd	🛠️ 	 	 	 	 	 	 	 	 	 	 	
VM Infrastructure	Ceph 	🔧 	 	 	 	 	 	 	 	 	 	 	
VM Infrastructure	VM auth things: Ldap/Pam/Groups 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
Toolforge 	Toolforge K8s 	 	 	 	 	 	 	 	 	 	 	 	
Toolforge 	Toolforge Grid 	 	 	 	 	 	 	 	 	 	 	 	
Web services 	Proxies (project and tools) 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
Web services 	PAWS 	 	 	 	 	 	🐾 	 	 	 	 	 	
Web services 	Quarry 	 	 	 	 	 	 	 	 	 	 	 	
Web services 	Toolsadmin 	 	 	 	 	 	 	 	 	 	 	 	
Web services 	Openstack Browser 	🔧 	 	 	 	 	 	 	 	 	 	 	
OpenStack 	Nova and Glance 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
OpenStack 	Keystone 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
OpenStack 	Designate 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
OpenStack 	Horizon 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
OpenStack 	Neutron 	🔧 	 	 	 	 	 	 	 	 	 	 	
Databases 	Replicas 	🔧 	 	 	 	 	 	 	 	 	 	 	
Databases 	Toolsdb 	🔧 	 	 	 	 	 	 	 	 	 	 	
Other skills 	DNS 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Puppet 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Grafana 	🔧 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Prometheus 	🐾 	 	 	 	 	 	 	 	 	 	 	
Other skills 	MediaWiki 	🔧 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Python coding 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Golang coding 	 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Shellscript coding 	🛠️ 	 	 	 	 	 	 	 	 	 	 	
Other skills 	Ruby coding 	🔧 	 	 	 	 	 	 	 	 	 	 	
Category 	Skill 	Andrew	Arturo	Bryan	David	Francesco	Rook 	Slavina	Nicholas	Raymond 	Alex 	Chico 	Taavi
Category 	Skill 	Staff 	Staff 	Staff	Staff	Staff 	Staff	Staff 	Staff 	Contractors	Volunteers	Volunteers	Volunteers
* As they are self-assessments, each rating should generally be compared along columns and not along rows – i.e. people's self-ratings in one field X should be compared against their self-ratings in another field Y, not against other people's self-ratings in the same field X.
== Legend ==
* 🛠️ Regularly works on/with this
* 🔧 Occasionally works on/with this
* 💡 Can mentor/teach someone
* 🐾 Is currently learning/wants to start learning
Portal:Cloud VPS/Admin/Local customization and hacks

This page contains information about local customization and hacks for Cloud VPS's Openstack.
Originally started at https://phabricator.wikimedia.org/T324774
This is a mix of what makes Cloud VPS's identity and inherited technical debt. Some of these things worth keeping, others don't. In general, they are diversions from an common out-of-the-box openstack cloud deployment,
== List ==
 Openstack local customization and hacks for Cloud VPS
Name 	Component 	Tech debt	Worth keeping	Comment
Project IDs 	keystone 	yes 	yes? 	Forcing project id == project name (via code patch)
Custom roles & policies (RBAC) 	all 	no? 	no 	The observer/member/projectadmin roles. No big deal, but they are now kind of supported natively by upstream. See phab:T330759
VM hostname 	nova 	no? 	yes 	Constrain VM hostname via regex (via code patch)
designate-sink plugins 	designate 	no? 	yes 	Custom designate-sink plugin to create/destroy DNS entries on VM creation/deletion and to do other cleanup on VM deletion (puppet cert deletion, config cleanup, etc)
cinder-backups posix backend 	cinder 	yes? 	yes? 	Use of cinder-backup posix backend, which exists in upstream code but seems to generally not be used in many real deployments.
backported bugfixes 	all 	no 	yes 	Backported bugfixes for cinder, trove, neutron, etc
No BYOI 	glance 	no 	yes 	We don't allow bring your own VM image. No big deal.
Puppet for VMs 	glance 	no 	yes? 	Not really related to how we deploy openstack. No big deal.
LDAP integration 	glance, keystone	no 	yes 	For Wikimedia Developer accounts etc. No big deal.
Cumin management 	glance 	no 	yes 	Not really related to how we deploy openstack. No big deal.
Horizon customization 	horizon 	no 	yes 	Removed some panels, added some other custom panels. No big deal.
neutron-linuxbridge-agent 	neutron 	yes 	no 	May no longer be supported upstream soon.
neutron flat network topology 	neutron 	no? 	no 	While fully upstream-supported use case, it greatly reduces what we can do with the SDN layer.
deploy via puppet & deb packages	all 	yes? 	no 	Our custom puppet manifests are hand-crafted and not used by anyone else in the community. We are suspicious of the deb packages may also have little community usage?
Portal:Cloud VPS/Admin/Maintenance notification

When scheduled system maintenance will cause service outages, we try to provide our users with advance notice of the downtime. For serious outages, we also try to provide advance notice to our users' users. This table is meant to provide general guidelines but does not represent a formal commitment between WMCS admins and users. Our uptime stats may be good, but our uptime commitment is 'best effort'.
All maintenance tasks should be logged in the server admin log. Project-specific tasks are to be logged per-prioject, Toolforge actions should be logged on the tools log, and cloud-wide actions should be logged in the admin log.
Most advance notices are provided to users via mailing lists: cloud-announce, wikitech-l, and wikitech-ambassadors.
 	Scheduled maintenance,
no expected downtime	< 5 minutes of downtime,
restarts or resets	expected downtime >= 5 minutes	expected downtime > 1 hour
Select user-managed tools, services, projects	SAL	SAL	1 day in advance:
* Notify specific tool owners or project admins	7 days in advance:
* Notify specific tool owners or project admins
WMCS-managed project
(e.g., PAWS) 	SAL	SAL	SAL 	1 day in advance:
* Cloud-announce
OpenStack control plane 	SAL	SAL	SAL 	1 day in advance:
* Cloud-announce
All of Toolforge,
Toolforge grid, or Toolforge k8s	SAL	1 day in advance:
* cloud-announce	7 days in advance:
* Cloud-announce
* Wikitech-l	7 days in advance:
* Cloud-announce
* Wikitech-l
* Wikitech-ambassadors
* Tech News (maybe)
All of Cloud VPS
(including Toolforge) 	SAL	1 day in advance:
* Cloud-announce	7 days in advance:
* cloud-announce
* Wikitech-l	7 days in advance:
* Cloud-announce
* Wikitech-l
* Wikitech-ambassadors
* Tech News (maybe)
Portal:Cloud VPS/Admin/Projects/Potentially unmaintained

The following projects appear to have a lack of maintenance. Some are based on the progress shown on the Trusty deprecation tickets in Phabricator, September-December 2018, and some are based on quota requests closed for inactivity. Presence on this list can indicate maintenance, communication, or responsibility issues.
Next time we run a purge we should be careful before accepting an for these projects.
Project 	Basis
design 	T204502
getstarted 	T204508
osmit 	T204527
queryrapi 	T204683
telnet 	T204694
wikidataconcepts 	T204695
wikidocumentaries	T214907
rcm 	T190788
codereview 	T190788
analytics 	T203374
dumps 	T159930
Portal:Cloud VPS/Admin/Runbooks

This page contains basic resources for developers who want to author and publish runbooks for Cloud VPS on Wikitech.
==Before you begin==
Permissions: Some processes documented in runbooks for Cloud VPS will need varying levels of permission to perform. Make sure you have appropriate permissions to complete your task.
==What is a runbook?==
A runbook is a detailed set of instructions that explain how to perform a common task or procedure, so it can be easily and accurately repeated by others. Runbooks are particularly useful for incident response operations. Creating runbooks in response to specific incidents, makes it possible for people to repeat the steps in response to similar incidents. Because they are often used to help people respond to incidents quickly, they should be easy to read and follow, consistent, and accurate.
==When should you use a runbook?==
A runbook should be used whenever a common task or procedure may need to be repeated by mulitple people.
==Tips for building useful runbooks==
*Keep them up-to-date. Revisit the instructions after each incident to make sure they are clear and accurate.
* Keep them simple. These are sets of detailed instructions, and they do not require extra history or context.
** Publish a separate runbook for each issue or incident
*Test them. Make sure the instructions are repeatable by others by asking others to follow them and provide feedback.
* Follow a template. Make sure your runbooks follow a template, so that people can find and update information easily.
===Cloud VPS runbook templates===
Instructions for each issue or alert should have its own page in order to reduce unnecessary information. You can see an example here with the issue Check for VMs leaked by the nova-fullstack test. Note that the page addresses one issue only.
You can find a template/outline for creating runbooks here: Portal:Cloud_VPS/Admin/Runbooks/Runbook_template
== Where to publish runbooks for Cloud VPS ==
Where to publish Cloud VPS runbooks
Portal:Cloud_VPS/Admin/Runbooks/(PAGE NAME)
Note: This is a subpage of admin documentation. Some procedures may require advanced admin permissions to complete.
Note: Many runbooks for Cloud VPS will include procedures that can only be followed by individuals with admin access. In order to avoid confusion and frustration for general users, you should note at the content level when a procedure will require admin permissions.
When an entire runbook requires admin permissions to complete procedures, mark it with the following template:
When runbooks include information for general users and special instructions for admins, mark any instructions (inline) for admins with the following template:
== Existing runbooks ==
==See Also==
*:Category:Runbooks - Wikitech pages marked as runbooks.
*WMCS Admin documentation - Advanced documenation for WMCS administrators.
*Runbooks entry on Wikipedia
Category:Documentation
 Cloud VPS