nyuuzyou/wikis huggingface dataset:
Portal:Cloud VPS/Admin/Virt capacity

== Assessing current usage ==
To see how busy and/or full the existing cloudvirts are, visit the capacity planning grafana dashboard or the eqiad1 summary dashboard.
The best place to see what cloudvirts are currently pooled or depooled is the scheduler pool list in hiera. Ideally, the comments next to that definition will explain the state of each cloudvirt.
== How much is enough? ==
We have adequate current capacity if:
* All cloudvirts are averaging less than 50% CPU usage with little to no CPU blockage during usage spikes
* Even the busiest cloudvirts still have enough free RAM that we're never in danger of running OOM. In theory the nova scheduler ensures this but it's worth keeping an eye on.
* No cloudvirt has a drive more than 80% full. Because the scheduler permits 1.5x disk allocation, over 70% means that in an extreme case (where all COW VMs use all their disk space at once) we could hit 100% usage.
We have adequate future capacity if:
* We can experience 30% usage growth and have all of the above still be true. As we begin to eat into that 30% reserve it's time to start thinking about ordering new hardware.
== Idle capacity, aka Lifeboats ==
We try to maintain one 'lifeboat' for every 10 cloudvirts. For example, as of March of 2019 we have 28 active cloudvirts, so should have 3 cloudvirts up and running but hosting no actual instances (or, actually, hosting one canary VM per host). The lifeboats are there in case VMs need to be evacuated off of a different cloudvirt because of a disk failure, impending DC work, or other issue that will take it off-line.
This is complicated by the fact that our cloudvirts are all different sizes, so some lifeboats (e.g. cloudvirt1008) might not be big enough to rescue all the VMs from a larger cloudvirt (e.g. cloudvirt1025). Best to avoid using only the smallest/slowest cloudvirts as lifeboats; ideally at least one of the newest or biggest cloudvirts is empty and ready at all times.
== What happens if a cloudvirt is overloaded? ==
There are different failure cases depending on which resource a cloudvirt runs out of. These are the problems we are hoping to avoid by maintaining adequate capacity:
RAM: Instances will spontaneously shut down as libvirt and/or the OOM-killer struggles to free up memory for the system.
CPU: Instances will run more slowly or present as obviously CPU-starved
Disk: Instances will fail to write to their own file systems. Various other strange things can happen, including the potential of file corruption.
Portal:Cloud VPS/Admin/notes/Neutron ideal model/meeting-2018-11-29

= CloudVPS ideal model =
* https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron
* https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_ideal_model
* https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Network_and_Policy
Attendants:
* chase
* andrew
* giovanni
* faidon
* arzhel
* arturo
== topics ==
* transport network renumbering from private to public:
** non-controversial
** non-urgent
** big impact for our users
** idea: adding another interface to our current virtual router with the new transport net
*** add another subnet object using the same network object (same physical network)
** old private and new public addresses use the same VLAN numbers
* historical context:
** nova-network to neutron migration enabled many more "cloud-native" features, like AWS and the like
** ideal: hypervisor has a virtual router. But that has other complications, complexity (direct virtual routing)
*** this doesn't prevent from having central router because of floating IPs
* neutron complexities when managing virtual router
** we don't have workflows like with juniper routers
* ideal model:
** NAT 172 to where and when? (the dmz_cidr mechanisms)
*** what happens when an instance talks to a mediawiki API/endpoint/URL: we see VM addrss in logs
*** should that no longer be the case?
*** What happens in Toolforge?
*** We don't have enough public addressed to it sanely?
*** IPv6 could be a solution
*** what's the difference on having CloudVPS from a NAT? What's the benefit on different addressing?
*** Faidon is pretty sure that not having dmz_cidr is better.
*** actual requirements for dmz_cidr? NFS for sure. Others, will have to be reviewed
*** unmaintainable ACLs because boundaries aren't clear (between Cloud VPS and prod)
** Faidon suggests that separation is a different topic from other ideal features (like IPv6)
** CloudVPS is just like AWS from the production point of view
*** implications in addressing
*** separate routers (cloudnet)
*** border ACLs, BGP, etc: CLoudVPS is The Internet from the prod POV
Portal:Cloud VPS/Admin/notes/Service predictions for cross realm situation

This page contains information and some predictions on how cloud-related services will go when we migrate them for compliance with the Cross-Realm_traffic_guidelines.
What 	Current model 	Predicted future model	Comments 	Level of effort
Cloud rec DNS 	case 4 compliant 	--- 	 	Medium
Cloud auth DNS 	case 4 compliant 	--- 	 	Medium
Toolforge project NFS 	case 1 compliant 	--- 	See phab:T291405 	High
Toolforge home NFS 	case 1 compliant 	--- 	See phab:T291405 	High
Misc project NFS 	case 1 compliant 	--- 	See phab:T291405 	High
Dumps NFS 	case 2 compliant 	--- 	Recently done, see https://phabricator.wikimedia.org/T272397 	---
Metrics services 	cloudmetrics servers in cloud-support production VLAN, with private IPv4. Cloud egress NAT exception. 	case 1 	See https://phabricator.wikimedia.org/T336854 	Medium
Wiki replicas 	case 2 compliant 	case 4 	Might move to case 4 in next iteration 	---
openstack APIs 	case 4 compliant 	--- 	 	Medium
cloudelastic cluster (elasticsearch) 	case 2 compliant 	 	cloudelastic servers in production public VLAN, with public IPv4. No cloud egress NAT exception. RO access from cloud using local firewalling in cloudelastic servers. More info: Help:CirrusSearch_elasticsearch_replicas.	---
cloud hypervisors (cloudvirt servers) 	case 4 compliant 	--- 	 	---
cloud network (cloudnet & cloudgw servers)	case 4 compliant 	--- 	 	---
ceph storage farm 	TODO 	case 4 	 	TODO
horizon 	labweb servers in production public VLAN with public IPv4. 	case 3 	 	Low
LDAP 	LDAP servers in production public VLAN with public IPv4. Cloud egress NAT exception. 	case 2 	 	Low
NFS backups 	cloudbackup servers in production private VLAN with private IPv4. 	case 4 	--- 	Medium
Gerrit 	gerrit.wikimedia.org & gerrit-replica.wikimedia.org servers in production public VLAN with public IPv4. Cloud egress NAT exception.	case 2 	Leave until deprecation by Gitlab? 	Low
APT repositories 	case 2 compliant 	--- 	See https://phabricator.wikimedia.org/T298042 	Low
kraz (IRCD) 	case 2 compliant 	 	See https://phabricator.wikimedia.org/T280225 	---
Maps NFS 	case 1 compliant 	--- 	See phab:T291405 	Medium
Scratch NFS 	case 1 compliant 	--- 	See phab:T291405 	Medium
main Wikis & API 	servers in production public VLAN with pulic IPv4. Cloud egress NAT exception. 	case 2 	Dropping the NAT exception is challenging, many moving parts. See News/2021 CloudVPS NAT wikis and https://phabricator.wikimedia.org/T209011 	High
nova-fullstack 	case 3 compliant 	--- 	See https://phabricator.wikimedia.org/T272587 	---
What 	Current model 	Predicted future model	Comments 	Level of effort
== See also ==
 Phabricator ticket T207536: Move various support services for Cloud VPS currently in prod into their own instances
Portal:Cloud VPS/Bare Metal

The concept of Cloud VPS 'bare metal' has taken on a life of its own and caused confusion in the tradition of Labs labs labs. This page attempts to explain some of the history behind it, and also state the current position of the Cloud Services team.
== Current status ==
"Bare metal" instances are currently not provided by Wikimedia Cloud VPS.
== History ==
The Cloud Services team was interested in Openstack Ironic and once had a quarterly goal for investigating the feasibility of using Openstack to manage and allocate physical hardware for tenant use. Ironic depends on Neutron which we figured out quickly was enough work on its own. We shifted focus to determine how maintainable it would be to hack our own solution for the same basic purpose. To that end we acquired a PoC server that was allocated into the instance VLAN. We hacked in DNS via LUA, and created DHCP reservations.
At the end of the quarterly goal we had a pilot host that somewhat functioned as expected. We did not sort out console access, security issues, support or really anything formal for general use. The long tail of this is essentially recreating Ironic on our own. We agreed as a team (coren/andrew/yuvi/chase) that this is not a supportable model at our level of staffing and to wait for Ironic and reassess. We have only the initial allocations from PoC which we kept and let subbu use with the understanding that it was not supported and would eventually be phased out.
We have talked about a possible specific exception if a compelling case is made but so far this has not happened.
== Capabilities & Limitations of the Proof of Concept ==
* Physical machine allocated for tenant use within the instance VLAN
* Uses the Cloud VPS puppetmaster or the puppetmaster of the appropriate tenant
* Cannot have NFS (wasn't part of the PoC)
* Has no association with security groups as it is our own hackery
* Has DNS/DHCP hacked in
* Is imaged from production carbon installer
* Is not tied in with any of the Openstack ecosystem (as such things like floating IPs do not work)
* Is something the Cloud Services team agreed is not a supportable model currently
== Things that could be confused with 'bare metal in Cloud VPS' ==
* Servers in the labs-support VLAN for production services (not part of the Openstack ecosystem) in support of Cloud VPS instances (production hardware supporting Cloud VPS instances - labsdb, cloudmetrics, etc)
* Cloud VPS/Openstack hardware that runs the components to keep Cloud VPS running like nova or designate. (production hardware that powers Cloud VPS)
* Servers in standard production VLANs providing services to Cloud VPS instances such as ldap (production hardware or ganeti VM's that support core Cloud VPS functionality)
* Servers acting as OpenStack Compute (nova) nodes which as not in the normal scheduler pool and instead hosting instances specifically placed there by OpenStack administrators (such as some of the clouddb-services project instances).
Portal:Cloud VPS/Infrastructure

Cloud VPS is a virtualization cloud that uses OpenStack Compute. Base images are managed with Glance, and authentication uses LDAP-backed Keystone.
Cloud VPS currently runs in a single data center in Ashburn, Virginia. In the future, it will span two or more data centers, with a slightly different configuration in each.
For troubleshooting immediate issues, visit Portal:Cloud_VPS/Admin/Troubleshooting.
== Cloud VPS Eqiad (Ashburn, VA) ==
=== Regions ===
We have one region: eqiad1-r (also referred to as eqiad1).
The eqiad name is based on the naming convention for data centers.
=== Horizon ===
Most users will manage their virtual servers using Horizon. Horizon is an upstream OpenStack web interface for the OpenStack APIs. Our Horizon site also includes several custom dashboards to access special WMCS features not available in stock Horizon.
Horizon is hosted on cloudweb1003.wikimedia.org and cloudweb1004.wikimedia.org and can be accessed at https://horizon.wikimedia.org.
Individual user accounts on WMCS can also be created via Striker which is at https://toolsadmin.wikimedia.org. Currently, any account created there is automatically added to the Tools project.
=== Controller ===
The OpenStack controller box cloudcontrol1005 runs the Glance and Keystone services, as well as nova-conductor and nova-scheduler. It is also the preferred place to access the OpenStack command-line client.
Other nodes cloudcontrol1006, cloudcontrol1007 are present as well.
=== Network ===
In the eqiad1-r region, we use Openstack Neutron which runs on servers cloudnet1003 and cloudnet1004.
=== Virtualization ===
See the deployments page for a list of hypervisors per region and their current status.
Cloudvirt hosts (also known as hypervisors) are pooled or depooled using the profile::openstack::eqiad1::nova::scheduler_pool key in Puppet Hiera.
=== Storage ===
Most Cloud VPS projects do not use shared NFS storage. If they need NFS, these are the available options:
* Each member of a project has a project-wide shared home directory.
* The project has a public shared volume, generally mounted to /data/project
All of the above are hosted on various NFS servers (clouddumps* and cloudstore*).
=== Monitoring ===
Most OpenStack-related services are monitored in Icinga just like other production services.
=== LDAP ===
LDAP is used for services throughout the WMF. The same LDAP database keeps track of project management and SSH keys for logins on VPS servers. LDAP is hosted on seaborgium and neptunium; The LDAP server software is OpenLDAP.
Each Cloud VPS instance has an /etc/ldap.conf file (managed by Puppet) with setting on how to access the LDAP servers.
=== DNS ===
DNS is handled by PowerDNS. Private DNS entries (e.g. foo.eqiad1.wikimedia.cloud) are created via Designate Sink and stored in a PDNS server using a MySQL backend. Public DNS entries are created via Horizon and the Designate API.
Category:Overviews
Portal:Cloud VPS/Roadmap

Like all software, Cloud VPS is a work in progress. Here is an outline of near- and long-term improvements planned for the platform.
== Done, as of 2024-01-01 ==
* OpenStack APIs moved to TLS and opened up for public access
* Clarify best practices for security and privacy within cloud projects
* Move to project-local NFS servers
* Object storage (via radosgw and Swift APIs)
* S3 API support for object storage
* Basic terraform/opentofu support
* Basic kubernetes as a service support (via cli/opentofu only)
== Near future (AKA things we have already started on) ==
* Cross-site policy and enforcement for hosted services
* Improved Trove stability and support
** Improved postgres integration (requires upstream fixes)
** Snapshot and backup support (requires Swift)
** Support for larger volume sizes (requires upstream fixes)
* Support unmanaged VMs (https://phabricator.wikimedia.org/T326818)
** VMs that act more like standard OpenStack servers, for power users
** No puppet or cumin management
** No ldap integration
* Kubernetes as a service (openstack Magnum) improvements
== Possible future features (AKA things we would like to work on someday) ==
* Better secrets management
* Self-service backups
* IPv6
* Separate Per-tenant networking
* Load balancing as a service
Portal:Data Services/Admin/Dumps

The clouddumps servers serve dumps to clients running in Cloud VPS and the internet. See also Dumps/Dump_servers.
== Interfaces ==
; Cloud VPS NFS
: Some Cloud VPS instances mount NFS shares from all clouddumps servers. The dumps_dist_active_vps hiera variable controls the symlinks in /public/dumps which clients are meant to use.
; Production NFS
: Some wikiprod hosts (e.g. stat*) mount dumps as NFS shares. Again these mount both hosts, but confusingly the dumps_dist_active_web hiera variable is used to control the user-facing symlinks.
; Web
: Clients on the internet can download dumps via the web server, which is accessed via the dumps.wikimedia.org service name. The dumps_dist_active_web and profile::dumps::distribution::web::is_primary_server hiera variables need to match the main DNS entry.
; rsync
: Mirrors have rsync access to the dumps shares. They use the dumps.wikimedia.org service name.
== Failover ==
=== Web/rsync ===
To failover the web/rsync server to the other node you need to:
# Update the dumps.wikimedia.org CNAME to point to the other server
# Update Hiera data:
#* Update dumps_dist_active_web Hiera variable to point to the other server
#* Update profile::dumps::distribution::web::is_primary_server to false on the old server and to true on the new one
# Wait for DNS TTL to expire (which should be longer than a full Puppet cycle)
=== Cloud VPS NFS ===
To failover the Cloud VPS NFS server to the other node you need to:
# Update the dumps_dist_active_vps Hiera variable
# Wait for a full Puppet cycle or two for most VMs to pick it up
Category:Data Services admin
Portal:Data Services/Admin/Runbooks

This page contains basic resources for developers who want to author and publish runbooks for Data Services on Wikitech.
==Before you begin==
Permissions: Some processes documented in runbooks for Data Services will need varying levels of permission to perform. Make sure you have appropriate permissions to complete your task.
==What is a runbook?==
A runbook is a detailed set of instructions that explain how to perform a common task or procedure, so it can be easily and accurately repeated by others. Runbooks are particularly useful for incident response operations. Creating runbooks in response to specific incidents, makes it possible for people to repeat the steps in response to similar incidents. Because they are often used to help people respond to incidents quickly, they should be easy to read and follow, consistent, and accurate.
==When should you use a runbook?==
A runbook should be used whenever a common task or procedure may need to be repeated by mulitple people.
==Tips for building useful runbooks==
*Keep them up-to-date. Revisit the instructions after each incident to make sure they are clear and accurate.
* Keep them simple. These are sets of detailed instructions, and they do not require extra history or context.
** Publish a separate runbook for each issue or incident
*Test them. Make sure the instructions are repeatable by others by asking others to follow them and provide feedback.
* Follow a template. Make sure your runbooks follow a template, so that people can find and update information easily.
===Data Services runbook templates===
Instructions for each issue or alert should have its own page in order to reduce unnecessary information. You can see an example here with the issue Failover an NFS cluster. Note that the page addresses one issue only.
You can find a template/outline for creating runbooks here: Portal:Data_Services/Admin/Runbooks/Runbook_template
== Where to publish runbooks for Data Services ==
Where to publish Data Services runbooks
Portal:Data_Services/Admin/Runbooks/(PAGE NAME)
Note: This is a subpage of admin documentation. Some procedures may require advanced admin permissions to complete.
Note: Most runbooks for Data Services will include procedures that can only be followed by individuals with admin access. In order to avoid confusion and frustration for general users, you should note at the content level when a procedure will require admin permissions.
When an entire runbook requires admin permissions to complete procedures, mark it with the following template:
When runbooks include information for general users and special instructions for admins, mark any instructions (inline) for admins with the following template:
== Existing runbooks ==
==See Also==
*:Category:Runbooks - Wikitech pages marked as runbooks.
*WMCS Admin documentation - Advanced documenation for WMCS administrators.
*Runbooks entry on Wikipedia
Category:Data Services admin
Category:Documentation
 Data Services
Portal:Data Services/Admin/Runbooks/Deploy wiki replicas view change

This runbook details how to change the Wiki Replicas views.
== Review process ==
== Deploying ==
This section details how to actually deploy a change.
# Confirm with DBAs that they don't have work in progress that might interfere
# Merge associated Gerrit change
# Run the update-views cookbook:
you@cumin$ sudo cookbook sre.wikireplicas.update-views -t PHABRICATOR_TASK [--database DB_NAME] [--table TABLE_NAME]
# On each replica server:
## Starting from clouddb1021 is usually a good idea
## Run Puppet
## Check that it works fine on a single database:sudo maintain-views --table <affected table> --databases <a single database present on that host> --auto-depool
## If that's fine, do it on all the databases: sudo maintain-views --table <affected table> --all-databases --replace-all --auto-depool
== Support contacts ==
If you are following this, you are probably already a part of the WMCS or Data Platform SRE team. Perhaps you can ask the team you are not on if you need more help?
== Related information ==
* Portal:Data Services/Admin/Wiki Replicas
Category:Runbooks
Category:Wiki Replica admin
Portal:Data Services/Admin/Runbooks/Runbook template

This is a template for Data Services runbooks.
== Admin permissions header ==
If your runbook requires admin permissions to complete, include this header at the very top:
Note: if your runbook is intended for general users, and also has some steps that require admin permissions, you should mark those steps with the following template, so folks can easily identify what they are able to do: 
== Error / Incident ==
What?
Provide a description of the error or incident.
== Debugging ==
How?
Provide a solution, detailed list of steps a person will need to follow to fix the error.
== Support contacts ==
Who?
You may choose to list support contacts, individuals who have knowledge about this topc and who may be able to confirm the accuracy of the runbook or who may can help resolve issues related to the error or incident.
Note: Including the template {{:Help:Cloud Services communication}} may be sufficient.
== Related information ==
Where?
Provide links to related tools, tickets of previous incidents, and documentation that may provide context or additional information.
== What else to include ==
* Diagrams
== Examples ==
* Portal:Cloud_VPS/Admin/Runbooks/Cloud_VPS_alert_Puppet_failure_on
* Portal:Cloud_VPS/Admin/Runbooks/Check_for_VMs_leaked_by_the_nova-fullstack_test
Portal:Data Services/Admin/Wiki Replicas/DNS

The Wiki Replica servers use service names managed by the Cloud VPS Designate API to expose databases to Cloud Services users. The /usr/local/sbin/wmcs-wikireplica-dns script is used to create and update these service names. These operations are performed from the currently active cloudcontrol server (typically cloudcontrol1007.wikimedia.org).
=== Changing the backend server for a section ===
The /etc/wikireplica_dns.yaml config file tells wmcs-wikireplica-dns which IPs to map to which service names. This file is managed by Puppet using the source file found at modules/openstack/files/util/wikireplica_dns.yaml. Management is done at the granularity of a section.
To change the backend server for one or more sections, edit the yaml config file and then run:
$ ssh cloudcontrol1007.eqiad.wmnet
$ sudo -i
# export OS_CLOUD=novaadmin
# wmcs-wikireplica-dns
Calling the script with no arguments will simply refresh/create all the records from the config file.
=== Adding a new wikidb alias ===
Aliases for each wikidb in each zone are made based on the wikidb mappings found in the s[1-8].dblist files managed in operations/mediawiki-config.git. The current values are read from https://noc.wikimedia.org/conf/ at runtime.
When a new wikidb is added or moved between sections, update the related <wikidb>.{analytics,web}.db.svc.eqiad.wmflabs CNAME records. Limiting the run to a single section will be significantly faster unless the section is s3 which has most of the aliases.
$ ssh cloudcontrol1007.wikimedia.org
$ sudo -i
# export OS_CLOUD=novaadmin
:# note: a section is incorrectly called shard here:
# wmcs-wikireplica-dns --aliases [--shard <sN>]
== See also ==
* Portal:Data Services/Admin/Wiki Replicas (wiki-replica admin docs)
* Wiki Replicas (wiki-replica end user docs)
DNS