Swift/Ring Management

Category:Swift
Swift rings are managed by an automated process; you edit a YAML file that describes the hosts in the cluster, and then the swift_ring_manager process (run by a systemd timer) gradually adjusts the deployed rings to match the state specified in the YAML file. The aim is that the YAML file should be reasonably intuitive; some common tasks are outlined here.
The swift ring manager code is installed on one front-end per Swift cluster (see the ring_manager entry in swift_clusters in hiera); the [code repository] is checked out into /srv/deployment/swift_ring_manager, and the script /usr/local/bin/swift_ring_manager is templated by puppet to ensure the correct version of python is used. That repository contains a test suite, which contains [an extensive set of example changes].
The configuration file is deployed as /etc/swift/hosts.yaml by Puppet; so to change the state of the rings, edit the relevant YAML file (found in modules/swift/files/CLUSTERNAME_hosts.yaml), and put through a CR as usual.
== Storage Schemes ==
Every hosts.yaml file must define one or more schemes, which specify which devices belong to which rings, and with what weight. You typically won't have to edit these, as WMF tends to buy hardware of consistent specification. Each scheme is a mapping of ring name to the members of that ring, with an additional member "weight" which contains a mapping of ring name to the weight of members of that ring. For example:
 schemes:
 prod:
 objects: [sdc1, sdd1, sde1]
 accounts: &ap [sda3, sdb3]
 containers: *ap
 ssds: &sp [sda4, sdb4]
 weight:
 objects: 4000
 accounts: &acw 100
 containers: *acw
 ssds: 300
Here, there is a single scheme "prod". There are three devices (/dev/sdc1, /dev/sdd1, /dev/sde1) which hold objects, with weight 4000. In common with the previous Swift ring management code, the rings are given friendly names ("objects" for the "object" ring, "ssds" for the "object-1" ring, etc.); the mapping of friendly name to ring is defined in the regularize_ring_names function.
== Adding a host ==
If a host is the same type as other hosts in the cluster, simply add the host to the relevant list in hosts.yaml. So
 hosts:
 prod:
 - hostname
would become
 hosts:
 prod:
 - hostname
 - newhostname
To add newhostname, a new host with the prod device scheme. Remember to also add the new node to swift::storagehosts
== Removing a host ==
Simply removing the host entry from hosts.yaml will cause it to be immediately removed from the rings; it is generally better to drain it first (i.e. gradually remove weight from all of its devices). This can be done thus:
 hosts:
 prod:
 - hostname:
 - drain
Using failed instead of drain will cause all weights to be set to 0 immediately, rather than gradually.
== Removing a device ==
Do this by setting its weight to zero. By default, this will take effect gradually; if you want to do so straight away, specify immediate as well as weight zero. As a convenience, you can also use drain for "weight 0" and failed for "weight 0, immediate". For example:
 hosts:
 prod:
 - hostname:
 - sdc1: 0
 - sdd1: [0, immediate]
 - sde1: failed
Here /dev/sdc1 will have weight on it gradually reduced to 0, whereas /dev/sdd1 and /dev/sde1 will both have their weight set to 0 immediately.
== Adjusting a device weight ==
This is essentially the same as the previous operation, but you can specify any weight you like. You may also specify immediate for a change you want to not be made gradually. For example:
 hosts:
 prod:
 - hostname:
 - sdc1: 2000
 - sdd1: [3500, immediate]
Here /dev/sdc1 will have its weight gradually changed to 2000, and /dev/sdd1 will have its weight immediately set to 3500.
== Rebalancing the rings without making other changes ==
Sometimes after a lot of changes have completed you may notice that the rings are somewhat unbalanced ("balance" in swift-ring-builder output is high); you can rebalance the rings by running, on the ring manager host:
 sudo /usr/local/bin/swift_ring_manager -o /var/cache/swift_rings --doit --rebalance -v
It's only useful to do this when the rings are otherwise not being changed (as a change to the rings will involve a rebalance anyway).
== Invoking swift_ring_manager.py ==
Typically, you don't need to do this - it should be run automatically on the cluster's ring manager node, which will make changes and distribute the ring files for you. You can run it yourself to see what changes it would make, however - the default operation is to merely say what changes if any would be made. The help message details all the command-line arguments:
 usage: swift_ring_manager.py [-h] [--doit] [--immediate-only]
 [--update-tmp-rings] [-v] [-c CONF_FILE]
 [--ring-dir RING_DIR] [-o OUT_DIR]
 [--skip-dispersion-check]
 [--skip-replication-check] [-s]
 [--host-file-path HOST_FILE_PATH]
 [--generate-host-file]
 
 Swift Ring Manager
 
 optional arguments:
 -h, --help show this help message and exit
 --doit Update ring files and deploy
 --immediate-only Only make changes flagged as immediate
 --update-tmp-rings Only update temporary copies of ring files
 -v, --verbose emit more debugging information
 -c CONF_FILE, --conf-file CONF_FILE
 path to configuration file
 --ring-dir RING_DIR directory containing ring builder files
 -o OUT_DIR, --out-dir OUT_DIR
 directory to put new rings into
 --skip-dispersion-check
 don't check dispersion OK before making changes
 --skip-replication-check
 don't check replication OK before making changes
 -s, --syslog Log to syslog rather than stdout
 --host-file-path HOST_FILE_PATH
 fake hosts list file path
 --generate-host-file generate a hosts file for future testing use