Wikimedia Cloud Services team/EnhancementProposals/ceph client refactor

This page contains a puppet refactor proposal for the ceph client code.
== Problem statement ==
The ceph rbd client code in puppet does not have a easy and single way to use it, some points:
* There's 3 ways of using it so far, sometimes it splits by host that uses it (ceph::client::rbd_cloudcontrol) sometimes by service (ceph::client::rbd_glance/rbd_libvirt) and for cinder it's in the main class instead (p:openstack::codfw1dev::cinder).
* There is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
** nova VM disks (often referred to as 'compute' in the puppet tree)
** glance images
** cinder volumes
** radosgw
* Setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.
== Proposed solution ==
In summary: we should refresh the code to support the mappings:
<openstack deployment>+<sevice> <-> <ceph deployment>+<access type>
Where:
* openstack deployment is one of eqiad1 or codfw1dev for now
* service is one of glance, cinder, nova, cinder-backups, ...
* ceph deployment is currently "ceph", though this should be extended in the future to allow multi-dc/multi-cluster deployments (task)
* access type this would be an abstraction over actual ceph auth permissions. Something like glance_client, nova_client, radosgw_client, osd_server, mon_server, ... this will then realize to the correct auth caps statements (ex. [osd] profile rbd pool=equiad1-compute).
=== abstraction ===
Inspired by how acme-chief manages certificates:
* the source of truth is a private hiera hash with this layout, what it is called a DC-specific configuration hash:
profile::ceph::auth::load_all::configuration:
 cinder:
 keydata: XXXXXX
 keyring_filename: /path/to/file.keyring
 caps:
 mds: "whatever"
 mgr: "whatever"
 mon: "whatever"
 osd: "whatever"
 glance:
 keydata: XXXXXX
 keyring_filename: /path/to/file.keyring
 caps:
 mds: "whatever"
 mgr: "whatever"
 mon: "whatever"
 osd: "whatever"
 osd:
 keydata: XXXXXX
 keyring_filename: /path/to/file.keyring
 caps:
 mds: "whatever"
 mgr: "whatever"
 mon: "whatever"
 osd: "whatever"
* the hiera hash is per-DC (i.e, eqiad, codfw). Future iterations may evolve this to support multiple ceph farms per DC, but we don't want this at this point
* clients may declare which creds they want by using an additional hiera key, for example, on file hieradata/roles/codfw/wmcs/openstack/codfw1dev/control.yaml:
profile::ceph::auth::deploy::selected_creds:
 - cinder
 - glance
Once this hiera structures are in place, we have 2 important profiles/modules to handle that data:
* profile::ceph::auth::load_all: profile loaded on mons, reads the hiera hash (has a parameter 'configuration'), and calls internally the inner module:
** ceph::auth::load_all receives a list of credentials and loads them to the ceph running system.
* profile::ceph::auth::deploy: profile loaded on ceph clients, like osd, cloudcontrol and friends. Reads configuration hash (has a parameter 'auths'), and calls internally the inner module:
** ceph::auth::deploy: deploys a keyring file on the filesystem.
=== code organization proposal ===
With the abstraction described above, we could use the puppet code like this:
* role::wmcs::ceph::mon
** profile::ceph::auth::load_all <-- reads DC-specific configuration hiera hash (i.e, eqiad or codfw)
* role::wmcs::ceph::osd
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files
* role::wmcs::openstack::codfw1dev::control
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files
A code example of this:
class profile::ceph::auth::load_all (
 Hash $configuration = lookup('ceph::auth'),
) {
 class { 'ceph::auth::load_all':
 configuration => $configuration,
 }
}
class ceph::auth::load_all (
 Hash $configuration,
) {
 # for each entry in the configuration hash
 # load it using 'ceph auth get-or-create'
}
Also:
class profile::ceph::auth::deploy (
 Hash $configuration = lookup('ceph::auth'),
 Array[String] $selected_creds = lookup('profile::ceph::auth::deploy::selected_creds'),
) {
 class { 'ceph::auth::deploy':
 configuration = $configuration,
 selected_creds = $selected_creds,
 }
}
class ceph::auth::deploy(
 Hash $configuration,
 Array[String] $selected_creds, 
) {
 # filter configuration for the selected creds
 # if we find a match, then create a keyring file resource with the keydata
}
Example hiera configuration:
File on private repository hieradata/common/codfw.yaml (which should be available to all systems on the DC --- TODO: we may want to have a more suitable place for this)
ceph::auth:
 cinder:
 keydata: XXXXXX
 keyring_filename: /path/to/file.keyring
 caps:
 caps_mds: "whatever"
 caps_mgr: "whatever"
 caps_mon: "whatever"
 caps_osd: "whatever"
 glance:
 keydata: XXXXXX
 keyring_filename: /path/to/file.keyring
 caps:
 caps_mds: "whatever"
 caps_mgr: "whatever"
 caps_mon: "whatever"
 caps_osd: "whatever"
 osd:
 keydata: XXXXXX
 keyring_filename: /path/to/file.keyring
 caps:
 caps_mds: "whatever"
 caps_mgr: "whatever"
 caps_mon: "whatever"
 caps_osd: "whatever"
File on the public repo hieradata/role/codfw/wmcs/openstack/codfw1dev/control.yaml, which should be available to all openstack codfw1dev cloudcontrol servers:
profile::ceph::auth::deploy:
 - cinder
 - glance
File on the public repo hieradata/role/codfw/wmcs/openstack/codfw1dev/virt_ceph.yaml, which should be available to all openstack codfw1dev virt servers:
profile::ceph::auth::deploy:
 - compute
== See also ==
* T292546: cloud NFS: figure out backups for cinder volumes
* T281250: ceph.operationalization: Refactor the current puppet code to allow per-cluster configurations (as opposed to per-DC as it does currently)
Potential inspiration:
* https://github.com/datacentred/puppet-ceph