Chapter 3. Managing high availability services with Pacemaker The Pacemaker service manages core container and active-passive services, such as Galera, RabbitMQ, Redis, and HAProxy. You use Pacemaker to view and manage general information about the managed services, virtual IP addresses, power management, and fencing. 3.1. Pacemaker resource bundles and containers Pacemaker manages Red Hat OpenStack Platform (RHOSP) services as Bundle Set resources, or bundles. Most of these services are active-active services that start in the same way and always run on each Controller node. Pacemaker manages the following resource types: Bundle A bundle resource configures and replicates the same container on all Controller nodes, maps the necessary storage paths to the container directories, and sets specific attributes related to the resource itself. Container A container can run different kinds of resources, from simple systemd services like HAProxy to complex services like Galera, which requires specific resource agents that control and set the state of the service on the different nodes. Important You cannot use podman or systemctl to manage bundles or containers. You can use the commands to check the status of the services, but you must use Pacemaker to perform actions on these services. Podman containers that Pacemaker controls have a RestartPolicy set to no by Podman. This is to ensure that Pacemaker, and not Podman, controls the container start and stop actions. 3.1.1. Simple Bundle Set resources (simple bundles) A simple Bundle Set resource, or simple bundle, is a set of containers that each include the same Pacemaker services that you want to deploy across the Controller nodes. The following example shows a list of simple bundles from the output of the pcs status command: For each bundle, you can see the following details: The name that Pacemaker assigns to the service. The reference to the container that is associated with the bundle. The list and status of replicas that are running on the different Controller nodes. The following example shows the settings for the haproxy-bundle simple bundle: The example shows the following information about the containers in the bundle: image : Image used by the container, which refers to the local registry of the undercloud. network : Container network type, which is "host" in the example. options : Specific options for the container. replicas : Indicates how many copies of the container must run in the cluster. Each bundle includes three containers, one for each Controller node. run-command : System command used to spawn the container. Storage Mapping : Mapping of the local path on each host to the container. The haproxy configuration is located in the /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg file instead of the /etc/haproxy/haproxy.cfg file. Note Although HAProxy provides high availability services by load balancing traffic to selected services, you configure HAProxy as a highly available service by managing it as a Pacemaker bundle service. 3.1.2. Complex Bundle Set resources (complex bundles) Complex Bundle Set resources, or complex bundles, are Pacemaker services that specify a resource configuration in addition to the basic container configuration that is included in simple bundles. This configuration is needed to manage multi-state resources, which are services that can have different states depending on the Controller node they run on. This example shows a list of complex bundles from the output of the pcs status command: This output shows the following information about each complex bundle: RabbitMQ: All three Controller nodes run a standalone instance of the service, similar to a simple bundle. Galera: All three Controller nodes are running as Galera masters under the same constraints. Redis: The overcloud-controller-0 container is running as the master, while the other two Controller nodes are running as slaves. Each container type might run under different constraints. The following example shows the settings for the galera-bundle complex bundle: This output shows that, unlike in a simple bundle, the galera-bundle resource includes explicit resource configuration that determines all aspects of the multi-state resource. Note Although a service can run on multiple Controller nodes at the same time, the Controller node itself might not be listening at the IP address that is required to reach those services. For information about how to check the IP address of a service, see Section 3.4, "Viewing resource information for virtual IPs in a high availability cluster" . 3.2. Checking Pacemaker cluster status You can check the status of the Pacemaker cluster in any node where Pacemaker is running, and view information about the number of resources that are active and running. Prerequisites High availability is deployed and running. Procedure Log in to any Controller node as the heat-admin user. Run the pcs status command: Example output: The main sections of the output show the following information about the cluster: Cluster name : Name of the cluster. [NUM] nodes configured : Number of nodes that are configured for the cluster. [NUM] resources configured : Number of resources that are configured for the cluster. Online : Names of the Controller nodes that are currently online. GuestOnline : Names of the guest nodes that are currently online. Each guest node consists of a complex Bundle Set resource. For more information about bundle sets, see Section 3.1, "Pacemaker resource bundles and containers" . 3.3. Checking bundle status in a high availability cluster You can check the status of a bundle from an undercloud node or log in to one of the Controller nodes to check the bundle status directly. Prerequisites High availability is deployed and running. Procedure Use one of the following options: Log in to an undercloud node and check the bundle status, in this example haproxy-bundle : Example output: The output shows that the haproxy process is running inside the container. Log in to a Controller node and check the bundle status, in this example haproxy : Example output: 3.4. Viewing resource information for virtual IPs in a high availability cluster To check the status of all virtual IPs (VIPs) or a specific VIP, run the pcs resource show command with the relevant options. Each IPaddr2 resource sets a virtual IP address that clients use to request access to a service. If the Controller node with that IP address fails, the IPaddr2 resource reassigns the IP address to a different Controller node. Prerequisites High availability is deployed and running. Procedure Log in to any Controller node as the heat-admin user. Use one of the following options: Show all resources that use virtual IPs by running the pcs resource show command with the --full option: Example output: Each IP address is initially attached to a specific Controller node. For example, 192.168.1.150 is started on overcloud-controller-0 . However, if that Controller node fails, the IP address is reassigned to other Controller nodes in the cluster. The following table describes the IP addresses in the example output and shows the original allocation of each IP address. Table 3.1. IP address description and allocation source IP Address Description Allocated From 10.200.0.6 Controller virtual IP address Part of the dhcp_start and dhcp_end range set to 10.200.0.5-10.200.0.24 in the undercloud.conf file 192.168.1.150 Public IP address ExternalAllocationPools attribute in the network-environment.yaml file 172.16.0.10 Provides access to OpenStack API services on a Controller node InternalApiAllocationPools in the network-environment.yaml file 172.16.0.11 Provides access to Redis service on a Controller node InternalApiAllocationPools in the network-environment.yaml file 172.18.0.10 Storage virtual IP address that provides access to the Glance API and to Swift Proxy services StorageAllocationPools attribute in the network-environment.yaml file 172.19.0.10 Provides access to storage management StorageMgmtAlloctionPools in the network-environment.yaml file View a specific VIP address by running the pcs resource show command with the name of the resource that uses that VIP, in this example ip-192.168.1.150 : Example output: 3.5. Viewing network information for virtual IPs in a high availability cluster You can view the network interface information for a Controller node that is assigned to a specific virtual IP (VIP), and view port number assignments for a specific service. Prerequisites High availability is deployed and running. Procedure Log in to the Controller node that is assigned to the IP address you want to view and run the ip addr show command on the network interface, in this example vlan100 : Example output: Run the netstat command to show all processes that listen to the IP address, in this example 192.168.1.150.haproxy : Example output: Note Processes that are listening to all local addresses, such as 0.0.0.0 , are also available through 192.168.1.150 . These processes include sshd , mysqld , dhclient , ntpd . View the default port number assignments and the services they listen to by opening the configuration file for the HA service, in this example /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg : TCP port 6080: nova_novncproxy TCP port 9696: neutron TCP port 8000: heat_cfn TCP port 80: horizon TCP port 8776: cinder In this example, most services that are defined in the haproxy.cfg file listen to the 192.168.1.150 IP address on all three Controller nodes. However, only the controller-0 node is listening externally to the 192.168.1.150 IP address. Therefore, if the controller-0 node fails, HAProxy only needs to re-assign 192.168.1.150 to another Controller node and all other services will already be running on the fallback Controller node. 3.6. Checking fencing agent and Pacemaker daemon status You can check the status of the fencing agent and the status of the Pacemaker daemons in any node where Pacemaker is running, and view information about the number of Controller nodes that are active and running. Prerequisites High availability is deployed and running. Procedure Log in to any Controller node as the heat-admin user. Run the pcs status command: Example output: The output shows the following sections of the pcs status command output: my-ipmilan-for-controller : Shows the type of fencing for each Controller node ( stonith:fence_ipmilan ) and whether or not the IPMI service is stopped or running. PCSD Status : Shows that all three Controller nodes are currently online. Daemon Status : Shoes the status of the three Pacemaker daemons: corosync , pacemaker , and pcsd . In the example, all three services are active and enabled. 3.7. Additional resources Configuring and Managing High Availability Clusters

Commands:
Podman container set: haproxy-bundle [192.168.24.1:8787/rhosp-rhel8/openstack-haproxy:pcmklatest] haproxy-bundle-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 haproxy-bundle-podman-1 (ocf::heartbeat:podman): Started overcloud-controller-1 haproxy-bundle-podman-2 (ocf::heartbeat:podman): Started overcloud-controller-2
sudo pcs resource show haproxy-bundle Bundle: haproxy-bundle Podman: image=192.168.24.1:8787/rhosp-rhel8/openstack-haproxy:pcmklatest network=host options="--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" replicas=3 run-command="/bin/bash /usr/local/bin/kolla_start" Storage Mapping: options=ro source-dir=/var/lib/kolla/config_files/haproxy.json target-dir=/var/lib/kolla/config_files/config.json (haproxy-cfg-files) options=ro source-dir=/var/lib/config-data/puppet-generated/haproxy/ target-dir=/var/lib/kolla/config_files/src (haproxy-cfg-data) options=ro source-dir=/etc/hosts target-dir=/etc/hosts (haproxy-hosts) options=ro source-dir=/etc/localtime target-dir=/etc/localtime (haproxy-localtime) options=ro source-dir=/etc/pki/ca-trust/extracted target-dir=/etc/pki/ca-trust/extracted (haproxy-pki-extracted) options=ro source-dir=/etc/pki/tls/certs/ca-bundle.crt target-dir=/etc/pki/tls/certs/ca-bundle.crt (haproxy-pki-ca-bundle-crt) options=ro source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt (haproxy-pki-ca-bundle-trust-crt) options=ro source-dir=/etc/pki/tls/cert.pem target-dir=/etc/pki/tls/cert.pem (haproxy-pki-cert) options=rw source-dir=/dev/log target-dir=/dev/log (haproxy-dev-log)
Podman container set: rabbitmq-bundle [192.168.24.1:8787/rhosp-rhel8/openstack-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2 Podman container set: galera-bundle [192.168.24.1:8787/rhosp-rhel8/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-controller-1 galera-bundle-2 (ocf::heartbeat:galera): Master overcloud-controller-2 Podman container set: redis-bundle [192.168.24.1:8787/rhosp-rhel8/openstack-redis:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave overcloud-controller-1 redis-bundle-2 (ocf::heartbeat:redis): Slave overcloud-controller-2
[...] Bundle: galera-bundle Podman: image=192.168.24.1:8787/rhosp-rhel8/openstack-mariadb:pcmklatest masters=3 network=host options="--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" replicas=3 run-command="/bin/bash /usr/local/bin/kolla_start" Network: control-port=3123 Storage Mapping: options=ro source-dir=/var/lib/kolla/config_files/mysql.json target-dir=/var/lib/kolla/config_files/config.json (mysql-cfg-files) options=ro source-dir=/var/lib/config-data/puppet-generated/mysql/ target-dir=/var/lib/kolla/config_files/src (mysql-cfg-data) options=ro source-dir=/etc/hosts target-dir=/etc/hosts (mysql-hosts) options=ro source-dir=/etc/localtime target-dir=/etc/localtime (mysql-localtime) options=rw source-dir=/var/lib/mysql target-dir=/var/lib/mysql (mysql-lib) options=rw source-dir=/var/log/mariadb target-dir=/var/log/mariadb (mysql-log-mariadb) options=rw source-dir=/dev/log target-dir=/dev/log (mysql-dev-log) Resource: galera (class=ocf provider=heartbeat type=galera) Attributes: additional_parameters=--open-files-limit=16384 cluster_host_map=overcloud-controller-0:overcloud-controller-0.internalapi.localdomain;overcloud-controller-1:overcloud-controller-1.internalapi.localdomain;overcloud-controller-2:overcloud-controller-2.internalapi.localdomain enable_creation=true wsrep_cluster_address=gcomm://overcloud-controller-0.internalapi.localdomain,overcloud-controller-1.internalapi.localdomain,overcloud-controller-2.internalapi.localdomain Meta Attrs: container-attribute-target=host master-max=3 ordered=true Operations: demote interval=0s timeout=120 (galera-demote-interval-0s) monitor interval=20 timeout=30 (galera-monitor-interval-20) monitor interval=10 role=Master timeout=30 (galera-monitor-interval-10) monitor interval=30 role=Slave timeout=30 (galera-monitor-interval-30) promote interval=0s on-fail=block timeout=300s (galera-promote-interval-0s) start interval=0s timeout=120 (galera-start-interval-0s) stop interval=0s timeout=120 (galera-stop-interval-0s) [...]
ssh heat-admin@overcloud-controller-0
[heat-admin@overcloud-controller-0 ~] USD sudo pcs status
Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-1 (version 2.0.1-4.el8-0eb7991564) - partition with quorum Last updated: Thu Feb 8 14:29:21 2018 Last change: Sat Feb 3 11:37:17 2018 by root via cibadmin on overcloud-controller-2 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ] Full list of resources: [...]
sudo podman exec -it haproxy-bundle-podman-0 ps -efww | grep haproxy*
root 7 1 0 06:08 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws haproxy 11 7 0 06:08 ? 00:00:17 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws
ps -ef | grep haproxy*
root 17774 17729 0 06:08 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws 42454 17819 17774 0 06:08 ? 00:00:21 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws root 288508 237714 0 07:04 pts/0 00:00:00 grep --color=auto haproxy* ps -ef | grep -e 17774 -e 17819 root 17774 17729 0 06:08 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws 42454 17819 17774 0 06:08 ? 00:00:22 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws root 301950 237714 0 07:07 pts/0 00:00:00 grep --color=auto -e 17774 -e 17819
ssh heat-admin@overcloud-controller-0
sudo pcs resource show --full
ip-10.200.0.6 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.1.150 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-172.16.0.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-172.16.0.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-172.18.0.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-172.19.0.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
sudo pcs resource show ip-192.168.1.150
Resource: ip-192.168.1.150 (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=192.168.1.150 cidr_netmask=32 Operations: start interval=0s timeout=20s (ip-192.168.1.150-start-timeout-20s) stop interval=0s timeout=20s (ip-192.168.1.150-stop-timeout-20s) monitor interval=10s timeout=20s (ip-192.168.1.150-monitor-interval-10s)
ip addr show vlan100
9: vlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether be:ab:aa:37:34:e7 brd ff:ff:ff:ff:ff:ff inet *192.168.1.151/24* brd 192.168.1.255 scope global vlan100 valid_lft forever preferred_lft forever inet *192.168.1.150/32* brd 192.168.1.255 scope global vlan100 valid_lft forever preferred_lft forever
sudo netstat -tupln | grep "192.168.1.150.haproxy"
tcp 0 0 192.168.1.150:8778 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8042 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:9292 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8080 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:80 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8977 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:6080 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:9696 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8000 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8004 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8774 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:5000 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8776 0.0.0.0:* LISTEN 61029/haproxy tcp 0 0 192.168.1.150:8041 0.0.0.0:* LISTEN 61029/haproxy
ssh heat-admin@overcloud-controller-0
[heat-admin@overcloud-controller-0 ~] USD sudo pcs status
my-ipmilan-for-controller-0 (stonith:fence_ipmilan): Started my-ipmilan-for-controller-0 my-ipmilan-for-controller-1 (stonith:fence_ipmilan): Started my-ipmilan-for-controller-1 my-ipmilan-for-controller-2 (stonith:fence_ipmilan): Started my-ipmilan-for-controller-2 PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 pcsd: active/enabled