Chapter 2. RHOSO MariaDB Galera clusters Red Hat OpenStack Services on OpenShift (RHOSO) deploys the following two MariaDB Galera clusters: openstack , the cluster hosting the databases for all RHOSO services. openstack-cell1 , the cluster hosting the databases specific to the Compute service (nova) for cell1 . Galera Custom Resources (CRs) configure both clusters. You can use the oc get galera command to retrieve more information about these Galera CRs, as shown in the following example: The Message and Ready columns show the startup state and the service availability of the Galera CR, specified by the Name column. When the Ready condition is True , the pods are started and ready to accept traffic. The mariadb-operator performs the following Galera cluster operations: Creates the pods that host the mysqld servers. Runs the logic for bootstrapping a Galera cluster. For example, the mariadb-operator starts the cluster using the most recent copy of the Galera database. Monitors the running Galera pods. Restarts the pods when the pods fail their health check. The mariadb-operator creates an OpenShift service object called openstack , which provides the IP address that the OpenStack services use to access the database. You can use the following command to reveal this IP address: The mariadb-operator also creates a 'headless' DNS service for the Galera pods, called openstack-galera that is only used for the internal Galera cluster communication. OpenStack pods never access this service. RHOSO marks a Galera pod as available based on its Readiness health check. When the Galera cluster is bootstrapped by the mariadb-operator , the first available Galera pod receives all the incoming database traffic. The other available Galera pods remain in a state of hot standby. When the active pod is disconnected from the Galera cluster, for any reason, such as pod stop or pod restart, failover occurs. At this time, the first available Galera pod in hot standby becomes the active pod, and the openstack service is updated accordingly. You can use the following commands to troubleshoot the Galera services: When you analyze the openstack service, IP field specifies the IP address assigned to the service and Endpoints field specifies the IP address of the currently active Galera pod that is receiving all the traffic. When you analyze the headless openstack-galera service, the IP field is not specified but the Endpoints field specify the IP addresses of all the Galera pods, of which there are typically 3. 2.1. Monitoring Galera startup You can use the following command to monitor the startup of the Galera pods for the required Galera Custom Resource (CR): Replace <cr> with the required Galera CR, which by default, is either openstack or openstack-cell1 . When you analyze the Galera CR: The Status reports the startup status of the Galera cluster. The Conditions report the status of the prerequisites that the Galera pods need to start. Note The Ready condition is true only when all the other conditions are True . When the mariadb-operator bootstraps a Galera cluster, it gathers information from every database replica, and then stores it in transient attributes. These transient Attributes appear in the Status of the Galera CR when the cluster is inspected while the Galera cluster is stopped and being restarted: Before starting a Galera Cluster, the mariadb-operator starts all the Galera pod replicas in a waiting state. Even if you can see the pods by using the oc get pods command, they have not started mysqld servers yet. The mariadb-operator introspects the content of the database copy of each pod to extract the Seqno , the database sequence number. After the mariadb-operator extracts the Seqno from all of the pods, it decides which pod holds the most recent version of the database and bootstraps a new Galera cluster from this pod. This pod starts a mysqld server and a transient attribute gcomm:// appears in the status of the Galera CR. When the first mysqld server is ready to serve traffic, the attribute Bootstrapped becomes true , and the transient Attributes for this pod are removed from the status of the Galera CR.

Commands:
oc get galera NAME READY MESSAGE openstack True Setup complete openstack-cell1 True Setup complete
oc get service -l mariadb/name=openstack NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE openstack ClusterIP 10.217.5.210 <none> 3306/TCP 7h
oc describe service openstack Name: openstack Namespace: openstack Labels: app=mariadb cr=mariadb-openstack mariadb/name=openstack mariadb/namespace=openstack mariadb/uid=796b4c64-fb39-4144-817f-34d2b309eb30 owner=mariadb-operator Annotations: <none> Selector: app=galera,cr=galera-openstack,statefulset.kubernetes.io/pod-name=openstack-galera-2 Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 172.30.133.220 IPs: 172.30.133.220 Port: database 3306/TCP TargetPort: 3306/TCP Endpoints: 192.168.56.28:3306 Session Affinity: None Events: <none>
oc describe service openstack-galera Name: openstack-galera Namespace: openstack Labels: <none> Annotations: <none> Selector: app=galera,cr=galera-openstack Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: None IPs: None Port: mysql 3306/TCP TargetPort: 3306/TCP Endpoints: 192.168.48.52:3306,192.168.52.43:3306,192.168.56.23:3306 Session Affinity: None Events: <none>
oc describe galera <cr>
Status: Conditions: Last Transition Time: 2024-04-22T07:32:06Z Message: Setup complete Reason: Ready Status: True Type: Ready Last Transition Time: 2024-04-22T07:31:49Z Message: Deployment completed Reason: Ready Status: True Type: DeploymentReady Last Transition Time: 2024-04-22T07:31:11Z Message: Exposing service completed Reason: Ready Status: True Type: ExposeServiceReady Last Transition Time: 2024-04-22T07:31:11Z Message: Input data complete Reason: Ready Status: True Type: InputReady Last Transition Time: 2024-04-22T07:31:11Z Message: RoleBinding created Reason: Ready Status: True Type: RoleBindingReady Last Transition Time: 2024-04-22T07:31:11Z Message: Role created Reason: Ready Status: True Type: RoleReady Last Transition Time: 2024-04-22T07:31:11Z Message: ServiceAccount created Reason: Ready Status: True Type: ServiceAccountReady Last Transition Time: 2024-04-22T07:31:11Z Message: Service config create completed Reason: Ready Status: True Type: ServiceConfigReady Last Transition Time: 2024-04-22T07:31:11Z Message: Input data complete Reason: Ready Status: True Type: TLSInputReady
Status: Attributes: openstack-galera-0: Seqno: 1232 openstack-galera-1: Container ID: cri-o://f56ec2389e878b462a54f5255dad83db29daf4d8e8cda338904bfd353b370165 Gcomm: gcomm:// Seqno: 1232 openstack-galera-2: Seqno: 1231 Bootstrapped: false