Chapter 5. Investigating the pods of the RHOSO High Availability services The operator of each Red Hat OpenStack Services on OpenShift (RHOSO) High Availability service monitors the status of the pods that they manage. If necessary, an operator will take an appropriate action with the aim of keeping at least one replica of the service with a status of Running . You can use the following command to check the status and availability of all the pods of the Galera, RabbitMQ, and memcached shared control plane services: You can use the following command to investigate a specific pod from this list, typically to determine why a pod cannot be started: Replace <pod-name> with the name of the pod from the list of pods that you want more information about. In the following example the rabbitmq-server-0 pod is being investigated: 5.1. Understanding the Taint/Toleration based pod eviction process Red Hat OpenShift Container Platform (RHOCP) implements a Taint/Toleration based pod eviction process, which determines how individual pods are evicted from worker nodes. Pods that are evicted are rescheduled on different nodes. RHOCP assigns taints to specific node conditions, such as not-ready and unreachable . When a node experiences one of these conditions, RHOCP automatically taints the node. After a node has become tainted, the pods must determine if they can tolerate this taint or not: Any pod that does not tolerate the taint is evicted immediately. Any pod that does tolerate the taint will never be evicted, unless the pod has a limited toleration to this taint: The tolerationSeconds parameter specifies the limited toleration of a pod, which is how long a pod can tolerate a specific taint (node condition) and remain connected to the node. If the node condition still exists after the specified tolerationSeconds period, the taint remains on the node and the pod is evicted. If the node condition clears before the specified tolerationSeconds period, then the pod is not evicted. RHOCP adds a default toleration for the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints of five minutes, by setting tolerationSeconds=300 . Important RHOSO 18.0 operators do not modify the default tolerations for taints, therefore pods that run on a tainted worker node take more than five minutes to be rescheduled. For more information about RHOCP node remediation, fencing, and maintenance, see Workload Availability remediation, fencing, and maintenance .

Commands:
get pods |egrep -e "galera|rabbit|memcache" NAME READY STATUS RESTARTS AGE memcached-0 1/1 Running 0 3h11m memcached-1 1/1 Running 0 3h11m memcached-2 1/1 Running 0 3h11m openstack-cell1-galera-0 1/1 Running 0 3h11m openstack-cell1-galera-1 1/1 Running 0 3h11m openstack-cell1-galera-2 1/1 Running 0 3h11m openstack-galera-0 1/1 Running 0 3h11m openstack-galera-1 1/1 Running 0 3h11m openstack-galera-2 1/1 Running 0 3h11m rabbitmq-cell1-server-0 1/1 Running 0 3h11m rabbitmq-cell1-server-1 1/1 Running 0 3h11m rabbitmq-cell1-server-2 1/1 Running 0 3h11m rabbitmq-server-0 1/1 Running 0 3h11m rabbitmq-server-1 1/1 Running 0 3h11m rabbitmq-server-2 1/1 Running 0 3h11m
oc describe pod/<pod-name>
oc describe pod/rabbitmq-server-0 Name: rabbitmq-server-0 Namespace: openstack Priority: 0 Service Account: rabbitmq-server Node: master-2/192.168.111.22 Start Time: Thu, 21 Mar 2024 08:39:57 -0400 Labels: app.kubernetes.io/component=rabbitmq app.kubernetes.io/name=rabbitmq app.kubernetes.io/part-of=rabbitmq controller-revision-hash=rabbitmq-server-5c886b79b4 statefulset.kubernetes.io/pod-name=rabbitmq-server-0 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["192.168.16.35/22"],"mac_address":"0a:58:c0:a8:10:23","gateway_ips":["192.168.16.1"],"routes":[{"dest":"192.16 k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "192.168.16.35" ], "mac": "0a:58:c0:a8:10:23", "default": true, "dns": {} }] openshift.io/scc: restricted-v2 seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Running