Chapter 1. Collecting diagnostic information for Support Use the Red Hat OpenStack Services on OpenShift (RHOSO) must-gather tool to collect diagnostic information about your Red Hat OpenShift Container Platform (RHOCP) cluster, including the RHOSO control plane and the deployed RHOSO services. Use the RHOCP sosreport tool to collect diagnostic information about your RHOSO data plane. 1.1. Collecting data from the control plane or data plane You can use the Red Hat OpenStack Services on OpenShift (RHOSO) must-gather tool to create a local directory to store logs, configurations, and status of the RHOSO control plane or data plane services. You can use the must-gather tool to collect diagnostic information about your RHOSO deployment: Service logs retrieved by the output of the pods and operators associated with the deployed services. The configuration of RHOSO services, such as the Red Hat OpenShift Container Platform (RHOCP) Secrets and ConfigMaps . The status of the RHOSO services, cuch as the services that are deployed in the RHOSO control plane. The RHOSO Custom Resource Definitions (CRDs). The RHOSO applied Custom Resources (CRs). The openstack and openstack-operators namespaces. RHOCP Events that are related to the RHOSO namespaces. CSVs, pkgmanifests , subscriptions , installplans , operatorgroup . Pods, Deployments, Statefulsets , ReplicaSets , Service, Routes, ConfigMaps , relevant Secrets. Network information, such as IPAddressPool , L2Advertisements , NetConfig , IPSet . SOS reports for RHOCP nodes that are running RHOSO service pods. Prerequisites You can access the cluster as a user with cluster-admin privileges. You can access the registry directly to invoke podman commands, to allow RHOCP to pull images from the registry and run them with the oc adm command: Procedure Navigate to the directory where you want to store the must-gather data. Pass one or more images or image streams to the must-gather tool to specify the data to collect. For example, the following command gathers both the default cluster data and the information that is specific to the deployed RHOSO control plane: 1 The default RHOCP must-gather image that is used to gather RHOCP cluster information. 2 The RHOSO must-gather image. This command creates a local directory that stores the logs, services configuration, and the status of the RHOSO control plane services. steps You can provide SOS reports to Red Hat Support to help diagnose and troubleshoot issues in your deployment. For information on how to use the SOS report tool, see Getting the most from your Support experience . 1.2. Customizing the gathered data You can use environmental variables to configure Red Hat OpenStack Services on OpenShift (RHOSO) must-gather collectors. For example, you can pass an empty SOS_SERVICES environmental variable to disable SOS gathering. Procedure To provide environmental variables, invoke the gathering command manually: The following is a list of available environmental variables: OSP_NS: Namespace where the RHOSO services are running. The default value is openstack . OSP_OPERATORS_NS: Namespace where the RHOSO operators are running. The default value is openstack-operators . CONCURRENCY: Must-gather can run many operations, so for efficiency the operations run in parallel with a concurrency of 5 by default. SOS_SERVICES: Comma-separated list of services to gather the SOS reports from. You can set a value of an empty string to skip sos report gathering for that service. SOS_ONLY_PLUGINS: List of SOS report plugins to use. You can set a value of an empty string to run all of the reports. The default value is: block,cifs,crio,devicemapper,devices,iscsi,lvm2, memory,multipath,nfs,nis,nvme,podman,process,processor,selinux,scsi,udev . SOS_EDPM: Comma-separated list of edpm nodes to gather SOS reports from. You can set a value of an empty string to skip sos report gathering, or use the keyword all to gather from all the nodes, such as: edpm-compute-0 , edpm-compute-1 . SOS_EDPM_PROFILES: List of sos report profiles to use. You can set a value of an empty string to run all of the reports. The default value is: container,openstack_edpm,system,storage,virt . SOS_EDPM_PLUGINS: Optional list of sos report plugins to use. OPENSTACK_DATABASES: Comma-separated list of RHOSO databases that you want to dump. You can set the value to the keyword all to dump all databases. The default value is an empty string and the database dump is skipped. ADDITIONAL_NAMESPACES: Comma-separated list of additional namespaces where you want to gather the associated resources. COMPRESSED_PATH: Defines the path to store the compressed form of the gathered data. DELETE_AFTER_COMPRESSION: The default value is 0 . If you set the value to 1 , the uncompressed data is deleted after the archive is created. 1.3. Inspect the gathered data You can use the Red Hat OpenStack Services on OpenShift (RHOSO) must-gather tool to get the kubernetes resources defined in the collection-scripts , and the sos-reports associated with both the CoreOS nodes and the EDPM ones. When the must-gather execution ends, it creates a directory containing all the gathered resources, such as: Global resources: You can use these to get some context about the status of the Red Hat OpenShift Container Platform (RHOCP) cluster and the RHOSO deployed resources. These resources include crds , apiservices , csvs , packagemanifests , webhooks , and network information such as nncp , nnce , IPAddressPool , and more. Namespaced resources: You need these to get the status of the RHOSO cluster and to troubleshoot any problems. sos-reports: You gather these from both the CoreOS nodes that are part of the RHOCP cluster, and the EDPM nodes that are part of the cluster. The information to connect to the EDPM nodes is retrieved by the OpenStackDataplaneNodeSets CR, and the resulting sos-report is retrieved from the remote nodes and then downloaded in the current must-gather directory. OpenStack Ctlplane Services: You can run command through the openstack-cli to check the relevant resources generated within the RHOSO cluster, such as endpoint list, networks, subnets, registered services, and so on. The following is an example of must-gather output: When you are troubleshooting, it is critical to check and analyze not only Secrets and services config files, but also the CRs associated with each service and the Pod logs. These are namespaced resources that you can find in the CRs and Pods directories. Additionally, some generic information for each namespace is collected. You can use the must-gather tool to retrieve: Events recorded for the current namespace. Network Attachment Definitions. PVCs attached to the deployed Pods. all_resources.log , which is an outline of the namespaces in terms of deployed resources. As depicted in the schema, the same pattern applies to the Pod resources. You can use the must-gather tool to retrieve the description and the associated logs for each Pod, including - if the Pod is in a CrashLookBackoff status.

Commands:
podman login registry.redhat.io
oc adm must-gather --image-stream=openshift/must-gather \ 1 --image=registry.redhat.io/rhoso-operators/openstack-must-gather-rhel9:1.0 2
oc adm must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather -- SOS_SERVICES= gather
+-----------------------------------+ | . | +-----------------------------+ | ├── apiservices | | ctlplane/neutron/ | | ├── crd | | ├── agent_list | | ├── csv | (control plane resources) | ├── extension_list | | ├── ctlplane |------------------------------------| ├── floating_ip_list | | │ ├── neutron | | ├── network_list | | │ ├── nova |----------------- | ├── port_list | | │ └── placement | | | ├── router_list | | ├── dbs | +---------------------------+ | ├── security_group_list | | ├── namespaces | | namespaces/openstack/ | | └── subnet_list | | │ ├── cert-manager | | ├── all_resources.log | +-----------------------------+ | │ ├── openshift-machine-api | | ├── buildconfig |----------------------------------- | │ ├── openshift-nmstate | | ├── configmaps | | | │ ├── openstack | | ├── cronjobs | +--------------------------------------------------------------------+ | │ └── openstack-operators | | ├── crs | | namespaces/openstack/secrets/glance/ | | ├── network | | ├── daemonset | | ├── cert-glance-default-public-route.yaml | | │ ├── ipaddresspools | | ├── deployments | | ├── glance-config-data.yaml | | │ ├── nnce | | ├── events.log | | ├── glance-config-data.yaml-00-config.conf | | │ └── nncp | | ├── installplans | | ├── glance-default-single-config-data.yaml | | ├── nodes | | ├── jobs | | ├── glance-default-single-config-data.yaml-00-config.conf | | ├── sos-reports | | ├── nad.log | | ├── glance-default-single-config-data.yaml-10-glance-httpd.conf | | │ ├── _all_nodes | | ├── pods | | ├── glance-default-single-config-data.yaml-httpd.conf | | │ ├── barbican | | ├── pvc.log | | ├── glance-default-single-config-data.yaml-ssl.conf | | │ ├── ceilometer | | ├── replicaset | | └── glance-scripts.yaml | | │ ├── glance | | ├── routes | +--------------------------------------------------------------------+ | │ ├── keystone | | ├── secrets | | | │ ├── neutron | | ├── services | +--------------------------------------------------------------------+ | │ ├── nova | | ├── statefulsets | | Note: if DO_NOT_MASK is passed in CI, secrets are dumped without | | │ ├── ovn | | └── subscriptions | | hiding any sensitive information. | | │ ├── ovs | +---------------------------+ +--------------------------------------------------------------------+ | │ ├── placement | | │ └── swift | | └── webhooks | | ├── mutating | | └── validating | +-----------------------------------+
+---------------------------+ | namespaces/openstack/ | ------------------------------------ | ├── buildconfig | | | ├── cronjobs | +--------------------------------------------------------+ | ├── crs | | namespaces/openstack/crs/ | | ├── daemonset | | ├── barbicanapis.barbican.openstack.org | | ├── deployments | | ├── barbicankeystonelisteners.barbican.openstack.org | | ├── events.log | | ├── barbicans.barbican.openstack.org | | ├── installplans | | ├── barbicanworkers.barbican.openstack.org | | ├── jobs | | ... | | ├── nad.log | | ... | | ├── pods | | ├── glanceapis.glance.openstack.org | | ├── all_resources.log | | └── glance-default-single.yaml | | ├── configmaps | | ├── glances.glance.openstack.org | | ├── pvc.log | | └── glance.yaml | | ├── replicaset | | ├── keystoneapis.keystone.openstack.org | | ├── routes | | ├── keystoneendpoints.keystone.openstack.org | | ├── secrets | | ├── keystoneservices.keystone.openstack.org | | ├── services | | ... | | ├── statefulsets | | ├── telemetries.telemetry.openstack.org | | └── subscriptions | | └── transporturls.rabbitmq.openstack.org | +---------------------------+ +--------------------------------------------------------+
+---------------------------+ | namespaces/openstack/ | ------------------------------------ | ├── buildconfig | | | ├── cronjobs | +-----------------------------------------------------------+ | ├── crs | | namespaces/openstack/pods/glance-dbpurge-28500481-f4jk9 | | ├── daemonset | | ├── glance-dbpurge-28500481-f4jk9-describe | | ├── deployments | | └── logs | | ├── events.log | | └── glance-dbpurge.log | | ├── installplans | | namespaces/openstack/pods/glance-default-single-0 | | ├── jobs | | ├── glance-default-single-0-describe | | ├── nad.log | | └── logs | | ├── pods | | ├── glance-api.log | | ├── all_resources.log | | ├── glance-httpd.log | | ├── configmaps | | └── glance-log.log | | ├── pvc.log | | namespaces/openstack/pods/glance-default-single-1 | | ├── replicaset | | ├── glance-default-single-1-describe | | ├── routes | | └── logs | | ├── secrets | | ├── glance-api.log | | ├── services | | ├── glance-httpd.log | | ├── statefulsets | | └── glance-log.log | | └── subscriptions | | namespaces/openstack/pods/glance-default-single-2 | +---------------------------+ | ├── glance-default-single-2-describe | | └── logs | | ├── glance-api.log | | ├── glance-httpd.log | | └── glance-log.log | +-----------------------------------------------------------+