Chapter 7. Migrating the Red Hat Ceph Storage cluster In the context of data plane adoption, where the Red Hat OpenStack Platform (RHOSP) services are redeployed in Red Hat OpenShift Container Platform (RHOCP), you migrate a director-deployed Red Hat Ceph Storage cluster by using a process called "externalizing" the Red Hat Ceph Storage cluster. There are two deployment topologies that include an internal Red Hat Ceph Storage cluster: RHOSP includes dedicated Red Hat Ceph Storage nodes to host object storage daemons (OSDs) Hyperconverged Infrastructure (HCI), where Compute and Storage services are colocated on hyperconverged nodes In either scenario, there are some Red Hat Ceph Storage processes that are deployed on RHOSP Controller nodes: Red Hat Ceph Storage monitors, Ceph Object Gateway (RGW), Rados Block Device (RBD), Ceph Metadata Server (MDS), Ceph Dashboard, and NFS Ganesha. To migrate your Red Hat Ceph Storage cluster, you must decommission the Controller nodes and move the Red Hat Ceph Storage daemons to a set of target nodes that are already part of the Red Hat Ceph Storage cluster. Prerequisites Complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see Red Hat Ceph Storage prerequisites . 7.1. Red Hat Ceph Storage daemon cardinality Red Hat Ceph Storage 7 and later applies strict constraints in the way daemons can be colocated within the same node. For more information, see the Red Hat Knowledgebase article Red Hat Ceph Storage: Supported configurations . Your topology depends on the available hardware and the amount of Red Hat Ceph Storage services in the Controller nodes that you retire. The amount of services that you can migrate depends on the amount of available nodes in the cluster. The following diagrams show the distribution of Red Hat Ceph Storage daemons on Red Hat Ceph Storage nodes where at least 3 nodes are required. The following scenario includes only RGW and RBD, without the Red Hat Ceph Storage dashboard: With the Red Hat Ceph Storage dashboard, but without Shared File Systems service (manila), at least 4 nodes are required. The Red Hat Ceph Storage dashboard has no failover: With the Red Hat Ceph Storage dashboard and the Shared File Systems service, a minimum of 5 nodes are required, and the Red Hat Ceph Storage dashboard has no failover: 7.2. Migrating the monitoring stack component to new nodes within an existing Red Hat Ceph Storage cluster The Red Hat Ceph Storage Dashboard module adds web-based monitoring and administration to the Ceph Manager. With director-deployed Red Hat Ceph Storage, the Red Hat Ceph Storage Dashboard is enabled as part of the overcloud deploy and is composed of the following components: Ceph Manager module Grafana Prometheus Alertmanager Node exporter The Red Hat Ceph Storage Dashboard containers are included through tripleo-container-image-prepare parameters, and high availability (HA) relies on HAProxy and Pacemaker to be deployed on the Red Hat OpenStack Platform (RHOSP) environment. For an external Red Hat Ceph Storage cluster, HA is not supported. In this procedure, you migrate and relocate the Ceph Monitoring components to free Controller nodes. Prerequisites Complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see Red Hat Ceph Storage prerequisites . 7.2.1. Migrating the monitoring stack to the target nodes To migrate the monitoring stack to the target nodes, you add the monitoring label to your existing nodes and update the configuration of each daemon. You do not need to migrate node exporters. These daemons are deployed across the nodes that are part of the Red Hat Ceph Storage cluster (the placement is '*'). Prerequisites Confirm that the firewall rules are in place and the ports are open for a given monitoring stack service. Note Depending on the target nodes and the number of deployed or active daemons, you can either relocate the existing containers to the target nodes, or select a subset of nodes that host the monitoring stack daemons. High availability (HA) is not supported. Reducing the placement with count: 1 allows you to migrate the existing daemons in a Hyperconverged Infrastructure, or hardware-limited, scenario without impacting other services. 7.2.1.1. Migrating the existing daemons to the target nodes The following procedure is an example of an environment with 3 Red Hat Ceph Storage nodes or ComputeHCI nodes. This scenario extends the monitoring labels to all the Red Hat Ceph Storage or ComputeHCI nodes that are part of the cluster. This means that you keep 3 placements for the target nodes. Procedure Add the monitoring label to all the Red Hat Ceph Storage or ComputeHCI nodes in the cluster: Verify that all the hosts on the target nodes have the monitoring label: Remove the labels from the Controller nodes: Dump the current monitoring stack spec: For each daemon, edit the current spec and replace the placement.hosts: section with the placement.label: section, for example: service_type: grafana service_name: grafana placement: label: monitoring networks: - 172.17.3.0/24 spec: port: 3100 This step also applies to Prometheus and Alertmanager specs. Apply the new monitoring spec to relocate the monitoring stack daemons: Verify that the daemons are deployed on the expected nodes: Note After you migrate the monitoring stack, you lose high availability. The monitoring stack daemons no longer have a Virtual IP address and HAProxy anymore. Node exporters are still running on all the nodes. Review the Red Hat Ceph Storage configuration to ensure that it aligns with the configuration on the target nodes. In particular, focus on the following configuration entries: Verify that the API_HOST/URL of the grafana , alertmanager and prometheus services points to the IP addresses on the storage network of the node where each daemon is relocated: Note The Ceph Dashboard, as the service provided by the Ceph mgr , is not impacted by the relocation. You might experience an impact when the active mgr daemon is migrated or is force-failed. However, you can define 3 replicas in the Ceph Manager configuration to redirect requests to a different instance. 7.3. Migrating Red Hat Ceph Storage MDS to new nodes within the existing cluster You can migrate the MDS daemon when Shared File Systems service (manila), deployed with either a cephfs-native or ceph-nfs back end, is part of the overcloud deployment. The MDS migration is performed by cephadm , and you move the daemons placement from a hosts-based approach to a label-based approach. This ensures that you can visualize the status of the cluster and where daemons are placed by using the ceph orch host command. You can also have a general view of how the daemons are co-located within a given host, as described in the Red Hat Knowledgebase article Red Hat Ceph Storage: Supported configurations . Prerequisites Complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see Red Hat Ceph Storage prerequisites . Procedure Verify that the Red Hat Ceph Storage cluster is healthy and check the MDS status: Retrieve more detailed information on the Ceph File System (CephFS) MDS status: Check the OSD blocklist and clean up the client list: Note When a file system client is unresponsive or misbehaving, the access to the file system might be forcibly terminated. This process is called eviction. Evicting a CephFS client prevents it from communicating further with MDS daemons and OSD daemons. Ordinarily, a blocklisted client cannot reconnect to the servers; you must unmount and then remount the client. However, permitting a client that was evicted to attempt to reconnect can be useful. Because CephFS uses the RADOS OSD blocklist to control client eviction, you can permit CephFS clients to reconnect by removing them from the blocklist. Get the hosts that are currently part of the Red Hat Ceph Storage cluster: Apply the MDS labels to the target nodes: Verify that all the hosts have the MDS label: Dump the current MDS spec: Edit the retrieved spec and replace the placement.hosts section with placement.label : Use the ceph orchestrator to apply the new MDS spec: This results in an increased number of MDS daemons. Check the new standby daemons that are temporarily added to the CephFS: To migrate MDS to the target nodes, set the MDS affinity that manages the MDS failover: Note It is possible to elect a dedicated MDS as "active" for a particular file system. To configure this preference, CephFS provides a configuration option for MDS called mds_join_fs , which enforces this affinity. When failing over MDS daemons, cluster monitors prefer standby daemons with mds_join_fs equal to the file system name with the failed rank. If no standby exists with mds_join_fs equal to the file system name, it chooses an unqualified standby as a replacement. Replace mds.mds.cephstorage-0.fqcshx with the daemon deployed on cephstorage-0 that was retrieved from the step. Remove the labels from the Controller nodes and force the MDS failover to the target node: The switch to the target node happens in the background. The new active MDS is the one that you set by using the mds_join_fs command. Check the result of the failover and the new deployed daemons: 7.4. Migrating Red Hat Ceph Storage RGW to external RHEL nodes For Hyperconverged Infrastructure (HCI) or dedicated Storage nodes, you must migrate the Ceph Object Gateway (RGW) daemons that are included in the Red Hat OpenStack Platform Controller nodes into the existing external Red Hat Enterprise Linux (RHEL) nodes. The external RHEL nodes typically include the Compute nodes for an HCI environment or Red Hat Ceph Storage nodes. Your environment must have Red Hat Ceph Storage 7 or later and be managed by cephadm or Ceph Orchestrator. Prerequisites Complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see Red Hat Ceph Storage prerequisites . 7.4.1. Migrating the Red Hat Ceph Storage RGW back ends You must migrate your Ceph Object Gateway (RGW) back ends from your Controller nodes to your Red Hat Ceph Storage nodes. To ensure that you distribute the correct amount of services to your available nodes, you use cephadm labels to refer to a group of nodes where a given daemon type is deployed. For more information about the cardinality diagram, see Red Hat Ceph Storage daemon cardinality . The following procedure assumes that you have three target nodes, cephstorage-0 , cephstorage-1 , cephstorage-2 . Procedure Add the RGW label to the Red Hat Ceph Storage nodes that you want to migrate your RGW back ends to: Locate the RGW spec and dump in the spec directory: This example assumes that 172.17.3.0/24 is the storage network. In the placement section, ensure that the label and rgw_frontend_port values are set: 1 Add the storage network where the RGW back ends are deployed. 2 Replace the Controller nodes with the label: rgw label. 3 Change the rgw_frontend_port value to 8090 to avoid conflicts with the Ceph ingress daemon. 4 Optional: if TLS is enabled, add the SSL certificate and key concatenation as described in Configuring RGW with TLS for an external Red Hat Ceph Storage cluster in Configuring persistent storage . Apply the new RGW spec by using the orchestrator CLI: This command triggers the redeploy, for example: Ensure that the new RGW back ends are reachable on the new ports, so you can enable an ingress daemon on port 8080 later. Log in to each Red Hat Ceph Storage node that includes RGW and add the iptables rule to allow connections to both 8080 and 8090 ports in the Red Hat Ceph Storage nodes: If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content: Save the file. Restart the nftables service: Verify that the rules are applied: From a Controller node, such as controller-0 , try to reach the RGW back ends: You should observe the following output: Repeat the verification for each node where a RGW daemon is deployed. If you migrated RGW back ends to the Red Hat Ceph Storage nodes, there is no internalAPI network, except in the case of HCI nodes. You must reconfigure the RGW keystone endpoint to point to the external network that you propagated: Replace <keystone_endpoint> with the Identity service (keystone) internal endpoint of the service that is deployed in the OpenStackControlPlane CR when you adopt the Identity service. For more information, see Adopting the Identity service . 7.4.2. Deploying a Red Hat Ceph Storage ingress daemon To deploy the Ceph ingress daemon, you perform the following actions: Remove the existing ceph_rgw configuration. Clean up the configuration created by director. Redeploy the Object Storage service (swift). When you deploy the ingress daemon, two new containers are created: HAProxy, which you use to reach the back ends. Keepalived, which you use to own the virtual IP address. You use the rgw label to distribute the ingress daemon to only the number of nodes that host Ceph Object Gateway (RGW) daemons. For more information about distributing daemons among your nodes, see Red Hat Ceph Storage daemon cardinality . After you complete this procedure, you can reach the RGW back end from the ingress daemon and use RGW through the Object Storage service CLI. Procedure Log in to each Controller node and remove the following configuration from the /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg file: Restart haproxy-bundle and confirm that it is started: Confirm that no process is connected to port 8080: You can expect the Object Storage service (swift) CLI to fail to establish the connection: Set the required images for both HAProxy and Keepalived: Create a file called rgw_ingress in controller-0 : Paste the following content into the rgw_ingress file: --- service_type: ingress service_id: rgw.rgw placement: label: rgw spec: backend_service: rgw.rgw virtual_ip: 10.0.0.89/24 frontend_port: 8080 monitor_port: 8898 virtual_interface_networks: - <external_network> ssl_cert: ... Replace <external_network> with your external network, for example, 10.0.0.0/24 . For more information, see Completing prerequisites for migrating Red Hat Ceph Storage RGW . If TLS is enabled, add the SSL certificate and key concatenation as described in Configuring RGW with TLS for an external Red Hat Ceph Storage cluster in Configuring persistent storage . Apply the rgw_ingress spec by using the Ceph orchestrator CLI: Wait until the ingress is deployed and query the resulting endpoint: 7.4.3. Updating the Object Storage service endpoints You must update the Object Storage service (swift) endpoints to point to the new virtual IP address (VIP) that you reserved on the same network that you used to deploy RGW ingress. Procedure List the current endpoints: Update the endpoints that are pointing to the Ingress VIP: Repeat this step for both internal and admin endpoints. Test the migrated service: 7.5. Migrating Red Hat Ceph Storage RBD to external RHEL nodes For Hyperconverged Infrastructure (HCI) or dedicated Storage nodes that are running Red Hat Ceph Storage 7 or later, you must migrate the daemons that are included in the Red Hat OpenStack Platform control plane into the existing external Red Hat Enterprise Linux (RHEL) nodes. The external RHEL nodes typically include the Compute nodes for an HCI environment or dedicated storage nodes. Prerequisites Complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see Red Hat Ceph Storage prerequisites . 7.5.1. Migrating Ceph Manager daemons to Red Hat Ceph Storage nodes You must migrate your Ceph Manager daemons from the Red Hat OpenStack Platform (RHOSP) Controller nodes to a set of target nodes. Target nodes are either existing Red Hat Ceph Storage nodes, or RHOSP Compute nodes if Red Hat Ceph Storage is deployed by director with a Hyperconverged Infrastructure (HCI) topology. Note The following procedure uses cephadm and the Ceph Orchestrator to drive the Ceph Manager migration, and the Ceph spec to modify the placement and reschedule the Ceph Manager daemons. Ceph Manager is run in an active/passive state. It also provides many modules, including the Ceph Orchestrator. Every potential module, such as the Ceph Dashboard, that is provided by ceph-mgr is implicitly migrated with Ceph Manager. Procedure SSH into the target node and enable the firewall rules that are required to reach a Ceph Manager service: Replace <target_node> with the hostname of the hosts that are listed in the Red Hat Ceph Storage environment. Run ceph orch host ls to see the list of the hosts. Repeat this step for each target node. Check that the rules are properly applied to the target node and persist them: If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content: Save the file. Restart the nftables service: Verify that the rules are applied: Prepare the target node to host the new Ceph Manager daemon, and add the mgr label to the target node: Repeat steps 1-7 for each target node that hosts a Ceph Manager daemon. Get the Ceph Manager spec: USD SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} USD mkdir -p USD{SPEC_DIR} USD sudo cephadm shell -- ceph orch ls --export mgr > USD{SPEC_DIR}/mgr Edit the retrieved spec and add the label: mgr section to the placement section: service_type: mgr service_id: mgr placement: label: mgr Save the spec. Apply the spec with cephadm by using the Ceph Orchestrator: Verification Verify that the new Ceph Manager daemons are created in the target nodes: The Ceph Manager daemon count should match the number of hosts where the mgr label is added. Note The migration does not shrink the Ceph Manager daemons. The count grows by the number of target nodes, and migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes decommissions the stand-by Ceph Manager instances. For more information, see Migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes . 7.5.2. Migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes You must move Ceph Monitor daemons from the Red Hat OpenStack Platform (RHOSP) Controller nodes to a set of target nodes. Target nodes are either existing Red Hat Ceph Storage nodes, or RHOSP Compute nodes if Red Hat Ceph Storage is deployed by director with a Hyperconverged Infrastructure (HCI) topology. Additional Ceph Monitors are deployed to the target nodes, and they are promoted as _admin nodes that you can use to manage the Red Hat Ceph Storage cluster and perform day 2 operations. To migrate the Ceph Monitor daemons, you must perform the following high-level steps: Configure the target nodes for Ceph Monitor migration . Drain the source node . Migrate your Ceph Monitor IP addresses to the target nodes . Redeploy the Ceph Monitor on the target node . Verify that the Red Hat Ceph Storage cluster is healthy . Repeat these steps for any additional Controller node that hosts a Ceph Monitor until you migrate all the Ceph Monitor daemons to the target nodes. 7.5.2.1. Configuring target nodes for Ceph Monitor migration Prepare the target Red Hat Ceph Storage nodes for the Ceph Monitor migration by performing the following actions: Enable firewall rules in a target node and persist them. Create a spec that is based on labels and apply it by using cephadm . Ensure that the Ceph Monitor quorum is maintained during the migration process. Procedure SSH into the target node and enable the firewall rules that are required to reach a Ceph Monitor service: Replace <target_node> with the hostname of the node that hosts the new Ceph Monitor. Check that the rules are properly applied to the target node and persist them: If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content: Save the file. Restart the nftables service: Verify that the rules are applied: To migrate the existing Ceph Monitors to the target Red Hat Ceph Storage nodes, retrieve the Red Hat Ceph Storage mon spec from the first Ceph Monitor, or the first Controller node: Add the label:mon section to the placement section: Save the spec. Apply the spec with cephadm by using the Ceph Orchestrator: Extend the mon label to the remaining Red Hat Ceph Storage target nodes to ensure that quorum is maintained during the migration process: Note Applying the mon spec allows the existing strategy to use labels instead of hosts . As a result, any node with the mon label can host a Ceph Monitor daemon. Perform this step only once to avoid multiple iterations when multiple Ceph Monitors are migrated. Check the status of the Red Hat Ceph Storage and the Ceph Orchestrator daemons list. Ensure that Ceph Monitors are in a quorum and listed by the ceph orch command: Set up a Ceph client on the first Controller node that is used during the rest of the procedure to interact with Red Hat Ceph Storage. Set up an additional IP address on the storage network that is used to interact with Red Hat Ceph Storage when the first Controller node is decommissioned: Back up the content of /etc/ceph in the ceph_client_backup directory. Edit /etc/os-net-config/config.yaml and add - ip_netmask: 172.17.3.200 after the IP address on the VLAN that belongs to the storage network. Replace 172.17.3.200 with any other available IP address on the storage network that can be statically assigned to controller-0 . Save the file and refresh the controller-0 network configuration: Verify that the IP address is present in the Controller node: Ping the IP address and confirm that it is reachable: Verify that you can interact with the Red Hat Ceph Storage cluster: steps Proceed to the step Draining the source node . 7.5.2.2. Draining the source node Drain the source node and remove the source node host from the Red Hat Ceph Storage cluster. Procedure On the source node, back up the /etc/ceph/ directory to run cephadm and get a shell for the Red Hat Ceph Storage cluster from the source node: Identify the active ceph-mgr instance: Fail the ceph-mgr if it is active on the source node: Replace <mgr_instance> with the Ceph Manager daemon to fail. From the cephadm shell, remove the labels on the source node: Replace <source_node> with the hostname of the source node. Optional: Ensure that you remove the Ceph Monitor daemon from the source node if it is still running: Drain the source node to remove any leftover daemons: Remove the source node host from the Red Hat Ceph Storage cluster: Note The source node is not part of the cluster anymore, and should not appear in the Red Hat Ceph Storage host list when you run sudo cephadm shell -- ceph orch host ls . However, if you run sudo podman ps in the source node, the list might show that both Ceph Monitors and Ceph Managers are still running. To clean up the existing containers and remove the cephadm data from the source node, contact Red Hat Support. Confirm that mons are still in quorum: steps Proceed to the step Migrating the Ceph Monitor IP address . 7.5.2.3. Migrating the Ceph Monitor IP address You must migrate your Ceph Monitor IP addresses to the target Red Hat Ceph Storage nodes. The IP address migration assumes that the target nodes are originally deployed by director and that the network configuration is managed by os-net-config . Procedure Get the original Ceph Monitor IP addresses from USDHOME/ceph_client_backup/ceph.conf file on the mon_host line, for example: Match the IP address retrieved in the step with the storage network IP addresses on the source node, and find the Ceph Monitor IP address: Confirm that the Ceph Monitor IP address is present in the os-net-config configuration that is located in the /etc/os-net-config directory on the source node: Edit the /etc/os-net-config/config.yaml file and remove the ip_netmask line. Save the file and refresh the node network configuration: Verify that the IP address is not present in the source node anymore, for example: SSH into the target node, for example cephstorage-0 , and add the IP address for the new Ceph Monitor. On the target node, edit /etc/os-net-config/config.yaml and add the - ip_netmask: 172.17.3.60 line that you removed in the source node. Save the file and refresh the node network configuration: Verify that the IP address is present in the target node. From the Ceph client node, controller-0 , ping the IP address that is migrated to the target node and confirm that it is still reachable: steps Proceed to the step Redeploying the Ceph Monitor on the target node . 7.5.2.4. Redeploying a Ceph Monitor on the target node You use the IP address that you migrated to the target node to redeploy the Ceph Monitor on the target node. Procedure From the Ceph client node, for example controller-0 , get the Ceph mon spec: Edit the retrieved spec and add the unmanaged: true keyword: service_type: mon service_id: mon placement: label: mon unmanaged: true Save the spec. Apply the spec with cephadm by using the Ceph Orchestrator: The Ceph Monitor daemons are marked as unmanaged , and you can now redeploy the existing daemon and bind it to the migrated IP address. Delete the existing Ceph Monitor on the target node: Replace <target_node> with the hostname of the target node that is included in the Red Hat Ceph Storage cluster. Redeploy the new Ceph Monitor on the target node by using the migrated IP address: Replace <ip_address> with the IP address of the migrated IP address. Get the Ceph Monitor spec: Edit the retrieved spec and set the unmanaged keyword to false : service_type: mon service_id: mon placement: label: mon unmanaged: false Save the spec. Apply the spec with cephadm by using the Ceph Orchestrator: The new Ceph Monitor runs on the target node with the original IP address. Identify the running mgr : Refresh the Ceph Manager information by force-failing it: Refresh the OSD information: steps Repeat the procedure starting from step Draining the source node for each node that you want to decommission. Proceed to the step Verifying the Red Hat Ceph Storage cluster after Ceph Monitor migration . 7.5.2.5. Verifying the Red Hat Ceph Storage cluster after Ceph Monitor migration After you finish migrating your Ceph Monitor daemons to the target nodes, verify that the the Red Hat Ceph Storage cluster is healthy. Procedure Verify that the Red Hat Ceph Storage cluster is healthy: Verify that the Red Hat Ceph Storage mons are running with the old IP addresses. SSH into the target nodes and verify that the Ceph Monitor daemons are bound to the expected IP and port: 7.6. Updating the Red Hat Ceph Storage cluster Ceph Dashboard configuration If the Ceph Dashboard is part of the enabled Ceph Manager modules, you need to reconfigure the failover settings. Procedure Regenerate the following Red Hat Ceph Storage configuration keys to point to the right mgr container: For each retrieved mgr daemon, update the corresponding entry in the Red Hat Ceph Storage configuration:

Commands:
| | | | |----|---------------------|-------------| | osd | mon/mgr/crash | rgw/ingress | | osd | mon/mgr/crash | rgw/ingress | | osd | mon/mgr/crash | rgw/ingress |
| | | | |-----|---------------------|-------------| | osd | mon/mgr/crash | rgw/ingress | | osd | mon/mgr/crash | rgw/ingress | | osd | mon/mgr/crash | dashboard/grafana | | osd | rgw/ingress | (free) |
| | | | |-----|---------------------|-------------------------| | osd | mon/mgr/crash | rgw/ingress | | osd | mon/mgr/crash | rgw/ingress | | osd | mon/mgr/crash | mds/ganesha/ingress | | osd | rgw/ingress | mds/ganesha/ingress | | osd | mds/ganesha/ingress | dashboard/grafana |
for item in USD(sudo cephadm shell -- ceph orch host ls --format json | jq -r '.[].hostname'); do sudo cephadm shell -- ceph orch host label add USDitem monitoring; done
[tripleo-admin@controller-0 ~]USD sudo cephadm shell -- ceph orch host ls HOST ADDR LABELS cephstorage-0.redhat.local 192.168.24.11 osd monitoring cephstorage-1.redhat.local 192.168.24.12 osd monitoring cephstorage-2.redhat.local 192.168.24.47 osd monitoring controller-0.redhat.local 192.168.24.35 _admin mon mgr monitoring controller-1.redhat.local 192.168.24.53 mon _admin mgr monitoring controller-2.redhat.local 192.168.24.10 mon _admin mgr monitoring
for i in 0 1 2; do sudo cephadm shell -- ceph orch host label rm "controller-USDi.redhat.local" monitoring; done Removed label monitoring from host controller-0.redhat.local Removed label monitoring from host controller-1.redhat.local Removed label monitoring from host controller-2.redhat.local
function export_spec { local component="USD1" local target_dir="USD2" sudo cephadm shell -- ceph orch ls --export "USDcomponent" > "USDtarget_dir/USDcomponent" } SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} mkdir -p USD{SPEC_DIR} for m in grafana prometheus alertmanager; do export_spec "USDm" "USDSPEC_DIR" done
service_type: grafana service_name: grafana placement: label: monitoring networks: - 172.17.3.0/24 spec: port: 3100
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} function migrate_daemon { local component="USD1" local target_dir="USD2" sudo cephadm shell -m "USDtarget_dir" -- ceph orch apply -i /mnt/ceph_specs/USDcomponent } for m in grafana prometheus alertmanager; do migrate_daemon "USDm" "USDSPEC_DIR" done
ceph orch ps | grep -iE "(prome|alert|grafa)" alertmanager.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:9093,9094 grafana.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:3100 prometheus.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:9092
ceph config dump | grep -i dashboard mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://172.17.3.83:9093 mgr advanced mgr/dashboard/GRAFANA_API_URL https://172.17.3.144:3100 mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://172.17.3.83:9092 mgr advanced mgr/dashboard/controller-0.ycokob/server_addr 172.17.3.33 mgr advanced mgr/dashboard/controller-1.lmzpuc/server_addr 172.17.3.147 mgr advanced mgr/dashboard/controller-2.xpdgfl/server_addr 172.17.3.138
ceph orch ps | grep -iE "(prome|alert|grafa)" alertmanager.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:9093,9094 alertmanager.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:9093,9094 alertmanager.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:9093,9094 grafana.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:3100 grafana.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:3100 grafana.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:3100 prometheus.cephstorage-0 cephstorage-0.redhat.local 172.17.3.83:9092 prometheus.cephstorage-1 cephstorage-1.redhat.local 172.17.3.53:9092 prometheus.cephstorage-2 cephstorage-2.redhat.local 172.17.3.144:9092
ceph config dump mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://172.17.3.83:9093 mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://172.17.3.83:9092 mgr advanced mgr/dashboard/GRAFANA_API_URL https://172.17.3.144:3100
sudo cephadm shell -- ceph fs ls name: cephfs, metadata pool: manila_metadata, data pools: [manila_data ] sudo cephadm shell -- ceph mds stat cephfs:1 {0=mds.controller-2.oebubl=up:active} 2 up:standby sudo cephadm shell -- ceph fs status cephfs cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active mds.controller-2.oebubl Reqs: 0 /s 696 196 173 0 POOL TYPE USED AVAIL manila_metadata metadata 152M 141G manila_data data 3072M 141G STANDBY MDS mds.controller-0.anwiwd mds.controller-1.cwzhog
sudo cephadm shell -- ceph fs dump e8 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 5 flags 12 joinable allow_snaps allow_multimds_snaps created 2024-01-18T19:04:01.633820+0000 modified 2024-01-18T19:04:05.393046+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=24553} failed damaged stopped data_pools [7] metadata_pool 9 inline_data disabled balancer standby_count_wanted 1 [mds.mds.controller-2.oebubl{0:24553} state up:active seq 2 addr [v2:172.17.3.114:6800/680266012,v1:172.17.3.114:6801/680266012] compat {c=[1],r=[1],i=[7ff]}] Standby daemons: [mds.mds.controller-0.anwiwd{-1:14715} state up:standby seq 1 addr [v2:172.17.3.20:6802/3969145800,v1:172.17.3.20:6803/3969145800] compat {c=[1],r=[1],i=[7ff]}] [mds.mds.controller-1.cwzhog{-1:24566} state up:standby seq 1 addr [v2:172.17.3.43:6800/2227381308,v1:172.17.3.43:6801/2227381308] compat {c=[1],r=[1],i=[7ff]}] dumped fsmap epoch 8
sudo cephadm shell -- ceph osd blocklist ls .. .. for item in USD(sudo cephadm shell -- ceph osd blocklist ls | awk '{print USD1}'); do sudo cephadm shell -- ceph osd blocklist rm USDitem; done
ceph orch host ls HOST ADDR LABELS STATUS cephstorage-0.redhat.local 192.168.24.25 osd cephstorage-1.redhat.local 192.168.24.50 osd cephstorage-2.redhat.local 192.168.24.47 osd controller-0.redhat.local 192.168.24.24 _admin mgr mon controller-1.redhat.local 192.168.24.42 mgr _admin mon controller-2.redhat.local 192.168.24.37 mgr _admin mon 6 hosts in cluster
for item in USD(sudo cephadm shell -- ceph orch host ls --format json | jq -r '.[].hostname'); do sudo cephadm shell -- ceph orch host label add USDitem mds; done
sudo cephadm shell -- ceph orch host ls HOST ADDR LABELS cephstorage-0.redhat.local 192.168.24.11 osd mds cephstorage-1.redhat.local 192.168.24.12 osd mds cephstorage-2.redhat.local 192.168.24.47 osd mds controller-0.redhat.local 192.168.24.35 _admin mon mgr mds controller-1.redhat.local 192.168.24.53 mon _admin mgr mds controller-2.redhat.local 192.168.24.10 mon _admin mgr mds
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} mkdir -p USD{SPEC_DIR} sudo cephadm shell -- ceph orch ls --export mds > USD{SPEC_DIR}/mds
service_type: mds service_id: mds service_name: mds.mds placement: label: mds
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -m USD{SPEC_DIR}/mds -- ceph orch apply -i /mnt/mds Scheduling new mds deployment
sudo cephadm shell -- ceph fs dump Active standby_count_wanted 1 [mds.mds.controller-0.awzplm{0:463158} state up:active seq 307 join_fscid=1 addr [v2:172.17.3.20:6802/51565420,v1:172.17.3.20:6803/51565420] compat {c=[1],r=[1],i=[7ff]}] Standby daemons: [mds.mds.cephstorage-1.jkvomp{-1:463800} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.135:6820/2075903648,v1:172.17.3.135:6821/2075903648] compat {c=[1],r=[1],i=[7ff]}] [mds.mds.controller-2.gfrqvc{-1:475945} state up:standby seq 1 addr [v2:172.17.3.114:6800/2452517189,v1:172.17.3.114:6801/2452517189] compat {c=[1],r=[1],i=[7ff]}] [mds.mds.cephstorage-0.fqcshx{-1:476503} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.92:6820/4120523799,v1:172.17.3.92:6821/4120523799] compat {c=[1],r=[1],i=[7ff]}] [mds.mds.cephstorage-2.gnfhfe{-1:499067} state up:standby seq 1 addr [v2:172.17.3.79:6820/2448613348,v1:172.17.3.79:6821/2448613348] compat {c=[1],r=[1],i=[7ff]}] [mds.mds.controller-1.tyiziq{-1:499136} state up:standby seq 1 addr [v2:172.17.3.43:6800/3615018301,v1:172.17.3.43:6801/3615018301] compat {c=[1],r=[1],i=[7ff]}]
sudo cephadm shell -- ceph config set mds.mds.cephstorage-0.fqcshx mds_join_fs cephfs
for i in 0 1 2; do sudo cephadm shell -- ceph orch host label rm "controller-USDi.redhat.local" mds; done Removed label mds from host controller-0.redhat.local Removed label mds from host controller-1.redhat.local Removed label mds from host controller-2.redhat.local
sudo cephadm shell -- ceph fs dump ... ... standby_count_wanted 1 [mds.mds.cephstorage-0.fqcshx{0:476503} state up:active seq 168 join_fscid=1 addr [v2:172.17.3.92:6820/4120523799,v1:172.17.3.92:6821/4120523799] compat {c=[1],r=[1],i=[7ff]}] Standby daemons: [mds.mds.cephstorage-2.gnfhfe{-1:499067} state up:standby seq 1 addr [v2:172.17.3.79:6820/2448613348,v1:172.17.3.79:6821/2448613348] compat {c=[1],r=[1],i=[7ff]}] [mds.mds.cephstorage-1.jkvomp{-1:499760} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.135:6820/452139733,v1:172.17.3.135:6821/452139733] compat {c=[1],r=[1],i=[7ff]}] sudo cephadm shell -- ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT crash 6/6 10m ago 10d * mds.mds 3/3 10m ago 32m label:mds sudo cephadm shell -- ceph orch ps | grep mds mds.mds.cephstorage-0.fqcshx cephstorage-0.redhat.local running (79m) 3m ago 79m 27.2M - 17.2.6-100.el9cp 1af7b794f353 2a2dc5ba6d57 mds.mds.cephstorage-1.jkvomp cephstorage-1.redhat.local running (79m) 3m ago 79m 21.5M - 17.2.6-100.el9cp 1af7b794f353 7198b87104c8 mds.mds.cephstorage-2.gnfhfe cephstorage-2.redhat.local running (79m) 3m ago 79m 24.2M - 17.2.6-100.el9cp 1af7b794f353 f3cb859e2a15
sudo cephadm shell -- ceph orch host label add cephstorage-0 rgw; sudo cephadm shell -- ceph orch host label add cephstorage-1 rgw; sudo cephadm shell -- ceph orch host label add cephstorage-2 rgw; Added label rgw to host cephstorage-0 Added label rgw to host cephstorage-1 Added label rgw to host cephstorage-2 sudo cephadm shell -- ceph orch host ls HOST ADDR LABELS STATUS cephstorage-0 192.168.24.54 osd rgw cephstorage-1 192.168.24.44 osd rgw cephstorage-2 192.168.24.30 osd rgw controller-0 192.168.24.45 _admin mon mgr controller-1 192.168.24.11 _admin mon mgr controller-2 192.168.24.38 _admin mon mgr 6 hosts in cluster
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} mkdir -p USD{SPEC_DIR} sudo cephadm shell -- ceph orch ls --export rgw > USD{SPEC_DIR}/rgw cat USD{SPEC_DIR}/rgw networks: - 172.17.3.0/24 placement: hosts: - controller-0 - controller-1 - controller-2 service_id: rgw service_name: rgw.rgw service_type: rgw spec: rgw_frontend_port: 8080 rgw_realm: default rgw_zone: default
--- networks: - 172.17.3.0/24 1 placement: label: rgw 2 service_id: rgw service_name: rgw.rgw service_type: rgw spec: rgw_frontend_port: 8090 3 rgw_realm: default rgw_zone: default rgw_frontend_ssl_certificate: ... 4 ssl: true
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -m USD{SPEC_DIR}/rgw -- ceph orch apply -i /mnt/rgw
osd.9 cephstorage-2 rgw.rgw.cephstorage-0.wsjlgx cephstorage-0 172.17.3.23:8090 starting rgw.rgw.cephstorage-1.qynkan cephstorage-1 172.17.3.26:8090 starting rgw.rgw.cephstorage-2.krycit cephstorage-2 172.17.3.81:8090 starting rgw.rgw.controller-1.eyvrzw controller-1 172.17.3.146:8080 running (5h) rgw.rgw.controller-2.navbxa controller-2 172.17.3.66:8080 running (5h) osd.9 cephstorage-2 rgw.rgw.cephstorage-0.wsjlgx cephstorage-0 172.17.3.23:8090 running (19s) rgw.rgw.cephstorage-1.qynkan cephstorage-1 172.17.3.26:8090 running (16s) rgw.rgw.cephstorage-2.krycit cephstorage-2 172.17.3.81:8090 running (13s)
iptables -I INPUT -p tcp -m tcp --dport 8080 -m conntrack --ctstate NEW -m comment --comment "ceph rgw ingress" -j ACCEPT iptables -I INPUT -p tcp -m tcp --dport 8090 -m conntrack --ctstate NEW -m comment --comment "ceph rgw backends" -j ACCEPT sudo iptables-save sudo systemctl restart iptables
100 ceph_rgw {'dport': ['8080','8090']} add rule inet filter TRIPLEO_INPUT tcp dport { 8080,8090 } ct state new counter accept comment "100 ceph_rgw"
sudo systemctl restart nftables
sudo nft list ruleset | grep ceph_rgw
curl http://cephstorage-0.storage:8090;
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
ceph config dump | grep keystone global basic rgw_keystone_url http://172.16.1.111:5000 ceph config set global rgw_keystone_url http://<keystone_endpoint>:5000
listen ceph_rgw bind 10.0.0.103:8080 transparent mode http balance leastconn http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc } http-request set-header X-Forwarded-Port %[dst_port] option httpchk GET /swift/healthcheck option httplog option forwardfor server controller-0.storage.redhat.local 172.17.3.73:8080 check fall 5 inter 2000 rise 2 server controller-1.storage.redhat.local 172.17.3.146:8080 check fall 5 inter 2000 rise 2 server controller-2.storage.redhat.local 172.17.3.156:8080 check fall 5 inter 2000 rise 2
sudo pcs resource restart haproxy-bundle haproxy-bundle successfully restarted sudo pcs status | grep haproxy * Container bundle set: haproxy-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-haproxy:pcmklatest]: * haproxy-bundle-podman-0 (ocf:heartbeat:podman): Started controller-0 * haproxy-bundle-podman-1 (ocf:heartbeat:podman): Started controller-1 * haproxy-bundle-podman-2 (ocf:heartbeat:podman): Started controller-2
ss -antop | grep 8080
(overcloud) [root@cephstorage-0 ~]# swift list HTTPConnectionPool(host='10.0.0.103', port=8080): Max retries exceeded with url: /swift/v1/AUTH_852f24425bb54fa896476af48cbe35d3?format=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc41beb0430>: Failed to establish a new connection: [Errno 111] Connection refused'))
ceph config set mgr mgr/cephadm/container_image_haproxy registry.redhat.io/rhceph/rhceph-haproxy-rhel9:latest ceph config set mgr mgr/cephadm/container_image_keepalived registry.redhat.io/rhceph/keepalived-rhel9:latest
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} vim USD{SPEC_DIR}/rgw_ingress
--- service_type: ingress service_id: rgw.rgw placement: label: rgw spec: backend_service: rgw.rgw virtual_ip: 10.0.0.89/24 frontend_port: 8080 monitor_port: 8898 virtual_interface_networks: - <external_network> ssl_cert:
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} cephadm shell -m USD{SPEC_DIR}/rgw_ingress -- ceph orch apply -i /mnt/rgw_ingress
sudo cephadm shell -- ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT crash 6/6 6m ago 3d * ingress.rgw.rgw 10.0.0.89:8080,8898 6/6 37s ago 60s label:rgw mds.mds 3/3 6m ago 3d controller-0;controller-1;controller-2 mgr 3/3 6m ago 3d controller-0;controller-1;controller-2 mon 3/3 6m ago 3d controller-0;controller-1;controller-2 osd.default_drive_group 15 37s ago 3d cephstorage-0;cephstorage-1;cephstorage-2 rgw.rgw ?:8090 3/3 37s ago 4m label:rgw
curl 10.0.0.89:8080 --- <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[ceph: root@controller-0 /]# -
(overcloud) [stack@undercloud-0 ~]USD openstack endpoint list | grep object | 1326241fb6b6494282a86768311f48d1 | regionOne | swift | object-store | True | internal | http://172.17.3.68:8080/swift/v1/AUTH_%(project_id)s | | 8a34817a9d3443e2af55e108d63bb02b | regionOne | swift | object-store | True | public | http://10.0.0.103:8080/swift/v1/AUTH_%(project_id)s | | fa72f8b8b24e448a8d4d1caaeaa7ac58 | regionOne | swift | object-store | True | admin | http://172.17.3.68:8080/swift/v1/AUTH_%(project_id)s |
(overcloud) [stack@undercloud-0 ~]USD openstack endpoint set --url "http://10.0.0.89:8080/swift/v1/AUTH_%(project_id)s" 95596a2d92c74c15b83325a11a4f07a3 (overcloud) [stack@undercloud-0 ~]USD openstack endpoint list | grep object-store | 6c7244cc8928448d88ebfad864fdd5ca | regionOne | swift | object-store | True | internal | http://172.17.3.79:8080/swift/v1/AUTH_%(project_id)s | | 95596a2d92c74c15b83325a11a4f07a3 | regionOne | swift | object-store | True | public | http://10.0.0.89:8080/swift/v1/AUTH_%(project_id)s | | e6d0599c5bf24a0fb1ddf6ecac00de2d | regionOne | swift | object-store | True | admin | http://172.17.3.79:8080/swift/v1/AUTH_%(project_id)s |
(overcloud) [stack@undercloud-0 ~]USD swift list --debug DEBUG:swiftclient:Versionless auth_url - using http://10.0.0.115:5000/v3 as endpoint DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://10.0.0.115:5000/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.0.0.115:5000 DEBUG:urllib3.connectionpool:http://10.0.0.115:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7795 DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "6f87c7ffdddf463bbc633980cfd02bb3", "name": "admin", "password_expires_at": null}, DEBUG:swiftclient:REQ: curl -i http://10.0.0.89:8080/swift/v1/AUTH_852f24425bb54fa896476af48cbe35d3?format=json -X GET -H "X-Auth-Token: gAAAAABj7KHdjZ95syP4c8v5a2zfXckPwxFQZYg0pgWR42JnUs83CcKhYGY6PFNF5Cg5g2WuiYwMIXHm8xftyWf08zwTycJLLMeEwoxLkcByXPZr7kT92ApT-36wTfpi-zbYXd1tI5R00xtAzDjO3RH1kmeLXDgIQEVp0jMRAxoVH4zb-DVHUos" -H "Accept-Encoding: gzip" DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: {'content-length': '2', 'x-timestamp': '1676452317.72866', 'x-account-container-count': '0', 'x-account-object-count': '0', 'x-account-bytes-used': '0', 'x-account-bytes-used-actual': '0', 'x-account-storage-policy-default-placement-container-count': '0', 'x-account-storage-policy-default-placement-object-count': '0', 'x-account-storage-policy-default-placement-bytes-used': '0', 'x-account-storage-policy-default-placement-bytes-used-actual': '0', 'x-trans-id': 'tx00000765c4b04f1130018-0063eca1dd-1dcba-default', 'x-openstack-request-id': 'tx00000765c4b04f1130018-0063eca1dd-1dcba-default', 'accept-ranges': 'bytes', 'content-type': 'application/json; charset=utf-8', 'date': 'Wed, 15 Feb 2023 09:11:57 GMT'} DEBUG:swiftclient:RESP BODY: b'[]'
dports="6800:7300" ssh heat-admin@<target_node> sudo iptables -I INPUT -p tcp --match multiport --dports USDdports -j ACCEPT;
sudo iptables-save sudo systemctl restart iptables
113 ceph_mgr {'dport': ['6800-7300', 8444]} add rule inet filter TRIPLEO_INPUT tcp dport { 6800-7300,8444 } ct state new counter accept comment "113 ceph_mgr"
sudo systemctl restart nftables
sudo nft list ruleset | grep ceph_mgr
sudo cephadm shell -- ceph orch host label add <target_node> mgr
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} mkdir -p USD{SPEC_DIR} sudo cephadm shell -- ceph orch ls --export mgr > USD{SPEC_DIR}/mgr
service_type: mgr service_id: mgr placement: label: mgr
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -m USD{SPEC_DIR}/mgr -- ceph orch apply -i /mnt/mgr
sudo cephadm shell -- ceph orch ps | grep -i mgr sudo cephadm shell -- ceph -s
for port in 3300 6789; { ssh heat-admin@<target_node> sudo iptables -I INPUT -p tcp -m tcp --dport USDport -m conntrack --ctstate NEW -j ACCEPT; }
sudo iptables-save sudo systemctl restart iptables
110 ceph_mon {'dport': [6789, 3300, '9100']} add rule inet filter TRIPLEO_INPUT tcp dport { 6789,3300,9100 } ct state new counter accept comment "110 ceph_mon"
sudo systemctl restart nftables
sudo nft list ruleset | grep ceph_mon
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} mkdir -p USD{SPEC_DIR} sudo cephadm shell -- ceph orch ls --export mon > USD{SPEC_DIR}/mon
service_type: mon service_id: mon placement: label: mon
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -m USD{SPEC_DIR}/mon -- ceph orch apply -i /mnt/mon
for item in USD(sudo cephadm shell -- ceph orch host ls --format json | jq -r '.[].hostname'); do sudo cephadm shell -- ceph orch host label add USDitem mon; sudo cephadm shell -- ceph orch host label add USDitem _admin; done
sudo cephadm shell -- ceph -s cluster: id: f6ec3ebe-26f7-56c8-985d-eb974e8e08e3 health: HEALTH_OK services: mon: 6 daemons, quorum controller-0,controller-1,controller-2,ceph-0,ceph-1,ceph-2 (age 19m) mgr: controller-0.xzgtvo(active, since 32m), standbys: controller-1.mtxohd, controller-2.ahrgsk osd: 8 osds: 8 up (since 12m), 8 in (since 18m); 1 remapped pgs data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 43 MiB used, 400 GiB / 400 GiB avail pgs: 1 active+clean
sudo cephadm shell -- ceph orch host ls HOST ADDR LABELS STATUS ceph-0 192.168.24.14 osd mon mgr _admin ceph-1 192.168.24.7 osd mon mgr _admin ceph-2 192.168.24.8 osd mon mgr _admin controller-0 192.168.24.15 _admin mgr mon controller-1 192.168.24.23 _admin mgr mon controller-2 192.168.24.13 _admin mgr mon
mkdir -p USDHOME/ceph_client_backup sudo cp -R /etc/ceph/* USDHOME/ceph_client_backup
sudo os-net-config -c /etc/os-net-config/config.yaml
ip -o a | grep 172.17.3.200
ping -c 3 172.17.3.200
sudo cephadm shell -c USDHOME/ceph_client_backup/ceph.conf -k USDHOME/ceph_client_backup/ceph.client.admin.keyring -- ceph -s
mkdir -p USDHOME/ceph_client_backup sudo cp -R /etc/ceph USDHOME/ceph_client_backup
sudo cephadm shell -- ceph mgr stat
sudo cephadm shell -- ceph mgr fail <mgr_instance>
for label in mon mgr _admin; do sudo cephadm shell -- ceph orch host label rm <source_node> USDlabel; done
sudo cephadm shell -- ceph orch daemon rm mon.<source_node> --force
sudo cephadm shell -- ceph orch host drain <source_node>
sudo cephadm shell -- ceph orch host rm <source_node> --force
sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5c1ad36472bc registry.redhat.io/ceph/rhceph@sha256:320c364dcc8fc8120e2a42f54eb39ecdba12401a2546763b7bef15b02ce93bc4 -n mon.contro... 35 minutes ago Up 35 minutes ago ceph-f6ec3ebe-26f7-56c8-985d-eb974e8e08e3-mon-controller-1 3b14cc7bf4dd registry.redhat.io/ceph/rhceph@sha256:320c364dcc8fc8120e2a42f54eb39ecdba12401a2546763b7bef15b02ce93bc4 -n mgr.contro... 35 minutes ago Up 35 minutes ago ceph-f6ec3ebe-26f7-56c8-985d-eb974e8e08e3-mgr-controller-1-mtxohd
sudo cephadm shell -- ceph -s sudo cephadm shell -- ceph orch ps | grep -i mon
mon_host = [v2:172.17.3.60:3300/0,v1:172.17.3.60:6789/0] [v2:172.17.3.29:3300/0,v1:172.17.3.29:6789/0] [v2:172.17.3.53:3300/0,v1:172.17.3.53:6789/0]
[tripleo-admin@controller-0 ~]USD ip -o -4 a | grep 172.17.3 9: vlan30 inet 172.17.3.60/24 brd 172.17.3.255 scope global vlan30\ valid_lft forever preferred_lft forever 9: vlan30 inet 172.17.3.13/32 brd 172.17.3.255 scope global vlan30\ valid_lft forever preferred_lft forever
[tripleo-admin@controller-0 ~]USD grep "172.17.3.60" /etc/os-net-config/config.yaml - ip_netmask: 172.17.3.60/24
sudo os-net-config -c /etc/os-net-config/config.yaml
[controller-0]USD ip -o a | grep 172.17.3.60
sudo os-net-config -c /etc/os-net-config/config.yaml
ip -o a | grep 172.17.3.60
[controller-0]USD ping -c 3 172.17.3.60
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -- ceph orch ls --export mon > USD{SPEC_DIR}/mon
service_type: mon service_id: mon placement: label: mon unmanaged: true
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -m USD{SPEC_DIR}/mon -- ceph orch apply -i /mnt/mon
sudo cephadm shell -- ceph orch daemon rm mon.<target_node> --force
sudo cephadm shell -- ceph orch daemon add mon <target_node>:<ip_address>
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -- ceph orch ls --export mon > USD{SPEC_DIR}/mon
service_type: mon service_id: mon placement: label: mon unmanaged: false
SPEC_DIR=USD{SPEC_DIR:-"USDPWD/ceph_specs"} sudo cephadm shell -m USD{SPEC_DIR}/mon -- ceph orch apply -i /mnt/mon
sudo cephadm shell -- ceph mgr stat
sudo cephadm shell -- ceph mgr fail
sudo cephadm shell -- ceph orch reconfig osd.default_drive_group
ceph -s cluster: id: f6ec3ebe-26f7-56c8-985d-eb974e8e08e3 health: HEALTH_OK
netstat -tulpn | grep 3300
mgr advanced mgr/dashboard/controller-0.ycokob/server_addr 172.17.3.33 mgr advanced mgr/dashboard/controller-1.lmzpuc/server_addr 172.17.3.147 mgr advanced mgr/dashboard/controller-2.xpdgfl/server_addr 172.17.3.138
sudo cephadm shell ceph orch ps | awk '/mgr./ {print USD1}'
ceph config set mgr mgr/dashboard/<>/server_addr/<ip addr>