Chapter 8. Replacing DistributedComputeHCI nodes During hardware maintenance you may need to scale down, scale up, or replace a DistributedComputeHCI node at an edge site. To replace a DistributedComputeHCI node, remove services from the node you are replacing, scale the number of nodes down, and then follow the procedures for scaling those nodes back up. 8.1. Removing the Compute (nova) service Disable the nova-compute service and delete the relevant network agent. Procedure Delete the node that you are removing from the stack: `openstack overcloud node delete --stack <dcn2> \ <computehci2-1> Delete the network agent on the node you are removing: (central) [stack@site-undercloud-0 ~]USD openstack network agent list | grep dcn2 ... | 17726d1a-e9d1-4e57-b40d-e742be5d073c | Open vSwitch agent | dcn2-computehci2-1.redhat.local | None | XXX | UP | neutron-openvswitch-agent | ... (central) [stack@site-undercloud-0 ~]USD openstack network agent delete 17726d1a-e9d1-4e57-b40d-e742be5d073c 8.2. Removing Red Hat Ceph Storage services To remove the Red Hat Ceph services mon , mgr and osd , you must disable and remove ceph-osd from the cluster services on the node you are removing, then stop and disable the mon , mgr and osd services. Procedure Use SSH to connect to the DistributedComputeHCI node you want to remove and log in as the root user. USD ssh heat-admin@<dcn-computehci-node> USD sudo su - # Identify the OSDs associated with the DistributedComputeHCI node you are removing: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dnc2-computehci2-1 ceph osd tree -c /etc/ceph/dcn2.conf ... -3 0.24399 host dcn2-computehci2-1 1 hdd 0.04880 osd.1 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000 15 hdd 0.04880 osd.15 up 1.00000 1.00000 18 hdd 0.04880 osd.18 up 1.00000 1.00000 ... Disable the OSDs on the relevant Ceph node: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd out 1 7 11 15 18 -c /etc/ceph/dcn2.conf marked out osd.1. marked out osd.7. marked out osd.11. marked out osd.15. marked out osd.18. Wait for Ceph osd rebalancing to finish. Monitor progress with the following command: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph -w -c /etc/ceph/dcn2.conf ... mon.dcn2-computehci2-2 has auth_allow_insecure_global_id_reclaim set to true The rebalancing is complete when you see that auth_allow_insecure_global_id_reclaim is set to true . Stop and disable the OSDs: [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@1 [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@7 [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@11 [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@15 [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@18 [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@1 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@1.service. [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@7 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@7.service. [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@11 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@11.service. [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@15 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@15.service. [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@18 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@18.service. Remove the OSDs from the CRUSH map: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.1 -c /etc/ceph/dcn2.conf removed item id 1 name 'osd.1' from crush map [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.7 -c /etc/ceph/dcn2.conf removed item id 7 name 'osd.7' from crush map [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.11 -c /etc/ceph/dcn2.conf removed item id 11 name 'osd.11' from crush map [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.15 -c /etc/ceph/dcn2.conf removed item id 15 name 'osd.15' from crush map [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.18 -c /etc/ceph/dcn2.conf removed item id 18 name 'osd.18' from crush map Remove the OSD auth keys: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.1 -c /etc/ceph/dcn2.conf updated [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.7 -c /etc/ceph/dcn2.conf updated [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.11 -c /etc/ceph/dcn2.conf updated [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.15 -c /etc/ceph/dcn2.conf updated [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.18 -c /etc/ceph/dcn2.conf updated Remove the OSDs from the cluster: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 1 -c /etc/ceph/dcn2.conf removed osd.1 [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 7 -c /etc/ceph/dcn2.conf removed osd.7 [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 11 -c /etc/ceph/dcn2.conf removed osd.11 [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 15 -c /etc/ceph/dcn2.conf removed osd.15 [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 18 -c /etc/ceph/dcn2.conf removed osd.18 Remove the DistributedComputeHCI node from the CRUSH map: [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush rm dcn2-computehci2-1 -c /etc/ceph/dcn2.conf removed item id -3 name 'dcn2-computehci2-1' from crush map Stop and disable the mon service: [root@dcn2-computehci2-1 ~]# systemctl --type=service | grep ceph ceph-crash@dcn2-computehci2-1.service loaded active running Ceph crash dump collector ceph-mgr@dcn2-computehci2-1.service loaded active running Ceph Manager ceph-mon@dcn2-computehci2-1.service loaded active running Ceph Monitor [root@dcn2-computehci2-1 ~]# systemctl stop ceph-mon@dcn2-computehci2-1 [root@dcn2-computehci2-1 ~]# systemctl disable ceph-mon@dcn2-computehci2-1 Removed /etc/systemd/system/multi-user.target.wants/ceph-mon@dcn2-computehci2-1.service. Use SSH to connect to another node in the same cluster and remove the monitor from the cluster. Note the v1 and v2 entries in the output: [root@dcn2-computehci2-0 ~]# podman exec ceph-mon-dcn2-computehci2-0 ceph mon remove dcn2-computehci2-1 -c /etc/ceph/dcn2.conf removing mon.dcn2-computehci2-1 at [v2:172.23.3.153:3300/0,v1:172.23.3.153:6789/0], there will be 2 monitors On all dcn2 nodes, remove the v1 and v2 monitor entries in /etc/ceph/dcn2.conf that were output in the step, and the node name from the 'mon initial members': Before mon host = [v2:172.23.3.150:3300,v1:172.23.3.150:6789],*[v2:172.23.3.153:3300,v1:172.23.3.153:6789]*,[v2:172.23.3.124:3300,v1:172.23.3.124:6789] + mon initial members = dcn2-computehci2-0,*dcn2-computehci2-1*,dcn2-computehci2-2 After mon host = [v2:172.23.3.150:3300,v1:172.23.3.150:6789],[v2:172.23.3.124:3300,v1:172.23.3.124:6789] + mon initial members = dcn2-computehci2-0,dcn2-computehci2-2 Stop and disable the mgr service: [root@dcn2-computehci2-1 ~]# systemctl --type=service | grep ceph ceph-crash@dcn2-computehci2-1.service loaded active running Ceph crash dump collector ceph-mgr@dcn2-computehci2-1.service loaded active running Ceph Manager [root@dcn2-computehci2-1 ~]# systemctl stop ceph-mgr@dcn2-computehci2-1 [root@dcn2-computehci2-1 ~]# systemctl --type=service | grep ceph ceph-crash@dcn2-computehci2-1.service loaded active running Ceph crash dump collector [root@dcn2-computehci2-1 ~]# systemctl disable ceph-mgr@dcn2-computehci2-1 Removed /etc/systemd/system/multi-user.target.wants/ceph-mgr@dcn2-computehci2-1.service. Verify that the mgr service for the node is removed from the cluster. [root@dcn2-computehci2-0 ~]# podman exec ceph-mon-dcn2-computehci2-0 ceph -s -c /etc/ceph/dcn2.conf cluster: id: b9b53581-d590-41ac-8463-2f50aa985001 health: HEALTH_WARN 3 pools have too many placement groups mons are allowing insecure global_id reclaim services: mon: 2 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0 (age 2h) mgr: dcn2-computehci2-2(active, since 20h), standbys: dcn2-computehci2-0 1 osd: 15 osds: 15 up (since 3h), 15 in (since 3h) data: pools: 3 pools, 384 pgs objects: 32 objects, 88 MiB usage: 16 GiB used, 734 GiB / 750 GiB avail pgs: 384 active+clean 1 The node that the mgr service is removed from is no longer listed when the mgr service is successfully removed. 8.3. Removing the Image service (glance) services Remove image services from a node when you remove it from service. Procedure To disable the Image service services, disable them using systemctl on the node you are removing: [root@dcn2-computehci2-1 ~]# systemctl stop tripleo_glance_api.service [root@dcn2-computehci2-1 ~]# systemctl stop tripleo_glance_api_tls_proxy.service [root@dcn2-computehci2-1 ~]# systemctl disable tripleo_glance_api.service Removed /etc/systemd/system/multi-user.target.wants/tripleo_glance_api.service. [root@dcn2-computehci2-1 ~]# systemctl disable tripleo_glance_api_tls_proxy.service Removed /etc/systemd/system/multi-user.target.wants/tripleo_glance_api_tls_proxy.service. 8.4. Removing the Block Storage (cinder) services You must remove the cinder-volume and etcd services from the DistributedComputeHCI node when you remove it from service. Procedure Identify and disable the cinder-volume service on the node you are removing: (central) [stack@site-undercloud-0 ~]USD openstack volume service list --service cinder-volume | cinder-volume | dcn2-computehci2-1@tripleo_ceph | az-dcn2 | enabled | up | 2022-03-23T17:41:43.000000 | (central) [stack@site-undercloud-0 ~]USD openstack volume service set --disable dcn2-computehci2-1@tripleo_ceph cinder-volume Log on to a different DistributedComputeHCI node in the stack: USD ssh heat-admin@dcn2-computehci2-0 Remove the cinder-volume service associated with the node that you are removing: [root@dcn2-computehci2-0 ~]# podman exec -it cinder_volume cinder-manage service remove cinder-volume dcn2-computehci2-1@tripleo_ceph Service cinder-volume on host dcn2-computehci2-1@tripleo_ceph removed. Stop and disable the tripleo_cinder_volume service on the node that you are removing: 8.5. Delete the DistributedComputeHCI node To preserve the environment, the openstack overcloud node delete command must include all relevant templates and environment files: Procedure Delete the DistributedComputeHCI node USD openstack overcloud node delete / --stack <stack-name> / <node_UUID> If you are going to reuse the node, use ironic to clean the disk. This is required if the node will host Ceph OSDs: openstack baremetal node manage USDUUID openstack baremetal node clean USDUUID --clean-steps '[{"interface":"deploy", "step": "erase_devices_metadata"}]' openstack baremetal provide USDUUID 8.6. Replacing a removed DistributedComputeHCI node 8.6.1. Replacing a removed DistributedComputeHCI node To add new HCI nodes to your DCN deployment, you must redeploy the edge stack with the additional node, perform a ceph export of that stack, and then perform a stack update for the central location. A stack update of the central location adds configurations specific to edge-sites. Prerequisites The node counts are correct in the nodes_data.yaml file of the stack that you want to replace the node in or add a new node to. Procedure You must set the EtcdIntialClusterState parameter to existing in one of the templates called by your deploy script: Redeploy using the deployment script specific to the stack: Export the Red Hat Ceph Storage data from the stack: Replace dcn_ceph_external.yaml with the newly generated dcn2_scale_up_ceph_external.yaml in the deploy script for the central location. Perform a stack update at central: 8.7. Verify the functionality of a replaced DistributedComputeHCI node Ensure the value of the status field is enabled , and that the value of the State field is up : (central) [stack@site-undercloud-0 ~]USD openstack compute service list -c Binary -c Host -c Zone -c Status -c State +----------------+-----------------------------------------+------------+---------+-------+ | Binary | Host | Zone | Status | State | +----------------+-----------------------------------------+------------+---------+-------+ ... | nova-compute | dcn1-compute1-0.redhat.local | az-dcn1 | enabled | up | | nova-compute | dcn1-compute1-1.redhat.local | az-dcn1 | enabled | up | | nova-compute | dcn2-computehciscaleout2-0.redhat.local | az-dcn2 | enabled | up | | nova-compute | dcn2-computehci2-0.redhat.local | az-dcn2 | enabled | up | | nova-compute | dcn2-computescaleout2-0.redhat.local | az-dcn2 | enabled | up | | nova-compute | dcn2-computehci2-2.redhat.local | az-dcn2 | enabled | up | ... Ensure that all network agents are in the up state: (central) [stack@site-undercloud-0 ~]USD openstack network agent list -c "Agent Type" -c Host -c Alive -c State +--------------------+-----------------------------------------+-------+-------+ | Agent Type | Host | Alive | State | +--------------------+-----------------------------------------+-------+-------+ | DHCP agent | dcn3-compute3-1.redhat.local | :-) | UP | | Open vSwitch agent | central-computehci0-1.redhat.local | :-) | UP | | DHCP agent | dcn3-compute3-0.redhat.local | :-) | UP | | DHCP agent | central-controller0-2.redhat.local | :-) | UP | | Open vSwitch agent | dcn3-compute3-1.redhat.local | :-) | UP | | Open vSwitch agent | dcn1-compute1-1.redhat.local | :-) | UP | | Open vSwitch agent | central-computehci0-0.redhat.local | :-) | UP | | DHCP agent | central-controller0-1.redhat.local | :-) | UP | | L3 agent | central-controller0-2.redhat.local | :-) | UP | | Metadata agent | central-controller0-1.redhat.local | :-) | UP | | Open vSwitch agent | dcn2-computescaleout2-0.redhat.local | :-) | UP | | Open vSwitch agent | dcn2-computehci2-5.redhat.local | :-) | UP | | Open vSwitch agent | central-computehci0-2.redhat.local | :-) | UP | | DHCP agent | central-controller0-0.redhat.local | :-) | UP | | Open vSwitch agent | central-controller0-1.redhat.local | :-) | UP | | Open vSwitch agent | dcn2-computehci2-0.redhat.local | :-) | UP | | Open vSwitch agent | dcn1-compute1-0.redhat.local | :-) | UP | ... Verify the status of the Ceph Cluster: Use SSH to connect to the new DistributedComputeHCI node and check the status of the Ceph cluster: [root@dcn2-computehci2-5 ~]# podman exec -it ceph-mon-dcn2-computehci2-5 \ ceph -s -c /etc/ceph/dcn2.conf Verify that both the ceph mon and ceph mgr services exist for the new node: services: mon: 3 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0,dcn2-computehci2-5 (age 3d) mgr: dcn2-computehci2-2(active, since 3d), standbys: dcn2-computehci2-0, dcn2-computehci2-5 osd: 20 osds: 20 up (since 3d), 20 in (since 3d) Verify the status of the ceph osds with 'ceph osd tree'. Ensure all osds for our new node are in STATUS up: [root@dcn2-computehci2-5 ~]# podman exec -it ceph-mon-dcn2-computehci2-5 ceph osd tree -c /etc/ceph/dcn2.conf ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.97595 root default -5 0.24399 host dcn2-computehci2-0 0 hdd 0.04880 osd.0 up 1.00000 1.00000 4 hdd 0.04880 osd.4 up 1.00000 1.00000 8 hdd 0.04880 osd.8 up 1.00000 1.00000 13 hdd 0.04880 osd.13 up 1.00000 1.00000 17 hdd 0.04880 osd.17 up 1.00000 1.00000 -9 0.24399 host dcn2-computehci2-2 3 hdd 0.04880 osd.3 up 1.00000 1.00000 5 hdd 0.04880 osd.5 up 1.00000 1.00000 10 hdd 0.04880 osd.10 up 1.00000 1.00000 14 hdd 0.04880 osd.14 up 1.00000 1.00000 19 hdd 0.04880 osd.19 up 1.00000 1.00000 -3 0.24399 host dcn2-computehci2-5 1 hdd 0.04880 osd.1 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000 15 hdd 0.04880 osd.15 up 1.00000 1.00000 18 hdd 0.04880 osd.18 up 1.00000 1.00000 -7 0.24399 host dcn2-computehciscaleout2-0 2 hdd 0.04880 osd.2 up 1.00000 1.00000 6 hdd 0.04880 osd.6 up 1.00000 1.00000 9 hdd 0.04880 osd.9 up 1.00000 1.00000 12 hdd 0.04880 osd.12 up 1.00000 1.00000 16 hdd 0.04880 osd.16 up 1.00000 1.00000 Verify the cinder-volume service for the new DistributedComputeHCI node is in Status 'enabled' and in State 'up': (central) [stack@site-undercloud-0 ~]USD openstack volume service list --service cinder-volume -c Binary -c Host -c Zone -c Status -c State +---------------+---------------------------------+------------+---------+-------+ | Binary | Host | Zone | Status | State | +---------------+---------------------------------+------------+---------+-------+ | cinder-volume | hostgroup@tripleo_ceph | az-central | enabled | up | | cinder-volume | dcn1-compute1-1@tripleo_ceph | az-dcn1 | enabled | up | | cinder-volume | dcn1-compute1-0@tripleo_ceph | az-dcn1 | enabled | up | | cinder-volume | dcn2-computehci2-0@tripleo_ceph | az-dcn2 | enabled | up | | cinder-volume | dcn2-computehci2-2@tripleo_ceph | az-dcn2 | enabled | up | | cinder-volume | dcn2-computehci2-5@tripleo_ceph | az-dcn2 | enabled | up | +---------------+---------------------------------+------------+---------+-------+ Note If the State of the cinder-volume service is down , then the service has not been started on the node. Use ssh to connect to the new DistributedComputeHCI node and check the status of the Glance services with 'systemctl': [root@dcn2-computehci2-5 ~]# systemctl --type service | grep glance tripleo_glance_api.service loaded active running glance_api container tripleo_glance_api_healthcheck.service loaded activating start start glance_api healthcheck tripleo_glance_api_tls_proxy.service loaded active running glance_api_tls_proxy container 8.8. Troubleshooting DistributedComputeHCI state down If the replacement node was deployed without the EtcdInitialClusterState parameter value set to existing , then the cinder-volume service of the replaced node shows down when you run openstack volume service list . Procedure Log onto the replacement node and check logs for the etcd service. Check that the logs show the etcd service is reporting a cluster ID mismatch in the /var/log/containers/stdouts/etcd.log log file: Set the EtcdInitialClusterState parameter to the value of existing in your deployment templates and rerun the deployment script. Use SSH to connect to the replacement node and run the following commands as root: Recheck the /var/log/containers/stdouts/etcd.log log file to verify that the node successfully joined the cluster: Check the state of the cinder-volume service, and confirm it reads up on the replacement node when you run openstack volume service list .

Commands:
`openstack overcloud node delete --stack <dcn2> <computehci2-1>
(central) [stack@site-undercloud-0 ~]USD openstack network agent list | grep dcn2 ... | 17726d1a-e9d1-4e57-b40d-e742be5d073c | Open vSwitch agent | dcn2-computehci2-1.redhat.local | None | XXX | UP | neutron-openvswitch-agent | ... (central) [stack@site-undercloud-0 ~]USD openstack network agent delete 17726d1a-e9d1-4e57-b40d-e742be5d073c
ssh heat-admin@<dcn-computehci-node> sudo su - #
podman exec ceph-mon-dnc2-computehci2-1 ceph osd tree -c /etc/ceph/dcn2.conf ... -3 0.24399 host dcn2-computehci2-1 1 hdd 0.04880 osd.1 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000 15 hdd 0.04880 osd.15 up 1.00000 1.00000 18 hdd 0.04880 osd.18 up 1.00000 1.00000 ...
podman exec ceph-mon-dcn2-computehci2-1 ceph osd out 1 7 11 15 18 -c /etc/ceph/dcn2.conf marked out osd.1. marked out osd.7. marked out osd.11. marked out osd.15. marked out osd.18.
podman exec ceph-mon-dcn2-computehci2-1 ceph -w -c /etc/ceph/dcn2.conf ... mon.dcn2-computehci2-2 has auth_allow_insecure_global_id_reclaim set to true
systemctl stop ceph-osd@1 systemctl stop ceph-osd@7 systemctl stop ceph-osd@11 systemctl stop ceph-osd@15 systemctl stop ceph-osd@18 systemctl disable ceph-osd@1 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@1.service. systemctl disable ceph-osd@7 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@7.service. systemctl disable ceph-osd@11 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@11.service. systemctl disable ceph-osd@15 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@15.service. systemctl disable ceph-osd@18 Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@18.service.
podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.1 -c /etc/ceph/dcn2.conf removed item id 1 name 'osd.1' from crush map podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.7 -c /etc/ceph/dcn2.conf removed item id 7 name 'osd.7' from crush map podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.11 -c /etc/ceph/dcn2.conf removed item id 11 name 'osd.11' from crush map podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.15 -c /etc/ceph/dcn2.conf removed item id 15 name 'osd.15' from crush map podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.18 -c /etc/ceph/dcn2.conf removed item id 18 name 'osd.18' from crush map
podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.1 -c /etc/ceph/dcn2.conf updated podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.7 -c /etc/ceph/dcn2.conf updated podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.11 -c /etc/ceph/dcn2.conf updated podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.15 -c /etc/ceph/dcn2.conf updated podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.18 -c /etc/ceph/dcn2.conf updated
podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 1 -c /etc/ceph/dcn2.conf removed osd.1 podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 7 -c /etc/ceph/dcn2.conf removed osd.7 podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 11 -c /etc/ceph/dcn2.conf removed osd.11 podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 15 -c /etc/ceph/dcn2.conf removed osd.15 podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 18 -c /etc/ceph/dcn2.conf removed osd.18
podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush rm dcn2-computehci2-1 -c /etc/ceph/dcn2.conf removed item id -3 name 'dcn2-computehci2-1' from crush map
systemctl --type=service | grep ceph ceph-crash@dcn2-computehci2-1.service loaded active running Ceph crash dump collector ceph-mgr@dcn2-computehci2-1.service loaded active running Ceph Manager ceph-mon@dcn2-computehci2-1.service loaded active running Ceph Monitor systemctl stop ceph-mon@dcn2-computehci2-1 systemctl disable ceph-mon@dcn2-computehci2-1 Removed /etc/systemd/system/multi-user.target.wants/ceph-mon@dcn2-computehci2-1.service.
podman exec ceph-mon-dcn2-computehci2-0 ceph mon remove dcn2-computehci2-1 -c /etc/ceph/dcn2.conf removing mon.dcn2-computehci2-1 at [v2:172.23.3.153:3300/0,v1:172.23.3.153:6789/0], there will be 2 monitors
systemctl --type=service | grep ceph ceph-crash@dcn2-computehci2-1.service loaded active running Ceph crash dump collector ceph-mgr@dcn2-computehci2-1.service loaded active running Ceph Manager systemctl stop ceph-mgr@dcn2-computehci2-1 systemctl --type=service | grep ceph ceph-crash@dcn2-computehci2-1.service loaded active running Ceph crash dump collector systemctl disable ceph-mgr@dcn2-computehci2-1 Removed /etc/systemd/system/multi-user.target.wants/ceph-mgr@dcn2-computehci2-1.service.
podman exec ceph-mon-dcn2-computehci2-0 ceph -s -c /etc/ceph/dcn2.conf cluster: id: b9b53581-d590-41ac-8463-2f50aa985001 health: HEALTH_WARN 3 pools have too many placement groups mons are allowing insecure global_id reclaim services: mon: 2 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0 (age 2h) mgr: dcn2-computehci2-2(active, since 20h), standbys: dcn2-computehci2-0 1 osd: 15 osds: 15 up (since 3h), 15 in (since 3h) data: pools: 3 pools, 384 pgs objects: 32 objects, 88 MiB usage: 16 GiB used, 734 GiB / 750 GiB avail pgs: 384 active+clean
systemctl stop tripleo_glance_api.service systemctl stop tripleo_glance_api_tls_proxy.service systemctl disable tripleo_glance_api.service Removed /etc/systemd/system/multi-user.target.wants/tripleo_glance_api.service. systemctl disable tripleo_glance_api_tls_proxy.service Removed /etc/systemd/system/multi-user.target.wants/tripleo_glance_api_tls_proxy.service.
(central) [stack@site-undercloud-0 ~]USD openstack volume service list --service cinder-volume | cinder-volume | dcn2-computehci2-1@tripleo_ceph | az-dcn2 | enabled | up | 2022-03-23T17:41:43.000000 | (central) [stack@site-undercloud-0 ~]USD openstack volume service set --disable dcn2-computehci2-1@tripleo_ceph cinder-volume
ssh heat-admin@dcn2-computehci2-0
podman exec -it cinder_volume cinder-manage service remove cinder-volume dcn2-computehci2-1@tripleo_ceph Service cinder-volume on host dcn2-computehci2-1@tripleo_ceph removed.
systemctl stop tripleo_cinder_volume.service systemctl disable tripleo_cinder_volume.service Removed /etc/systemd/system/multi-user.target.wants/tripleo_cinder_volume.service
openstack overcloud node delete / --stack <stack-name> / <node_UUID>
openstack baremetal node manage USDUUID openstack baremetal node clean USDUUID --clean-steps '[{"interface":"deploy", "step": "erase_devices_metadata"}]' openstack baremetal provide USDUUID
parameter_defaults: EtcdInitialClusterState: existing
(undercloud) [stack@site-undercloud-0 ~]USD ./overcloud_deploy_dcn2.sh ... Overcloud Deployed without error
(undercloud) [stack@site-undercloud-0 ~]USD sudo -E openstack overcloud export ceph --stack dcn1,dcn2 --config-download-dir /var/lib/mistral --output-file ~/central/dcn2_scale_up_ceph_external.yaml
(undercloud) [stack@site-undercloud-0 ~]USD ./overcloud_deploy.sh Overcloud Deployed without error
(central) [stack@site-undercloud-0 ~]USD openstack compute service list -c Binary -c Host -c Zone -c Status -c State +----------------+-----------------------------------------+------------+---------+-------+ | Binary | Host | Zone | Status | State | +----------------+-----------------------------------------+------------+---------+-------+ | nova-compute | dcn1-compute1-0.redhat.local | az-dcn1 | enabled | up | | nova-compute | dcn1-compute1-1.redhat.local | az-dcn1 | enabled | up | | nova-compute | dcn2-computehciscaleout2-0.redhat.local | az-dcn2 | enabled | up | | nova-compute | dcn2-computehci2-0.redhat.local | az-dcn2 | enabled | up | | nova-compute | dcn2-computescaleout2-0.redhat.local | az-dcn2 | enabled | up | | nova-compute | dcn2-computehci2-2.redhat.local | az-dcn2 | enabled | up |
(central) [stack@site-undercloud-0 ~]USD openstack network agent list -c "Agent Type" -c Host -c Alive -c State +--------------------+-----------------------------------------+-------+-------+ | Agent Type | Host | Alive | State | +--------------------+-----------------------------------------+-------+-------+ | DHCP agent | dcn3-compute3-1.redhat.local | :-) | UP | | Open vSwitch agent | central-computehci0-1.redhat.local | :-) | UP | | DHCP agent | dcn3-compute3-0.redhat.local | :-) | UP | | DHCP agent | central-controller0-2.redhat.local | :-) | UP | | Open vSwitch agent | dcn3-compute3-1.redhat.local | :-) | UP | | Open vSwitch agent | dcn1-compute1-1.redhat.local | :-) | UP | | Open vSwitch agent | central-computehci0-0.redhat.local | :-) | UP | | DHCP agent | central-controller0-1.redhat.local | :-) | UP | | L3 agent | central-controller0-2.redhat.local | :-) | UP | | Metadata agent | central-controller0-1.redhat.local | :-) | UP | | Open vSwitch agent | dcn2-computescaleout2-0.redhat.local | :-) | UP | | Open vSwitch agent | dcn2-computehci2-5.redhat.local | :-) | UP | | Open vSwitch agent | central-computehci0-2.redhat.local | :-) | UP | | DHCP agent | central-controller0-0.redhat.local | :-) | UP | | Open vSwitch agent | central-controller0-1.redhat.local | :-) | UP | | Open vSwitch agent | dcn2-computehci2-0.redhat.local | :-) | UP | | Open vSwitch agent | dcn1-compute1-0.redhat.local | :-) | UP |
podman exec -it ceph-mon-dcn2-computehci2-5 ceph -s -c /etc/ceph/dcn2.conf
services: mon: 3 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0,dcn2-computehci2-5 (age 3d) mgr: dcn2-computehci2-2(active, since 3d), standbys: dcn2-computehci2-0, dcn2-computehci2-5 osd: 20 osds: 20 up (since 3d), 20 in (since 3d)
podman exec -it ceph-mon-dcn2-computehci2-5 ceph osd tree -c /etc/ceph/dcn2.conf ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.97595 root default -5 0.24399 host dcn2-computehci2-0 0 hdd 0.04880 osd.0 up 1.00000 1.00000 4 hdd 0.04880 osd.4 up 1.00000 1.00000 8 hdd 0.04880 osd.8 up 1.00000 1.00000 13 hdd 0.04880 osd.13 up 1.00000 1.00000 17 hdd 0.04880 osd.17 up 1.00000 1.00000 -9 0.24399 host dcn2-computehci2-2 3 hdd 0.04880 osd.3 up 1.00000 1.00000 5 hdd 0.04880 osd.5 up 1.00000 1.00000 10 hdd 0.04880 osd.10 up 1.00000 1.00000 14 hdd 0.04880 osd.14 up 1.00000 1.00000 19 hdd 0.04880 osd.19 up 1.00000 1.00000 -3 0.24399 host dcn2-computehci2-5 1 hdd 0.04880 osd.1 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000 15 hdd 0.04880 osd.15 up 1.00000 1.00000 18 hdd 0.04880 osd.18 up 1.00000 1.00000 -7 0.24399 host dcn2-computehciscaleout2-0 2 hdd 0.04880 osd.2 up 1.00000 1.00000 6 hdd 0.04880 osd.6 up 1.00000 1.00000 9 hdd 0.04880 osd.9 up 1.00000 1.00000 12 hdd 0.04880 osd.12 up 1.00000 1.00000 16 hdd 0.04880 osd.16 up 1.00000 1.00000
(central) [stack@site-undercloud-0 ~]USD openstack volume service list --service cinder-volume -c Binary -c Host -c Zone -c Status -c State +---------------+---------------------------------+------------+---------+-------+ | Binary | Host | Zone | Status | State | +---------------+---------------------------------+------------+---------+-------+ | cinder-volume | hostgroup@tripleo_ceph | az-central | enabled | up | | cinder-volume | dcn1-compute1-1@tripleo_ceph | az-dcn1 | enabled | up | | cinder-volume | dcn1-compute1-0@tripleo_ceph | az-dcn1 | enabled | up | | cinder-volume | dcn2-computehci2-0@tripleo_ceph | az-dcn2 | enabled | up | | cinder-volume | dcn2-computehci2-2@tripleo_ceph | az-dcn2 | enabled | up | | cinder-volume | dcn2-computehci2-5@tripleo_ceph | az-dcn2 | enabled | up | +---------------+---------------------------------+------------+---------+-------+
systemctl --type service | grep glance tripleo_glance_api.service loaded active running glance_api container tripleo_glance_api_healthcheck.service loaded activating start start glance_api healthcheck tripleo_glance_api_tls_proxy.service loaded active running glance_api_tls_proxy container
2022-04-06T18:00:11.834104130+00:00 stderr F 2022-04-06 18:00:11.834045 E | rafthttp: request cluster ID mismatch (got 654f4cf0e2cfb9fd want 918b459b36fe2c0c)
systemctl stop tripleo_etcd rm -rf /var/lib/etcd/* systemctl start tripleo_etcd
2022-04-06T18:24:22.130059875+00:00 stderr F 2022-04-06 18:24:22.129395 I | etcdserver/membership: added member 96f61470cd1839e5 [https://dcn2-computehci2-4.internalapi.redhat.local:2380] to cluster 654f4cf0e2cfb9fd