Chapter 6. Migrating the Object Storage service to Red Hat OpenStack Services on OpenShift nodes If you are using the Red Hat OpenStack Platform Object Storage service (swift) as an Object Storage service, you must migrate your Object Storage service to Red Hat OpenStack Services on OpenShift nodes. If you are using the Object Storage API of the Ceph Object Gateway (RGW), you can skip this chapter. The data migration happens replica by replica. For example, if you have 3 replicas, move them one at a time to ensure that the other 2 replicas are still operational, which enables you to continue to use the Object Storage service during the migration. Note Data migration to the new deployment is a long-running process that executes mostly in the background. The Object Storage service replicators move data from old to new nodes, which might take a long time depending on the amount of storage used. To reduce downtime, you can use the old nodes if they are running and continue with adopting other services while waiting for the migration to complete. Performance might be degraded due to the amount of replication traffic in the network. 6.1. Migrating the Object Storage service data from RHOSP to RHOSO nodes The Object Storage service (swift) migration involves the following steps: Add new nodes to the Object Storage service rings. Set weights of existing nodes to 0. Rebalance rings by moving one replica. Copy rings to old nodes and restart services. Check replication status and repeat the two steps until the old nodes are drained. Remove the old nodes from the rings. Prerequisites Adopt the Object Storage service. For more information, see Adopting the Object Storage service . For DNS servers, ensure that all existing nodes are able to resolve the hostnames of the Red Hat OpenShift Container Platform (RHOCP) pods, for example, by using the external IP of the DNSMasq service as the nameserver in /etc/resolv.conf : Track the current status of the replication by using the swift-dispersion tool: The command might need a few minutes to complete. It creates 0-byte objects that are distributed across the Object Storage service deployment, and you can use the swift-dispersion-report afterward to show the current replication status: The output of the swift-dispersion-report command looks similar to the following: Procedure Add new nodes by scaling up the SwiftStorage resource from 0 to 3: This command creates three storage instances on the Red Hat OpenShift Container Platform (RHOCP) cluster that use Persistent Volume Claims. Wait until all three pods are running: From the current rings, get the storage management IP addresses of the nodes to drain: The output looks similar to the following: Drain the old nodes. In the following example, the old node 172.20.0.100 is drained: Depending on your deployment, you might have more nodes to include in the command. Copy and apply the updated rings to the original nodes. Run the ssh commands for your existing nodes that store the Object Storage service data: Track the replication progress by using the swift-dispersion-report tool: The output shows less than 100% of copies found. Repeat the command until all container and object copies are found: Move the replica to the new nodes by rebalancing and distributing the rings: Monitor the swift-dispersion-report output again, wait until all copies are found, and then repeat this step until all your replicas are moved to the new nodes. Remove the nodes from the rings: Note Even if all replicas are on the new nodes and the swift-dispersion-report command reports 100% of the copies found, there might still be data on the old nodes. The replicators remove this data, but it might take more time. Verification Check the disk usage of all disks in the cluster: Confirm that there are no more \*.db or *.data files in the /srv/node directory on the nodes: 6.2. Troubleshooting the Object Storage service migration You can troubleshoot issues with the Object Storage service (swift) migration. If the replication is not working and the swift-dispersion-report is not back to 100% availability, check the replicator progress to help you debug: The following shows an example of the output: You can also check the ring consistency and replicator status: The output might show a md5 mismatch until approximately 2 minutes after pushing the new rings. After the 2 minutes, the output looks similar to the following example:

Commands:
oc get service dnsmasq-dns -o jsonpath="{.status.loadBalancer.ingress[0].ip}" | USDCONTROLLER1_SSH sudo tee /etc/resolv.conf
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-dispersion-populate'
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-dispersion-report'
Queried 1024 containers for dispersion reporting, 5s, 0 retries 100.00% of container copies found (3072 of 3072) Sample represents 100.00% of the container partition space Queried 1024 objects for dispersion reporting, 4s, 0 retries There were 1024 partitions missing 0 copies. 100.00% of object copies found (3072 of 3072) Sample represents 100.00% of the object partition space
oc patch openstackcontrolplane openstack --type=merge -p='{"spec":{"swift":{"template":{"swiftStorage":{"replicas": 3}}}}}'
oc wait pods --for condition=Ready -l component=swift-storage
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-ring-builder object.builder' | tail -n +7 | awk '{print USD4}' | sort -u
172.20.0.100:6200 swift-storage-0.swift-storage.openstack.svc:6200 swift-storage-1.swift-storage.openstack.svc:6200 swift-storage-2.swift-storage.openstack.svc:6200
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c ' swift-ring-tool get swift-ring-tool drain 172.20.0.100 swift-ring-tool rebalance swift-ring-tool push'
oc extract --confirm cm/swift-ring-files USDCONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz USDCONTROLLER1_SSH "systemctl restart tripleo_swift_*"
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c "swift-ring-tool get && swift-dispersion-report"
Queried 1024 containers for dispersion reporting, 6s, 0 retries There were 5 partitions missing 1 copy. 99.84% of container copies found (3067 of 3072) Sample represents 100.00% of the container partition space Queried 1024 objects for dispersion reporting, 7s, 0 retries There were 739 partitions missing 1 copy. There were 285 partitions missing 0 copies. 75.94% of object copies found (2333 of 3072) Sample represents 100.00% of the object partition space
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c ' swift-ring-tool get swift-ring-tool rebalance swift-ring-tool push' oc extract --confirm cm/swift-ring-files USDCONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz USDCONTROLLER1_SSH "systemctl restart tripleo_swift_*"
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c ' swift-ring-tool get swift-ring-tool remove 172.20.0.100 swift-ring-tool rebalance swift-ring-tool push'
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -d'
USDCONTROLLER1_SSH "find /srv/node/ -type f -name '*.db' -o -name '*.data' | wc -l"
CONTROLLER1_SSH tail /var/log/containers/swift/swift.log | grep object-server
Mar 14 06:05:30 standalone object-server[652216]: <f+++++++++ 4e2/9cbea55c47e243994b0b10d8957184e2/1710395823.58025.data Mar 14 06:05:30 standalone object-server[652216]: Successful rsync of /srv/node/vdd/objects/626/4e2 to swift-storage-1.swift-storage.openstack.svc::object/d1/objects/626 (0.094) Mar 14 06:05:30 standalone object-server[652216]: Removing partition: /srv/node/vdd/objects/626 Mar 14 06:05:31 standalone object-server[652216]: <f+++++++++ 85f/cf53b5a048e5b19049e05a548cde185f/1710395796.70868.data Mar 14 06:05:31 standalone object-server[652216]: Successful rsync of /srv/node/vdb/objects/829/85f to swift-storage-2.swift-storage.openstack.svc::object/d1/objects/829 (0.095) Mar 14 06:05:31 standalone object-server[652216]: Removing partition: /srv/node/vdb/objects/829
oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -r --md5'
[...] Oldest completion was 2024-03-14 16:53:27 (3 minutes ago) by 172.20.0.100:6000. Most recent completion was 2024-03-14 16:56:38 (12 seconds ago) by swift-storage-0.swift-storage.openstack.svc:6200. =============================================================================== [2024-03-14 16:56:50] Checking ring md5sums 4/4 hosts matched, 0 error[s] while checking hosts. [...]