Chapter 21. Replacing Networker nodes In certain circumstances a Red Hat OpenStack Platform (RHOSP) node with a Networker profile in a high availability cluster might fail. (For more information, see Tagging nodes into profiles in the Director Installation and Usage guide.) In these situations, you must remove the node from the cluster and replace it with a new Networker node that runs the Networking service (neutron) agents. The topics in this section are: Section 21.1, "Preparing to replace network nodes" Section 21.2, "Replacing a Networker node" Section 21.3, "Rescheduling nodes and cleaning up the Networking service" 21.1. Preparing to replace network nodes Replacing a Networker node on a Red Hat OpenStack Platform (RHOSP) overcloud, requires that you perform several preparation steps. Completing all of the required preparation steps helps you to avoid complications during the Networker node replacement process. Prerequisites Your RHOSP deployment is highly available with three or more Networker nodes. Procedure Log in to your undercloud as the stack user. Source the undercloud credentials file: Check the current status of the overcloud stack on the undercloud: The overcloud stack and its subsequent child stacks should have a status of either CREATE_COMPLETE or UPDATE_COMPLETE . Ensure that you have a recent backup image of the undercloud node by running the Relax-and-Recover tool. For more information, see the Backing up and restoring the undercloud and control plane nodes guide. Log in to a Controller node as root. Open an interactive bash shell on the container and check the status of the Galera cluster: Ensure that the Controller nodes are in Master mode. Sample output Log on to the RHOSP director node and check the nova-compute service: The output should show all non-maintenance mode nodes as up. Make sure all undercloud services are running: 21.2. Replacing a Networker node In certain circumstances a Red Hat OpenStack Platform (RHOSP) node with a Networker profile in a high availability cluster might fail. Replacing a Networker node requires running the openstack overcloud deploy command to update the overcloud with a the new node. Prerequisites Your RHOSP deployment is highly available with three or more Networker nodes. The node that you add must be able to connect to the other nodes in the cluster over the network. You have performed the steps described in Section 21.1, "Preparing to replace network nodes" Procedure Log in to your undercloud as the stack user. Source the undercloud credentials file: Example Identify the index of the node to remove: Sample output Set the node into maintenance mode by using the baremetal node maintenance set command. Example Create a JSON file to add the new node to the node pool that contains RHOSP director. Example For more information, see Adding nodes to the overcloud in the Director Installation and Usage guide. Run the openstack overcloud node import command to register the new node. Example After registering the new node, launch the introspection process by using the following commands: Tag the new node with the Networker profile by using the openstack baremetal node set command. Example Create a ~/templates/remove-networker.yaml environment file that defines the index of the node that you intend to remove: Example Create a ~/templates/node-count-networker.yaml environment file and set the total count of Networker nodes in the file. Example Run the openstack overcloud deploy command and include the core heat templates, environment files, and the environment files that you modified. Important The order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence. RHOSP director removes the old Networker node, creates a new one, and updates the overcloud stack. Verification Check the status of the overcloud stack: Verify that the new Networker node is listed, and the old one is removed. Sample output Additional resources Adding nodes to the overcloud in the Director Installation and Usage guide Registering nodes for the overcloud in the Director Installation and Usage guide baremetal node manage in the Command Line Interface Reference overcloud node introspect in the Command Line Interface Reference Environment files in the Advanced Overcloud Customization guide Including environment files in overcloud creation in the Advanced Overcloud Customization guide 21.3. Rescheduling nodes and cleaning up the Networking service As a part of replacing a Red Hat OpenStack Platform (RHOSP) Networker node, remove all Networking service agents on the removed node from the database. Doing so ensures that the Networking service does not identify the agents as out-of-service ("dead"). For ML2/OVS users, removing agents from the removed node enables the DHCP resources to be automatically rescheduled to other Networker nodes. Prerequisites Your RHOSP deployment is highly available with three or more Networker nodes. Procedure Log in to your undercloud as the stack user. Source the overcloud credentials file: Example Verify that the RHOSP Networking service processes exist, and are marked out-of-service ( xxx ) for the overcloud-networker-1 . Sample output for ML2/OVN Sample output for ML2/OVS Capture the UUIDs of the agents registered for overcloud-networker-1 . Delete any remaining overcloud-networker-1 agents from the database. Sample output Additional resources network agent list in the Command Line Interface Reference

Commands:
source ~/stackrc
openstack stack list --nested
pcs status
* Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]: * galera-bundle-0 (ocf::heartbeat:galera): Master controller-0 * galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 * galera-bundle-2 (ocf::heartbeat:galera): Master controller-2
sudo systemctl status tripleo_nova_compute openstack baremetal node list
sudo systemctl -t service
source ~/stackrc
openstack baremetal node list -c UUID -c Name -c "Instance UUID"
+--------------------------------------+------+--------------------------------------+ | UUID | Name | Instance UUID | +--------------------------------------+------+--------------------------------------+ | 36404147-7c8a-41e6-8c72-6af1e339da2a | None | 7bee57cf-4a58-4eaf-b851-f3203f6e5e05 | | 91eb9ac5-7d52-453c-a017-0f2fb289c3cd | None | None | | 75b25e9a-948d-424a-9b3b-0f2fb289c3cd | None | None | | 038727da-6a5c-425f-bd45-16aa2bc4ba91 | None | 763bfec2-9354-466a-ae65-1fdf45d35c61 | | dc2292e6-4056-46e0-8848-165d06fcc948 | None | 2017b481-706f-44e1-852a-57fb03ecef11 | | c7eadcea-e377-4392-9fc3-716f1bd57527 | None | 5f73c7d7-4826-49a5-b6be-0a95c6bdd2f8 | | da3a8d19-8a59-4e9d-923a-29254d688f6d | None | cfefaf60-8311-4bc3-9416-46852e2cb83f | | 807cb6ce-6b94-4cd1-9969-d390650854c7 | None | c07c13e6-a845-4791-9628-c8514585fe27 | | 0c245daa-7817-4ae9-a883-fed2e9c68d6c | None | 844c9a88-713a-4ff1-8737-30858d724593 | | e6499ef7-3db2-4ab4-bfa7-feb44c6591c6 | None | aef7c27a-f0b4-4814-b0ff-d3f792321212 | | 7545385c-bc49-4eb9-b13c-201368ce1c62 | None | c2e40164-c659-4849-a28f-a7b270ed2970 | +--------------------------------------+------+--------------------------------------+
openstack baremetal node maintenance set e6499ef7-3db2-4ab4-bfa7-ef59539bf972
{ "nodes":[ { "mac":[ "dd:dd:dd:dd:dd:dd" ], "cpu":"4", "memory":"6144", "disk":"40", "arch":"x86_64", "pm_type":"ipmi", "pm_user":"admin", "pm_password":"p@55w0rd!", "pm_addr":"192.168.24.207" } ] }
openstack overcloud node import newnode.json
openstack baremetal node manage <node> openstack overcloud node introspect <node> --provide
openstack baremetal node set --property capabilities='profile:networker,boot_option:local' 91eb9ac5-7d52-453c-a017-c0e3d823efd0
parameters: NetworkerRemovalPolicies: [{'resource_list': ['1']}]
parameter_defaults: OvercloudNetworkerFlavor: networker NetworkerCount: 3
openstack overcloud deploy --templates -e <your_environment_files> -e /home/stack/templates/node-count-networker.yaml -e /home/stack/templates/remove-networker.yaml
openstack stack list --nested
openstack server list -c ID -c Name -c Status
+--------------------------------------+------------------------+--------+ | ID | Name | Status | +--------------------------------------+------------------------+--------+ | 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0 | ACTIVE | | 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1 | ACTIVE | | 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2 | ACTIVE | | a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 | ACTIVE | | cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 | ACTIVE | | 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 | ACTIVE | | 844c9a88-713a-4ff1-8737-6410bf551d4f | overcloud-networker-0 | ACTIVE | | c2e40164-c659-4849-a28f-507eb7edb79f | overcloud-networker-2 | ACTIVE | | 425a0828-b42f-43b0-940c-7fb02522753a | overcloud-networker-3 | ACTIVE | +--------------------------------------+------------------------+--------+
source ~/overcloudrc
openstack network agent list -c ID -c Binary -c Host -c Alive | grep overcloud-networker-1
+--------------------------------------+-----------------------+-------+-------------------------------+ | ID | Host | Alive | Binary | +--------------------------------------+-----------------------+-------+-------------------------------+ | 26316f47-4a30-4baf-ba00-d33c9a9e0844 | overcloud-networker-1 | xxx | ovn-controller | +--------------------------------------+-----------------------+-------+-------------------------------+
+--------------------------------------+-----------------------+-------+------------------------+ | ID | Host | Alive | Binary | +--------------------------------------+-----------------------+-------+------------------------+ | 8377-66d75323e466c-b838-1149e10441ee | overcloud-networker-1 | xxx | neutron-metadata-agent | | b55d-797668c336707-a2cf-cba875eeda21 | overcloud-networker-1 | xxx | neutron-l3-agent | | 9dcb-00a9e32ecde42-9458-01cfa9742862 | overcloud-networker-1 | xxx | neutron-ovs-agent | | be83-e4d9329846540-9ae6-1540947b2ffd | overcloud-networker-1 | xxx | neutron-dhcp-agent | +--------------------------------------+-----------------------+-------+------------------------+
AGENT_UUIDS=USD(openstack network agent list -c ID -c Host -c Alive -c Binary -f value | grep overcloud-networker-1 | cut -d\ -f1)
for agent in USDAGENT_UUIDS; do neutron agent-delete USDagent ; done
Deleted agent(s): 26316f47-4a30-4baf-ba00-d33c9a9e0844