Chapter 5. Adopting the data plane Adopting the Red Hat OpenStack Services on OpenShift (RHOSO) data plane involves the following steps: Stop any remaining services on the Red Hat OpenStack Platform (RHOSP) 17.1 control plane. Deploy the required custom resources. Perform a fast-forward upgrade on Compute services from RHOSP 17.1 to RHOSO 18.0. If applicable, adopt Networker nodes to the RHOSO data plane. Warning After the RHOSO control plane manages the newly deployed data plane, you must not re-enable services on the RHOSP 17.1 control plane and data plane. If you re-enable services, workloads are managed by two control planes or two data planes, resulting in data corruption, loss of control of existing workloads, inability to start new workloads, or other issues. 5.1. Stopping infrastructure management and Compute services You must stop cloud Controller nodes, database nodes, and messaging nodes on the Red Hat OpenStack Platform 17.1 control plane. Do not stop nodes that are running the Compute, Storage, or Networker roles on the control plane. The following procedure applies to a single node standalone director deployment. You must remove conflicting repositories and packages from your Compute hosts, so that you can install libvirt packages when these hosts are adopted as data plane nodes, where modular libvirt daemons are no longer running in podman containers. Prerequisites Define the shell variables. Replace the following example values with values that apply to your environment: Replace ["standalone.localdomain"]="192.168.122.100" with the name and IP address of the Compute node. Replace <path_to_SSH_key> with the path to your SSH key. Procedure Remove the conflicting repositories and packages from all Compute hosts: 5.2. Adopting Compute services to the RHOSO data plane Adopt your Compute (nova) services to the Red Hat OpenStack Services on OpenShift (RHOSO) data plane. Prerequisites You have stopped the remaining control plane nodes, repositories, and packages on the Compute service (nova) hosts. For more information, see Stopping infrastructure management and Compute services . You have configured the Ceph back end for the NovaLibvirt service. For more information, see Configuring a Ceph back end . You have configured IP Address Management (IPAM): If neutron-sriov-nic-agent is running on your Compute service nodes, ensure that the physical device mappings match the values that are defined in the OpenStackDataPlaneNodeSet custom resource (CR). For more information, see Pulling the configuration from a director deployment . You have defined the shell variables to run the script that runs the fast-forward upgrade: Replace ["standalone.localdomain"]="192.168.122.100" with the name and IP address of the Compute service node. Note Do not set a value for the CEPH_FSID parameter if the local storage back end is configured by the Compute service for libvirt. The storage back end must match the source cloud storage back end. You cannot change the storage back end during adoption. Procedure Create an SSH authentication secret for the data plane nodes: Replace <path_to_SSH_key> with the path to your SSH key. Generate an ssh key-pair nova-migration-ssh-key secret: If you use a local storage back end for libvirt, create a nova-compute-extra-config service to remove pre-fast-forward workarounds and configure Compute services to use a local storage back end: USD oc apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 19-nova-compute-cell1-workarounds.conf: | [workarounds] disable_compute_service_check_for_ffu=true EOF Note The secret nova-cell<X>-compute-config auto-generates for each cell<X> . You must specify values for the nova-cell<X>-compute-config and nova-migration-ssh-key parameters for each custom OpenStackDataPlaneService CR that is related to the Compute service. If TLS Everywhere is enabled, append the following content to the OpenStackDataPlaneService CR: tlsCerts: contents: - dnsnames - ips networks: - ctlplane issuer: osp-rootca-issuer-internal caCerts: combined-ca-bundle edpmServiceType: nova If you use a Ceph back end for libvirt, create a nova-compute-extra-config service to remove pre-fast-forward upgrade workarounds and configure Compute services to use a Ceph back end: USD oc apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 19-nova-compute-cell1-workarounds.conf: | [workarounds] disable_compute_service_check_for_ffu=true 03-ceph-nova.conf: | [libvirt] images_type=rbd images_rbd_pool=vms images_rbd_ceph_conf=/etc/ceph/ceph.conf images_rbd_glance_store_name=default_backend images_rbd_glance_copy_poll_interval=15 images_rbd_glance_copy_timeout=600 rbd_user=openstack rbd_secret_uuid=USDCEPH_FSID EOF The resources in the ConfigMap contain cell-specific configurations. Create a secret for the subscription manager: Replace <subscription_manager_username> with the applicable user name. Replace <subscription_manager_password> with the applicable password. Create a secret for the Red Hat registry: Replace <registry_username> with the applicable user name. Replace <registry_password> with the applicable password. Deploy the OpenStackDataPlaneNodeSet CR: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack spec: tlsEnabled: false 1 networkAttachments: - ctlplane preProvisioned: true services: - redhat - bootstrap - download-cache - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - reboot-os - install-certs - libvirt - nova - ovn - neutron-metadata - telemetry env: - name: ANSIBLE_CALLBACKS_ENABLED value: "profile_tasks" - name: ANSIBLE_FORCE_COLOR value: "True" nodes: standalone: hostName: standalone 2 ansible: ansibleHost: USD{computes[standalone.localdomain]} networks: - defaultRoute: true fixedIP: USD{computes[standalone.localdomain]} name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: tenant subnetName: subnet1 nodeTemplate: ansibleSSHPrivateKeySecret: dataplane-adoption-secret ansible: ansibleUser: root ansibleVarsFrom: - secretRef: name: subscription-manager - secretRef: name: redhat-registry ansibleVars: rhc_release: 9.2 rhc_repositories: - {name: "*", state: disabled} - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled} - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled} - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled} - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled} - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled} - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled} edpm_bootstrap_release_version_package: [] # edpm_network_config # Default nic config template for a EDPM node # These vars are edpm_network_config role vars edpm_network_config_template: | --- {% set mtu_list = [ctlplane_mtu] %} {% for network in nodeset_networks %} {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic1 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in nodeset_networks %} - type: vlan mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} addresses: - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }} {% endfor %} edpm_network_config_hide_sensitive_logs: false # # These vars are for the network config templates themselves and are # considered EDPM network defaults. neutron_physical_bridge_name: br-ctlplane neutron_public_interface_name: eth0 # edpm_nodes_validation edpm_nodes_validation_validate_controllers_icmp: false edpm_nodes_validation_validate_gateway_icmp: false # edpm ovn-controller configuration edpm_ovn_bridge_mappings: <bridge_mappings> 3 edpm_ovn_bridge: br-int edpm_ovn_encap_type: geneve ovn_monitor_all: true edpm_ovn_remote_probe_interval: 60000 edpm_ovn_ofctrl_wait_before_clear: 8000 timesync_ntp_servers: - hostname: clock.redhat.com - hostname: clock2.redhat.com edpm_bootstrap_command: | # FIXME: perform dnf upgrade for other packages in EDPM ansible # here we only ensuring that decontainerized libvirt can start dnf -y upgrade openstack-selinux rm -f /run/virtlogd.pid gather_facts: false # edpm firewall, change the allowed CIDR if needed edpm_sshd_configure_firewall: true edpm_sshd_allowed_ranges: ['192.168.122.0/24'] # Do not attempt OVS major upgrades here edpm_ovs_packages: - openvswitch3.3 EOF 1 If TLS Everywhere is enabled, change spec:tlsEnabled to true . 2 If your deployment has a custom DNS Domain, modify the spec:nodes:[NODE NAME]:hostName to use fqdn for the node. 3 Replace <bridge_mappings> with the value of the bridge mappings in your configuration, for example, "datacentre:br-ctlplane" . Ensure that you use the same ovn-controller settings in the OpenStackDataPlaneNodeSet CR that you used in the Compute service nodes before adoption. This configuration is stored in the external_ids column in the Open_vSwitch table in the Open vSwitch database: Replace <bridge_mappings> with the value of the bridge mappings in your configuration, for example, "datacentre:br-ctlplane" . If you use a Ceph back end for Block Storage service (cinder), prepare the adopted data plane workloads: USD oc patch osdpns/openstack --type=merge --patch " spec: services: - redhat - bootstrap - download-cache - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - reboot-os - ceph-client - install-certs - ovn - neutron-metadata - libvirt - nova - telemetry nodeTemplate: extraMounts: - extraVolType: Ceph volumes: - name: ceph secret: secretName: ceph-conf-files mounts: - name: ceph mountPath: "/etc/ceph" readOnly: true " Note Ensure that you use the same list of services from the original OpenStackDataPlaneNodeSet CR, except for the inserted ceph-client service. Optional: Enable neutron-sriov-nic-agent in the OpenStackDataPlaneNodeSet CR: USD oc patch openstackdataplanenodeset openstack --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-sriov" }, { "op": "add", "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_physical_device_mappings", "value": "dummy_sriov_net:dummy-dev" }, { "op": "add", "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_bandwidths", "value": "dummy-dev:40000000:40000000" }, { "op": "add", "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_hypervisors", "value": "dummy-dev:standalone.localdomain" } ]' Optional: Enable neutron-dhcp in the OpenStackDataPlaneNodeSet CR: USD oc patch openstackdataplanenodeset openstack --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-dhcp" }]' Note To use neutron-dhcp with OVN for the Bare Metal Provisioning service (ironic), you must set the disable_ovn_dhcp_for_baremetal_ports configuration option for the Networking service (neutron) to true . You can set this configuration in the NeutronAPI spec: .. spec: serviceUser: neutron ... customServiceConfig: | [DEFAULT] dhcp_agent_notification = True [ovn] disable_ovn_dhcp_for_baremetal_ports = true Run the pre-adoption validation: Create the validation service: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: name: pre-adoption-validation spec: playbook: osp.edpm.pre_adoption_validation EOF Create a OpenStackDataPlaneDeployment CR that runs only the validation: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-pre-adoption spec: nodeSets: - openstack servicesOverride: - pre-adoption-validation EOF When the validation is finished, confirm that the status of the Ansible EE pods is Completed : Wait for the deployment to reach the Ready status: Important If any openstack-pre-adoption validations fail, you must reference the Ansible logs to determine which ones were unsuccessful, and then try the following troubleshooting options: If the hostname validation failed, check that the hostname of the data plane node is correctly listed in the OpenStackDataPlaneNodeSet CR. If the kernel argument check failed, ensure that the kernel argument configuration in the edpm_kernel_args and edpm_kernel_hugepages variables in the OpenStackDataPlaneNodeSet CR is the same as the kernel argument configuration that you used in the Red Hat OpenStack Platform (RHOSP) 17.1 node. If the tuned profile check failed, ensure that the edpm_tuned_profile variable in the OpenStackDataPlaneNodeSet CR is configured to use the same profile as the one set on the RHOSP 17.1 node. Remove the remaining director services: Create an OpenStackDataPlaneService CR to clean up the data plane services you are adopting: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: name: tripleo-cleanup spec: playbook: osp.edpm.tripleo_cleanup EOF Create the OpenStackDataPlaneDeployment CR to run the clean-up: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: tripleo-cleanup spec: nodeSets: - openstack servicesOverride: - tripleo-cleanup EOF When the clean-up is finished, deploy the OpenStackDataPlaneDeployment CR: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack spec: nodeSets: - openstack EOF Note If you have other node sets to deploy, such as Networker nodes, you can add them in the nodeSets list in this step, or create separate OpenStackDataPlaneDeployment CRs later. You cannot add new node sets to an OpenStackDataPlaneDeployment CR after deployment. Verification Confirm that all the Ansible EE pods reach a Completed status: Wait for the data plane node set to reach the Ready status: Verify that the Networking service (neutron) agents are running: steps You must perform a fast-forward upgrade on your Compute services. For more information, see Performing a fast-forward upgrade on Compute services . 5.3. Performing a fast-forward upgrade on Compute services You must upgrade the Compute services from Red Hat OpenStack Platform 17.1 to Red Hat OpenStack Services on OpenShift (RHOSO) 18.0 on the control plane and data plane by completing the following tasks: Update the cell1 Compute data plane services version. Remove pre-fast-forward upgrade workarounds from the Compute control plane services and Compute data plane services. Run Compute database online migrations to update live data. Procedure Wait for cell1 Compute data plane services version to update: Note The query returns an empty result when the update is completed. No downtime is expected for virtual machine workloads. Review any errors in the nova Compute agent logs on the data plane, and the nova-conductor journal records on the control plane. Patch the OpenStackControlPlane CR to remove the pre-fast-forward upgrade workarounds from the Compute control plane services: USD oc patch openstackcontrolplane openstack -n openstack --type=merge --patch ' spec: nova: template: cellTemplates: cell0: conductorServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false cell1: metadataServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false conductorServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false apiServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false metadataServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false schedulerServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false ' Wait until the Compute control plane services CRs are ready: Complete the steps in Adopting Compute services to the RHOSO data plane . Remove the pre-fast-forward upgrade workarounds from the Compute data plane services: USD oc apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 20-nova-compute-cell1-workarounds.conf: | [workarounds] disable_compute_service_check_for_ffu=false --- apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-nova-compute-ffu namespace: openstack spec: nodeSets: - openstack servicesOverride: - nova EOF Note The service included in the servicesOverride key must match the name of the service that you included in the OpenStackDataPlaneNodeSet CR. For example, if you use a custom service called nova-custom , ensure that you add it to the servicesOverride key. Wait for the Compute data plane services to be ready: Run Compute database online migrations to complete the fast-forward upgrade: Verification Discover the Compute hosts in the cell: Verify if the existing test VM instance is running: Verify if the Compute services can stop the existing test VM instance: Verify if the Compute services can start the existing test VM instance: Note After the data plane adoption, the Compute hosts continue to run Red Hat Enterprise Linux (RHEL) 9.2. To take advantage of RHEL 9.4, perform a minor update procedure after finishing the adoption procedure. 5.4. Adopting Networker services to the RHOSO data plane Adopt the Networker nodes in your existing Red Hat OpenStack Platform deployment to the Red Hat OpenStack Services on OpenShift (RHOSO) data plane. You decide which services you want to run on the Networker nodes, and create a separate OpenStackDataPlaneNodeSet custom resource (CR) for the Networker nodes. You might also decide to implement the following options if they apply to your environment: Depending on your topology, you might need to run the neutron-metadata service on the nodes, specifically when you want to serve metadata to SR-IOV ports that are hosted on Compute nodes. If you want to continue running OVN gateway services on Networker nodes, keep ovn service in the list to deploy. Optional: You can run the neutron-dhcp service on your Networker nodes instead of your Compute nodes. You might not need to use neutron-dhcp with OVN, unless your deployment uses DHCP relays, or advanced DHCP options that are supported by dnsmasq but not by the OVN DHCP implementation. Prerequisites Define the shell variable. The following value is an example from a single node standalone director deployment: Replace ["standalone.localdomain"]="192.168.122.100" with the name and IP address of the Networker node. Procedure Deploy the OpenStackDataPlaneNodeSet CR for your Networker nodes: Note You can reuse most of the nodeTemplate section from the OpenStackDataPlaneNodeSet CR that is designated for your Compute nodes. You can omit some of the variables because of the limited set of services that are running on Networker nodes. USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-networker spec: tlsEnabled: false 1 networkAttachments: - ctlplane preProvisioned: true services: - bootstrap - download-cache - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - install-certs - ovn env: - name: ANSIBLE_CALLBACKS_ENABLED value: "profile_tasks" - name: ANSIBLE_FORCE_COLOR value: "True" nodes: standalone: hostName: standalone ansible: ansibleHost: USD{networkers[standalone.localdomain]} networks: - defaultRoute: true fixedIP: USD{networkers[standalone.localdomain]} name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: tenant subnetName: subnet1 nodeTemplate: ansibleSSHPrivateKeySecret: dataplane-adoption-secret ansible: ansibleUser: root ansibleVarsFrom: - prefix: subscription_manager_ secretRef: name: subscription-manager - secretRef: name: redhat-registry ansibleVars: edpm_bootstrap_release_version_package: [] # edpm_network_config # Default nic config template for a EDPM node # These vars are edpm_network_config role vars edpm_network_config_template: | --- {% set mtu_list = [ctlplane_mtu] %} {% for network in nodeset_networks %} {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic1 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in nodeset_networks %} - type: vlan mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} addresses: - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }} {% endfor %} edpm_network_config_hide_sensitive_logs: false # # These vars are for the network config templates themselves and are # considered EDPM network defaults. neutron_physical_bridge_name: br-ctlplane neutron_public_interface_name: eth0 # edpm_nodes_validation edpm_nodes_validation_validate_controllers_icmp: false edpm_nodes_validation_validate_gateway_icmp: false # edpm ovn-controller configuration edpm_ovn_bridge_mappings: <bridge_mappings> 2 edpm_ovn_bridge: br-int edpm_ovn_encap_type: geneve ovn_monitor_all: true edpm_ovn_remote_probe_interval: 60000 edpm_ovn_ofctrl_wait_before_clear: 8000 # serve as a OVN gateway edpm_enable_chassis_gw: true 3 timesync_ntp_servers: - hostname: clock.redhat.com - hostname: clock2.redhat.com edpm_bootstrap_command: | subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }} subscription-manager release --set=9.2 subscription-manager repos --disable=* subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms gather_facts: false enable_debug: false # edpm firewall, change the allowed CIDR if needed edpm_sshd_configure_firewall: true edpm_sshd_allowed_ranges: ['192.168.122.0/24'] # SELinux module edpm_selinux_mode: enforcing # Do not attempt OVS major upgrades here edpm_ovs_packages: - openvswitch3.1 EOF 1 If TLS Everywhere is enabled, change spec:tlsEnabled to true . 2 Set to the same values that you used in your Red Hat OpenStack Platform 17.1 deployment. 3 Set to true to run ovn-controller in gateway mode. Ensure that you use the same ovn-controller settings in the OpenStackDataPlaneNodeSet CR that you used in the Networker nodes before adoption. This configuration is stored in the external_ids column in the Open_vSwitch table in the Open vSwitch database: Replace <bridge_mappings> with the value of the bridge mappings in your configuration, for example, "datacentre:br-ctlplane" . Optional: Enable neutron-metadata in the OpenStackDataPlaneNodeSet CR: USD oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-metadata" }]' Replace <networker_CR_name> with the name of the CR that you deployed for your Networker nodes, for example, openstack-networker . Optional: Enable neutron-dhcp in the OpenStackDataPlaneNodeSet CR: USD oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-dhcp" }]' Run the pre-adoption-validation service for Networker nodes: Create a OpenStackDataPlaneDeployment CR that runs only the validation: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-pre-adoption-networker spec: nodeSets: - openstack-networker servicesOverride: - pre-adoption-validation EOF When the validation is finished, confirm that the status of the Ansible EE pods is Completed : Wait for the deployment to reach the Ready status: Deploy the OpenStackDataPlaneDeployment CR for Networker nodes: USD oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-networker spec: nodeSets: - openstack-networker EOF Note Alternatively, you can include the Networker node set in the nodeSets list before you deploy the main OpenStackDataPlaneDeployment CR. You cannot add new node sets to the OpenStackDataPlaneDeployment CR after deployment. Verification Confirm that all the Ansible EE pods reach a Completed status: Wait for the data plane node set to reach the Ready status: Replace <networker_CR_name> with the name of the CR that you deployed for your Networker nodes, for example, openstack-networker . Verify that the Networking service (neutron) agents are running. The list of agents varies depending on the services you enabled:

Commands:
EDPM_PRIVATEKEY_PATH="<path_to_SSH_key>" declare -A computes computes=( ["standalone.localdomain"]="192.168.122.100" # )
PacemakerResourcesToStop=( "galera-bundle" "haproxy-bundle" "rabbitmq-bundle") echo "Stopping pacemaker services" for i in {1..3}; do SSH_CMD=CONTROLLERUSD{i}_SSH if [ ! -z "USD{!SSH_CMD}" ]; then echo "Using controller USDi to run pacemaker commands" for resource in USD{PacemakerResourcesToStop[*]}; do if USD{!SSH_CMD} sudo pcs resource config USDresource; then USD{!SSH_CMD} sudo pcs resource disable USDresource fi done break fi done
oc apply -f - <<EOF apiVersion: network.openstack.org/v1beta1 kind: NetConfig metadata: name: netconfig spec: networks: - name: ctlplane dnsDomain: ctlplane.example.com subnets: - name: subnet1 allocationRanges: - end: 192.168.122.120 start: 192.168.122.100 - end: 192.168.122.200 start: 192.168.122.150 cidr: 192.168.122.0/24 gateway: 192.168.122.1 - name: internalapi dnsDomain: internalapi.example.com subnets: - name: subnet1 allocationRanges: - end: 172.17.0.250 start: 172.17.0.100 cidr: 172.17.0.0/24 vlan: 20 - name: External dnsDomain: external.example.com subnets: - name: subnet1 allocationRanges: - end: 10.0.0.250 start: 10.0.0.100 cidr: 10.0.0.0/24 gateway: 10.0.0.1 - name: storage dnsDomain: storage.example.com subnets: - name: subnet1 allocationRanges: - end: 172.18.0.250 start: 172.18.0.100 cidr: 172.18.0.0/24 vlan: 21 - name: storagemgmt dnsDomain: storagemgmt.example.com subnets: - name: subnet1 allocationRanges: - end: 172.20.0.250 start: 172.20.0.100 cidr: 172.20.0.0/24 vlan: 23 - name: tenant dnsDomain: tenant.example.com subnets: - name: subnet1 allocationRanges: - end: 172.19.0.250 start: 172.19.0.100 cidr: 172.19.0.0/24 vlan: 22 EOF
PODIFIED_DB_ROOT_PASSWORD=USD(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d) CEPH_FSID=USD(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //' alias openstack="oc exec -t openstackclient -- openstack" declare -A computes export computes=( ["standalone.localdomain"]="192.168.122.100" # )
oc apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: dataplane-adoption-secret namespace: openstack data: ssh-privatekey: | USD(cat <path_to_SSH_key> | base64 | sed 's/^/ /') EOF
cd "USD(mktemp -d)" ssh-keygen -f ./id -t ecdsa-sha2-nistp521 -N '' get secret nova-migration-ssh-key || oc create secret generic nova-migration-ssh-key -n openstack --from-file=ssh-privatekey=id --from-file=ssh-publickey=id.pub --type kubernetes.io/ssh-auth rm -f id* cd -
oc apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 19-nova-compute-cell1-workarounds.conf: | [workarounds] disable_compute_service_check_for_ffu=true EOF
tlsCerts: contents: - dnsnames - ips networks: - ctlplane issuer: osp-rootca-issuer-internal caCerts: combined-ca-bundle edpmServiceType: nova
oc apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 19-nova-compute-cell1-workarounds.conf: | [workarounds] disable_compute_service_check_for_ffu=true 03-ceph-nova.conf: | [libvirt] images_type=rbd images_rbd_pool=vms images_rbd_ceph_conf=/etc/ceph/ceph.conf images_rbd_glance_store_name=default_backend images_rbd_glance_copy_poll_interval=15 images_rbd_glance_copy_timeout=600 rbd_user=openstack rbd_secret_uuid=USDCEPH_FSID EOF
oc create secret generic subscription-manager --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'
oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<registry_username>": "<registry_password>"}}'
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack spec: tlsEnabled: false 1 networkAttachments: - ctlplane preProvisioned: true services: - redhat - bootstrap - download-cache - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - reboot-os - install-certs - libvirt - nova - ovn - neutron-metadata - telemetry env: - name: ANSIBLE_CALLBACKS_ENABLED value: "profile_tasks" - name: ANSIBLE_FORCE_COLOR value: "True" nodes: standalone: hostName: standalone 2 ansible: ansibleHost: USD{computes[standalone.localdomain]} networks: - defaultRoute: true fixedIP: USD{computes[standalone.localdomain]} name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: tenant subnetName: subnet1 nodeTemplate: ansibleSSHPrivateKeySecret: dataplane-adoption-secret ansible: ansibleUser: root ansibleVarsFrom: - secretRef: name: subscription-manager - secretRef: name: redhat-registry ansibleVars: rhc_release: 9.2 rhc_repositories: - {name: "*", state: disabled} - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled} - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled} - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled} - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled} - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled} - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled} edpm_bootstrap_release_version_package: [] # edpm_network_config # Default nic config template for a EDPM node # These vars are edpm_network_config role vars edpm_network_config_template: | --- {% set mtu_list = [ctlplane_mtu] %} {% for network in nodeset_networks %} {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic1 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in nodeset_networks %} - type: vlan mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} addresses: - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }} {% endfor %} edpm_network_config_hide_sensitive_logs: false # # These vars are for the network config templates themselves and are # considered EDPM network defaults. neutron_physical_bridge_name: br-ctlplane neutron_public_interface_name: eth0 # edpm_nodes_validation edpm_nodes_validation_validate_controllers_icmp: false edpm_nodes_validation_validate_gateway_icmp: false # edpm ovn-controller configuration edpm_ovn_bridge_mappings: <bridge_mappings> 3 edpm_ovn_bridge: br-int edpm_ovn_encap_type: geneve ovn_monitor_all: true edpm_ovn_remote_probe_interval: 60000 edpm_ovn_ofctrl_wait_before_clear: 8000 timesync_ntp_servers: - hostname: clock.redhat.com - hostname: clock2.redhat.com edpm_bootstrap_command: | # FIXME: perform dnf upgrade for other packages in EDPM ansible # here we only ensuring that decontainerized libvirt can start dnf -y upgrade openstack-selinux rm -f /run/virtlogd.pid gather_facts: false # edpm firewall, change the allowed CIDR if needed edpm_sshd_configure_firewall: true edpm_sshd_allowed_ranges: ['192.168.122.0/24'] # Do not attempt OVS major upgrades here edpm_ovs_packages: - openvswitch3.3 EOF
ovs-vsctl list Open . external_ids : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
oc patch osdpns/openstack --type=merge --patch " spec: services: - redhat - bootstrap - download-cache - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - reboot-os - ceph-client - install-certs - ovn - neutron-metadata - libvirt - nova - telemetry nodeTemplate: extraMounts: - extraVolType: Ceph volumes: - name: ceph secret: secretName: ceph-conf-files mounts: - name: ceph mountPath: "/etc/ceph" readOnly: true "
oc patch openstackdataplanenodeset openstack --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-sriov" }, { "op": "add", "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_physical_device_mappings", "value": "dummy_sriov_net:dummy-dev" }, { "op": "add", "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_bandwidths", "value": "dummy-dev:40000000:40000000" }, { "op": "add", "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_hypervisors", "value": "dummy-dev:standalone.localdomain" } ]'
oc patch openstackdataplanenodeset openstack --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-dhcp" }]'
.. spec: serviceUser: neutron customServiceConfig: | [DEFAULT] dhcp_agent_notification = True [ovn] disable_ovn_dhcp_for_baremetal_ports = true
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: name: pre-adoption-validation spec: playbook: osp.edpm.pre_adoption_validation EOF
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-pre-adoption spec: nodeSets: - openstack servicesOverride: - pre-adoption-validation EOF
watch oc get pod -l app=openstackansibleee
oc logs -l app=openstackansibleee -f --max-log-requests 20
oc wait --for condition=Ready openstackdataplanedeployment/openstack-pre-adoption --timeout=10m
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: name: tripleo-cleanup spec: playbook: osp.edpm.tripleo_cleanup EOF
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: tripleo-cleanup spec: nodeSets: - openstack servicesOverride: - tripleo-cleanup EOF
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack spec: nodeSets: - openstack EOF
watch oc get pod -l app=openstackansibleee
oc logs -l app=openstackansibleee -f --max-log-requests 20
oc wait --for condition=Ready osdpns/openstack --timeout=30m
oc exec openstackclient -- openstack network agent list +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+ | 174fc099-5cc9-4348-b8fc-59ed44fcfb0e | DHCP agent | standalone.localdomain | nova | :-) | UP | neutron-dhcp-agent | | 10482583-2130-5b0d-958f-3430da21b929 | OVN Metadata agent | standalone.localdomain | | :-) | UP | neutron-ovn-metadata-agent | | a4f1b584-16f1-4937-b2b0-28102a3f6eaa | OVN Controller agent | standalone.localdomain | | :-) | UP | ovn-controller | +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+
oc exec openstack-cell1-galera-0 -c galera -- mysql -rs -uroot -pUSDPODIFIED_DB_ROOT_PASSWORD -e "select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='nova-compute';"
oc patch openstackcontrolplane openstack -n openstack --type=merge --patch ' spec: nova: template: cellTemplates: cell0: conductorServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false cell1: metadataServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false conductorServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false apiServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false metadataServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false schedulerServiceTemplate: customServiceConfig: | [workarounds] disable_compute_service_check_for_ffu=false '
oc wait --for condition=Ready --timeout=300s Nova/nova
oc apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 20-nova-compute-cell1-workarounds.conf: | [workarounds] disable_compute_service_check_for_ffu=false --- apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-nova-compute-ffu namespace: openstack spec: nodeSets: - openstack servicesOverride: - nova EOF
oc wait --for condition=Ready openstackdataplanedeployment/openstack-nova-compute-ffu --timeout=5m
oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
USD{BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test 2>&1 || echo FAIL
USD{BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test ACTIVE" && USD{BASH_ALIASES[openstack]} server stop test || echo PASS USD{BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test SHUTOFF" || echo FAIL USD{BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test 2>&1 || echo PASS
USD{BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test SHUTOFF" && USD{BASH_ALIASES[openstack]} server start test || echo PASS USD{BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test ACTIVE" && USD{BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test --fit-width -f json | jq -r '.state' | grep running || echo FAIL
declare -A networkers networkers=( ["standalone.localdomain"]="192.168.122.100" # )
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-networker spec: tlsEnabled: false 1 networkAttachments: - ctlplane preProvisioned: true services: - bootstrap - download-cache - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - install-certs - ovn env: - name: ANSIBLE_CALLBACKS_ENABLED value: "profile_tasks" - name: ANSIBLE_FORCE_COLOR value: "True" nodes: standalone: hostName: standalone ansible: ansibleHost: USD{networkers[standalone.localdomain]} networks: - defaultRoute: true fixedIP: USD{networkers[standalone.localdomain]} name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: tenant subnetName: subnet1 nodeTemplate: ansibleSSHPrivateKeySecret: dataplane-adoption-secret ansible: ansibleUser: root ansibleVarsFrom: - prefix: subscription_manager_ secretRef: name: subscription-manager - secretRef: name: redhat-registry ansibleVars: edpm_bootstrap_release_version_package: [] # edpm_network_config # Default nic config template for a EDPM node # These vars are edpm_network_config role vars edpm_network_config_template: | --- {% set mtu_list = [ctlplane_mtu] %} {% for network in nodeset_networks %} {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic1 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in nodeset_networks %} - type: vlan mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} addresses: - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }} {% endfor %} edpm_network_config_hide_sensitive_logs: false # # These vars are for the network config templates themselves and are # considered EDPM network defaults. neutron_physical_bridge_name: br-ctlplane neutron_public_interface_name: eth0 # edpm_nodes_validation edpm_nodes_validation_validate_controllers_icmp: false edpm_nodes_validation_validate_gateway_icmp: false # edpm ovn-controller configuration edpm_ovn_bridge_mappings: <bridge_mappings> 2 edpm_ovn_bridge: br-int edpm_ovn_encap_type: geneve ovn_monitor_all: true edpm_ovn_remote_probe_interval: 60000 edpm_ovn_ofctrl_wait_before_clear: 8000 # serve as a OVN gateway edpm_enable_chassis_gw: true 3 timesync_ntp_servers: - hostname: clock.redhat.com - hostname: clock2.redhat.com edpm_bootstrap_command: | subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }} subscription-manager release --set=9.2 subscription-manager repos --disable=* subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms gather_facts: false enable_debug: false # edpm firewall, change the allowed CIDR if needed edpm_sshd_configure_firewall: true edpm_sshd_allowed_ranges: ['192.168.122.0/24'] # SELinux module edpm_selinux_mode: enforcing # Do not attempt OVS major upgrades here edpm_ovs_packages: - openvswitch3.1 EOF
ovs-vsctl list Open . external_ids : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-metadata" }]'
oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[ { "op": "add", "path": "/spec/services/-", "value": "neutron-dhcp" }]'
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-pre-adoption-networker spec: nodeSets: - openstack-networker servicesOverride: - pre-adoption-validation EOF
watch oc get pod -l app=openstackansibleee
oc logs -l app=openstackansibleee -f --max-log-requests 20
oc wait --for condition=Ready openstackdataplanedeployment/openstack-pre-adoption-networker --timeout=10m
oc apply -f - <<EOF apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-networker spec: nodeSets: - openstack-networker EOF
watch oc get pod -l app=openstackansibleee
oc logs -l app=openstackansibleee -f --max-log-requests 20
oc wait --for condition=Ready osdpns/<networker_CR_name> --timeout=30m
oc exec openstackclient -- openstack network agent list +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+ | 174fc099-5cc9-4348-b8fc-59ed44fcfb0e | DHCP agent | standalone.localdomain | nova | :-) | UP | neutron-dhcp-agent | | 10482583-2130-5b0d-958f-3430da21b929 | OVN Metadata agent | standalone.localdomain | | :-) | UP | neutron-ovn-metadata-agent | | a4f1b584-16f1-4937-b2b0-28102a3f6eaa | OVN Controller Gateway agent | standalone.localdomain | | :-) | UP | ovn-controller | +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+