Configuring the Compute service for instance creation Red Hat OpenStack Platform 17.1 Configuring and managing the Red Hat OpenStack Platform Compute service (nova) for creating instances OpenStack Documentation Team rhos-docs@redhat.com

Commands:
parameter_defaults: NovaNfsEnabled: true NovaNfsOptions: "context=system_u:object_r:nfs_t:s0" NovaNfsShare: "192.0.2.254:/export/nova" NovaNfsVersion: "4.2" NovaSchedulerEnabledFilters: - AggregateInstanceExtraSpecsFilter - ComputeFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter
parameter_defaults: ComputeExtraConfig: nova::compute::force_raw_images: True
parameter_defaults: ComputeExtraConfig: nova::config::nova_config: DEFAULT/timeout_nbd: value: '20'
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_vcpus> [--private --project <project_id>] <flavor_name>
(overcloud)USD openstack flavor set --property <key=value> --property <key=value> ... <flavor_name>
(overcloud)USD openstack flavor set --property hw:cpu_sockets=2 --property hw:cpu_cores=2 processor_topology_flavor
openstack flavor set cpu_limits_flavor --property quota:cpu_quota=10000 --property quota:cpu_period=20000
--property trait:HW_CPU_HYPERTHREADING=forbidden
--property trait:HW_CPU_HYPERTHREADING=required
openstack flavor set numa_top_flavor --property hw:numa_nodes=2 --property hw:numa_cpus.0=0,1,2,3,4,5 --property hw:numa_cpus.1=6,7 --property hw:numa_mem.0=3072 --property hw:numa_mem.1=1024
openstack flavor set <flavor> --property hw:cpu_realtime="yes" --property hw:cpu_realtime_mask=^0-1
--property hw:cpu_policy=dedicated
<alias>:<count>
openstack flavor set --property trait:HW_CPU_X86_AVX512BW=required avx512-flavor
openstack flavor set --property resources:CUSTOM_BAREMETAL_SMALL=1 --property resources:VCPU=0 --property resources:MEMORY_MB=0 --property resources:DISK_GB=0 compute-small
[stack@director ~]USD source ~/stackrc
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_cpu_pinning.yaml Compute:ComputeCPUPinning Compute Controller
(undercloud)USD openstack overcloud node introspect --all-manageable --provide
(undercloud)USD openstack baremetal node set --resource-class baremetal.CPU-PINNING <node>
- name: Controller count: 3 - name: Compute count: 3 - name: ComputeCPUPinning count: 1 defaults: resource_class: baremetal.CPU-PINNING network_config: template: /home/stack/templates/nic-config/myRoleTopology.j2 1
(undercloud)USD openstack overcloud node provision --stack <stack> [--network-config \] --output /home/stack/templates/overcloud-baremetal-deployed.yaml /home/stack/templates/overcloud-baremetal-deploy.yaml
(undercloud)USD watch openstack baremetal node list
parameter_defaults: ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2 ComputeCPUPinningNetworkConfigTemplate: /home/stack/templates/nic-configs/<cpu_pinning_net_top>.j2 ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2
parameter_defaults: NovaSchedulerEnabledFilters: - AvailabilityZoneFilter - ComputeFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter - ServerGroupAntiAffinityFilter - ServerGroupAffinityFilter - PciPassthroughFilter - NUMATopologyFilter
parameter_defaults: ComputeCPUPinningParameters: NovaComputeCpuDedicatedSet: 1,3,5,7
parameter_defaults: ComputeCPUPinningParameters: NovaComputeCpuSharedSet: 2,6
parameter_defaults: ComputeCPUPinningParameters: NovaReservedHugePages: <ram>
parameter_defaults: ComputeCPUPinningParameters: IsolCpusList: 1-3,5-7
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -r /home/stack/templates/roles_data_cpu_pinning.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/cpu_pinning.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml
(undercloud)USD source ~/overcloudrc
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> pinned_cpus
(overcloud)USD openstack flavor set --property hw:cpu_policy=dedicated pinned_cpus
(overcloud)USD openstack flavor set --property hw:mem_page_size=<page_size> pinned_cpus
(overcloud)USD openstack flavor set --property hw:cpu_thread_policy=require pinned_cpus
(overcloud)USD openstack server create --flavor pinned_cpus --image <image> pinned_cpu_instance
(undercloud)USD source ~/overcloudrc
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> floating_cpus
(overcloud)USD openstack flavor set --property hw:cpu_policy=shared floating_cpus
(overcloud)USD openstack flavor set --property hw:mem_page_size=<page_size> pinned_cpus
(undercloud)USD source ~/overcloudrc
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <number_of_reserved_vcpus> --property hw:cpu_policy=mixed mixed_CPUs_flavor
(overcloud)USD openstack flavor set --property hw:cpu_dedicated_mask=<CPU_number> mixed_CPUs_flavor
(overcloud)USD openstack flavor set --property hw:mem_page_size=<page_size> pinned_cpus
grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,2 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:1,3
parameter_defaults: NovaComputeCpuDedicatedSet: 2-15,18-31
parameter_defaults: NovaComputeCpuSharedSet: 0,1,16,17
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(overcloud)USD openstack flavor set --property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=share dedicated_emulator_threads
[stack@director ~]USD source ~/stackrc
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: <cpu_mode>
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: 'custom' NovaLibvirtCPUModels: <cpu_model>
NovaLibvirtCPUModels: - SandyBridge - IvyBridge - Haswell-noTSX-IBRS
sudo podman exec -it nova_libvirt virsh cpu-models <arch>
sudo podman exec -it nova_virtqemud virsh cpu-models <arch>
parameter_defaults: ComputeParameters: NovaLibvirtCPUModelExtraFlags: <cpu_feature_flags>
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: 'custom' NovaLibvirtCPUModels: - IvyBridge - Cascadelake-Server NovaLibvirtCPUModelExtraFlags: 'pcid,+ssbd,-mtrr'
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
cp /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml /home/stack/templates/enable-swap.yaml
parameter_defaults: swap_size_megabytes: <swap size in MB> swap_path: <full path to location of swap, default: /swap>
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/enable-swap.yaml
NovaReservedHostMemory = total_RAM - ( (vm_no * (avg_instance_size + overhead)) + (resource1 * resource_ram) + (resourcen * resource_ram))
parameter_defaults: ComputeParameters: NovaReservedHugePages: ["node:0,size:1GB,count:1","node:1,size:1GB,count:1"]
parameter_defaults: ComputeParameters: KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32"
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: 'custom' NovaLibvirtCPUModels: 'Haswell-noTSX' NovaLibvirtCPUModelExtraFlags: 'vmx, pdpe1gb'
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: 'custom' NovaLibvirtCPUModels: 'Haswell-noTSX' NovaLibvirtCPUModelExtraFlags: 'vmx, pdpe1gb, +pcid'
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> huge_pages
openstack flavor set huge_pages --property hw:mem_page_size=<page_size>
openstack server create --flavor huge_pages --image <image> huge_pages_instance
heat_template_version: <version> description: > Huge pages configuration resources: userdata: type: OS::Heat::MultipartMime properties: parts: - config: {get_resource: hugepages_config} hugepages_config: type: OS::Heat::SoftwareConfig properties: config: | #!/bin/bash hostname | grep -qiE 'co?mp' || exit 0 systemctl mask dev-hugepages.mount || true for pagesize in 2M 1G;do if ! [ -d "/dev/hugepagesUSD{pagesize}" ]; then mkdir -p "/dev/hugepagesUSD{pagesize}" cat << EOF > /etc/systemd/system/dev-hugepagesUSD{pagesize}.mount [Unit] Description=USD{pagesize} Huge Pages File System Documentation=https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems DefaultDependencies=no Before=sysinit.target ConditionPathExists=/sys/kernel/mm/hugepages ConditionCapability=CAP_SYS_ADMIN ConditionVirtualization=!private-users [Mount] What=hugetlbfs Where=/dev/hugepagesUSD{pagesize} Type=hugetlbfs Options=pagesize=USD{pagesize} [Install] WantedBy = sysinit.target EOF fi done systemctl daemon-reload for pagesize in 2M 1G;do systemctl enable --now dev-hugepagesUSD{pagesize}.mount done outputs: OS::stack_id: value: {get_resource: userdata}
parameter_defaults NovaComputeOptVolumes: - /opt/dev:/opt/dev NovaLibvirtOptVolumes: - /opt/dev:/opt/dev
resource_registry: OS::TripleO::NodeUserData: ./hugepages.yaml
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/firstboot.yaml
parameter_defaults: NovaLibvirtFileBackedMemory: 102400
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
mkfs.ext4 /dev/sdb
mount /dev/sdb /var/lib/libvirt/qemu/ram
[stack@director ~]USD source ~/stackrc
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_amd_sev.yaml Compute:ComputeAMDSEV Controller
(undercloud)USD openstack overcloud node introspect --all-manageable --provide
(undercloud)USD openstack baremetal node set --resource-class baremetal.AMD-SEV <node>
- name: Controller count: 3 - name: Compute count: 3 - name: ComputeAMDSEV count: 1 defaults: resource_class: baremetal.AMD-SEV network_config: template: /home/stack/templates/nic-config/myRoleTopology.j2 1
(undercloud)USD openstack overcloud node provision --stack <stack> [--network-config \] --output /home/stack/templates/overcloud-baremetal-deployed.yaml /home/stack/templates/overcloud-baremetal-deploy.yaml
(undercloud)USD watch openstack baremetal node list
parameter_defaults: ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2 ComputeAMDSEVNetworkConfigTemplate: /home/stack/templates/nic-configs/<amd_sev_net_top>.j2 ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2
lscpu | grep sev
parameter_defaults: ComputeAMDSEVExtraConfig: nova::config::nova_config: libvirt/num_memory_encrypted_guests: value: 15
parameter_defaults: ComputeAMDSEVParameters: NovaHWMachineType: x86_64=q35
parameter_defaults: ComputeAMDSEVParameters: NovaReservedHostMemory: <libvirt/num_memory_encrypted_guests * 16>
parameter_defaults: ComputeAMDSEVParameters: KernelArgs: "hugepagesz=1GB hugepages=32 default_hugepagesz=1GB mem_encrypt=on kvm_amd.sev=1"
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -r /home/stack/templates/roles_data_amd_sev.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/<compute_environment_file>.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml
(overcloud)USD openstack image create ... --property hw_firmware_type=uefi amd-sev-image
(overcloud)USD openstack image set --property hw_mem_encryption=True amd-sev-image
(overcloud)USD openstack image set --property hw_machine_type=q35 amd-sev-image
(overcloud)USD openstack image set --property trait:HW_CPU_X86_AMD_SEV=required amd-sev-image
(overcloud)USD openstack flavor create --vcpus 1 --ram 512 --disk 2 --property hw:mem_encryption=True m1.small-amd-sev
(overcloud)USD openstack flavor set --property trait:HW_CPU_X86_AMD_SEV=required m1.small-amd-sev
(overcloud)USD openstack server create --flavor m1.small-amd-sev --image amd-sev-image amd-sev-instance
dmesg | grep -i sev AMD Secure Encrypted Virtualization (SEV) active
source ~/stackrc
parameter_defaults: NovaMaxDiskDevicesToAttach: <max_device_limit>
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<environment_file>.yaml
[stack@director ~]USD source ~/stackrc
parameter_defaults: NovaNfsEnabled: True NovaNfsShare: <nfs_share>
parameter_defaults: NovaNfsOptions: 'context=system_u:object_r:nfs_t:s0,<additional_nfs_mount_options>'
man 8 mount.
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/nfs_instance_disk_backend.yaml
parameter_defaults: ComputeParameters: NovaGlanceEnableRbdDownload: True NovaEnableRbdBackend: False
parameter_defaults: ComputeParameters: NovaGlanceEnableRbdDownload: True NovaEnableRbdBackend: False NovaGlanceRbdDownloadMultistoreID: <rbd_backend_id>
parameter_defaults: ComputeExtraConfig: nova::config::nova_config: glance/rbd_user: value: 'glance' glance/rbd_pool: value: 'images' glance/rbd_ceph_conf: value: '/etc/ceph/ceph.conf' glance/rbd_connect_timeout: value: '5'
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(overcloud)USD openstack image create ... trait-image
(overcloud)USD openstack --os-placement-api-version 1.6 trait list
(overcloud)USD openstack --os-placement-api-version 1.6 trait create CUSTOM_TRAIT_NAME
(overcloud)USD existing_traits=USD(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')
(overcloud)USD echo USDexisting_traits
(overcloud)USD openstack --os-placement-api-version 1.6 resource provider trait set USDexisting_traits --trait <TRAIT_NAME> <host_uuid>
(overcloud)USD openstack image set --property trait:HW_CPU_X86_AVX512BW=required trait-image
(overcloud)USD openstack image set --property trait:COMPUTE_VOLUME_MULTI_ATTACH=forbidden trait-image
(overcloud)USD openstack flavor create --vcpus 1 --ram 512 --disk 2 trait-flavor
(overcloud)USD openstack --os-placement-api-version 1.6 trait list
(overcloud)USD openstack --os-placement-api-version 1.6 trait create CUSTOM_TRAIT_NAME
(overcloud)USD existing_traits=USD(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')
(overcloud)USD echo USDexisting_traits
(overcloud)USD openstack --os-placement-api-version 1.6 resource provider trait set USDexisting_traits --trait <TRAIT_NAME> <host_uuid>
(overcloud)USD openstack flavor set --property trait:HW_CPU_X86_AVX512BW=required trait-flavor
(overcloud)USD openstack flavor set --property trait:COMPUTE_VOLUME_MULTI_ATTACH=forbidden trait-flavor
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(overcloud)USD openstack --os-placement-api-version 1.6 trait list
(overcloud)USD openstack --os-placement-api-version 1.6 trait create CUSTOM_TRAIT_NAME
(overcloud)USD existing_traits=USD(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')
(overcloud)USD echo USDexisting_traits
(overcloud)USD openstack --os-placement-api-version 1.6 resource provider trait set USDexisting_traits --trait <TRAIT_NAME> <host_uuid>
(overcloud)USD openstack --os-compute-api-version 2.53 aggregate set --property trait:<TRAIT_NAME>=required <aggregate_name>
(overcloud)USD openstack flavor set --property trait:<TRAIT_NAME>=required <flavor> (overcloud)USD openstack image set --property trait:<TRAIT_NAME>=required <image>
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
parameter_defaults: NovaSchedulerEnabledFilters: - AggregateInstanceExtraSpecsFilter - ComputeFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter
parameter_defaults: ControllerExtraConfig: nova::scheduler::filter::ram_weight_multiplier: '2.0'
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
openstack server create --image cedef40a-ed67-4d10-800e-17455edce175 --flavor 1 --hint different_host=a0cf03a5-d921-4877-bb5c-86d26cf818e1 --hint different_host=8c19174f-4220-44f0-824a-cd1eeef10287 server-1
parameter_defaults: ComputeExtraConfig: nova::config::nova_config: filter_scheduler/isolated_hosts: value: server1, server2 filter_scheduler/isolated_images: value: 342b492c-128f-4a42-8d3a-c5088cf27d13, ebd267a6-ca86-4d6c-9a0e-bd132d6b7d09
parameter_defaults: ComputeExtraConfig: nova::config::nova_config: DEFAULT/compute_monitors: value: 'cpu.virt_driver'
openstack server create --image cedef40a-ed67-4d10-800e-17455edce175 --flavor 1 --hint same_host=a0cf03a5-d921-4877-bb5c-86d26cf818e1 --hint same_host=8c19174f-4220-44f0-824a-cd1eeef10287 server-1
openstack server group create --policy affinity <group_name>
openstack server create --image <image> --flavor <flavor> --hint group=<group_uuid> <instance_name>
openstack server group create --policy anti-affinity <group_name>
openstack server create --image <image> --flavor <flavor> --hint group=<group_uuid> <instance_name>
openstack server create --image <image> --flavor <flavor> --hint build_near_host_ip=<ip_address> --hint cidr=<subnet_mask> <instance_name>
(node_resource_availability - minval) / (maxval - minval)
(w1_multiplier * norm(w1)) + (w2_multiplier * norm(w2)) +
ControllerExtraConfig: nova::scheduler::filter::scheduler_weight_classes: 'nova.scheduler.weights.ram.RAMWeigher' nova::scheduler::filter::ram_weight_multiplier: '2.0'
openstack --os-compute-api-version 2.15 server group create --policy soft-affinity <group_name>
openstack --os-compute-api-version 2.15 server group create --policy soft-affinity <group_name>
meta: schema_version: '1.0' providers: - identification: uuid: <node_uuid>
meta: schema_version: '1.0' providers: - identification: uuid: <node_uuid> inventories: additional: - CUSTOM_EXAMPLE_RESOURCE_CLASS: total: <total_available> reserved: <reserved> min_unit: <min_unit> max_unit: <max_unit> step_size: <step_size> allocation_ratio: <allocation_ratio>
meta: schema_version: '1.0' providers: - identification: uuid: <node_uuid> inventories: additional: traits: additional: - 'CUSTOM_EXAMPLE_TRAIT'
meta: schema_version: 1.0 providers: - identification: uuid: USDCOMPUTE_NODE inventories: additional: CUSTOM_LLC: # Describing LLC on this Compute node total: 22 1 reserved: 2 2 min_unit: 1 3 max_unit: 11 4 step_size: 1 5 allocation_ratio: 1.0 6 traits: additional: # This Compute node enables support for P-state control - CUSTOM_P_STATE_ENABLED
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/provider.yaml
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_roles_data_custom_traits.yaml Compute:Compute Controller
########################## # GPU configuration # ########################## ComputeGpuParameters: NovaVGPUTypesDeviceAddressesMapping: {'nvidia-319': ['0000:82:00.0'], 'nvidia-320': ['0000:04:00.0']} CustomProviderInventories: - name: computegpu-0.localdomain_pci_0000_04_00_0 traits: - CUSTOM_NVIDIA_12 - name: computegpu-0.localdomain_pci_0000_82_00_0 traits: - CUSTOM_NVIDIA_11 - name: computegpu-1.localdomain_pci_0000_04_00_0 traits: - CUSTOM_NVIDIA_12 - name: computegpu-1.localdomain_pci_0000_82_00_0 traits: - CUSTOM_NVIDIA_11 - uuid: USDCOMPUTE_NODE inventories: CUSTOM_EXAMPLE_RESOURCE_CLASS: total: 100 1 reserved: 0 2 min_unit: 1 3 max_unit: 10 4 step_size: 1 5 allocation_ratio: 1.0 6 CUSTOM_ANOTHER_EXAMPLE_RESOURCE_CLASS: total: 100 traits: # This Compute node enables support for P-state and C-state control - CUSTOM_P_STATE_ENABLED - CUSTOM_C_STATE_ENABLED
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/roles_data_roles_data_custom_traits.yaml
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(overcloud)# openstack aggregate create <aggregate_name>
(overcloud)# openstack aggregate set --property <key=value> --property <key=value> <aggregate_name>
(overcloud)# openstack aggregate add host <aggregate_name> <host_name>
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> host-agg-flavor
(overcloud)USD openstack image create host-agg-image
(overcloud)# openstack flavor set --property aggregate_instance_extra_specs:ssd=true host-agg-flavor
(overcloud)# openstack image set --property os_type=linux host-agg-image
(overcloud)# openstack aggregate create --zone <availability_zone> <aggregate_name>
(overcloud)# openstack aggregate set --zone <availability_zone> <aggregate_name>
(overcloud)# openstack aggregate set --property <key=value> <aggregate_name>
(overcloud)# openstack aggregate add host <aggregate_name> <host_name>
(overcloud)# openstack aggregate show <aggregate_name>
(overcloud)# openstack aggregate remove host <aggregate_name> <host_name>
(overcloud)# openstack aggregate delete <aggregate_name>
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml \
(overcloud)# openstack project list
(overcloud)# openstack aggregate set --property filter_tenant_id<ID0>=<project_id0> --property filter_tenant_id<ID1>=<project_id1> --property filter_tenant_id<IDn>=<project_idn> <aggregate_name>
(overcloud)# openstack aggregate set --property filter_tenant_id0=78f1 --property filter_tenant_id1=9d3t --property filter_tenant_id2=aa29 project-isolated-aggregate
(overcloud)# openstack aggregate set --property filter_tenant_id=78f1 single-project-isolated-aggregate
[stack@director ~]USD source ~/stackrc
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_pci_passthrough.yaml Compute:ComputePCI Compute Controller
(undercloud)USD openstack overcloud node introspect --all-manageable --provide
(undercloud)USD openstack baremetal node set --resource-class baremetal.PCI-PASSTHROUGH <node>
- name: Controller count: 3 - name: Compute count: 3 - name: ComputePCI count: 1 defaults: resource_class: baremetal.PCI-PASSTHROUGH network_config: template: /home/stack/templates/nic-config/myRoleTopology.j2 1
(undercloud)USD openstack overcloud node provision --stack <stack> [--network-config \] --output /home/stack/templates/overcloud-baremetal-deployed.yaml /home/stack/templates/overcloud-baremetal-deploy.yaml
(undercloud)USD watch openstack baremetal node list
parameter_defaults: ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2 ComputePCINetworkConfigTemplate: /home/stack/templates/nic-configs/<pci_passthrough_net_top>.j2 ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2
parameter_defaults: NovaSchedulerEnabledFilters: - AvailabilityZoneFilter - ComputeFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter - ServerGroupAntiAffinityFilter - ServerGroupAffinityFilter - PciPassthroughFilter - NUMATopologyFilter
parameter_defaults: ControllerExtraConfig: nova::pci::aliases: - name: "a1" product_id: "1572" vendor_id: "8086" device_type: "type-PF"
parameter_defaults: ControllerExtraConfig: nova::pci::aliases: - name: "a1" product_id: "1572" vendor_id: "8086" device_type: "type-PF" numa_policy: "preferred"
parameter_defaults: ComputePCIParameters: NovaPCIPassthrough: - vendor_id: "8086" product_id: "1572"
parameter_defaults: ComputePCIExtraConfig: nova::pci::aliases: - name: "a1" product_id: "1572" vendor_id: "8086" device_type: "type-PF"
parameter_defaults: ComputePCIParameters: KernelArgs: "intel_iommu=on iommu=pt \ vfio-pci.ids=<pci_device_id> rd.driver.pre=vfio-pci"
parameter_defaults: ComputePCIParameters: KernelArgs: "amd_iommu=on iommu=pt"
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -r /home/stack/templates/roles_data_pci_passthrough.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/pci_passthrough_controller.yaml -e /home/stack/templates/pci_passthrough_compute.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml
(overcloud)USD openstack flavor set --property "pci_passthrough:alias"="a1:2" device_passthrough
(overcloud)USD openstack flavor set --property "hw:pci_numa_affinity_policy"="required" device_passthrough
(overcloud)USD openstack image set --property hw_pci_numa_affinity_policy=required device_passthrough_image
openstack server create --flavor device_passthrough --image <image> --wait test-pci
lspci -nn | grep <device_name>
NovaPCIPassthrough: - address: "*:0a:00.*" physical_network: physnet1
NovaPCIPassthrough: - address: domain: ".*" bus: "02" slot: "01" function: "[0-2]" physical_network: net1
[stack@director ~]USD source ~/stackrc
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_vdpa.yaml ComputeVdpa Compute Controller
############################################################################### Role: ComputeVdpa # ############################################################################### - name: ComputeVdpa description: | VDPA Compute Node role CountDefault: 1 # Create external Neutron bridge tags: - compute - external_bridge networks: InternalApi: subnet: internal_api_subnet Tenant: subnet: tenant_subnet Storage: subnet: storage_subnet HostnameFormatDefault: '%stackname%-computevdpa-%index%' deprecated_nic_config_name: compute-vdpa.yaml
(undercloud)USD openstack overcloud node introspect --all-manageable --provide
(undercloud)USD openstack baremetal node set --resource-class baremetal.VDPA <node>
- name: Controller count: 3 - name: Compute count: 3 - name: ComputeVdpa count: 1 defaults: resource_class: baremetal.VDPA network_config: template: /home/stack/templates/nic-config/<role_topology_file>
- type: ovs_bridge name: br-tenant members: - type: sriov_pf name: enp6s0f0 numvfs: 8 use_dhcp: false vdpa: true link_mode: switchdev - type: sriov_pf name: enp6s0f1 numvfs: 8 use_dhcp: false vdpa: true link_mode: switchdev
(undercloud)USD openstack overcloud node provision [--stack <stack>] [--network-config \] --output <deployment_file> /home/stack/templates/overcloud-baremetal-deploy.yaml
(undercloud)USD watch openstack baremetal node list
parameter_defaults: ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2 ComputeVdpaNetworkConfigTemplate: /home/stack/templates/nic-configs/<role_topology_file> ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2
parameter_defaults: NovaSchedulerEnabledFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']
parameter_defaults: ComputeVdpaParameters: NovaPCIPassthrough: - vendor_id: "15b3" product_id: "101d" address: "06:00.0" physical_network: "tenant" - vendor_id: "15b3" product_id: "101d" address: "06:00.1" physical_network: "tenant"
parameter_defaults: ComputeVdpaParameters: KernelArgs: "intel_iommu=on iommu=pt"
parameter_defaults: NeutronBridgeMappings: - <bridge_map_1> - <bridge_map_n> NeutronTunnelTypes: '<tunnel_types>' NeutronNetworkType: '<network_types>' NeutronNetworkVLANRanges: - <network_vlan_range_1> - <network_vlan_range_n>
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -r /home/stack/templates/roles_data_vdpa.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/vdpa_compute.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml
openstack port show vdpa-port
sudo podman exec -it nova_libvirt virsh dumpxml <instance_name> | grep mdev
sudo podman exec -it nova_virtqemud virsh dumpxml <instance_name> | grep mdev
[stack@director ~]USD source ~/stackrc
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_gpu.yaml Compute:ComputeGpu Compute Controller
(undercloud)USD openstack overcloud node introspect --all-manageable --provide
(undercloud)USD openstack baremetal node set --resource-class baremetal.GPU <node>
- name: Controller count: 3 - name: Compute count: 3 - name: ComputeGpu count: 1 defaults: resource_class: baremetal.GPU network_config: template: /home/stack/templates/nic-config/myRoleTopology.j2 1
(undercloud)USD openstack overcloud node provision --stack <stack> [--network-config \] --output /home/stack/templates/overcloud-baremetal-deployed.yaml /home/stack/templates/overcloud-baremetal-deploy.yaml
(undercloud)USD watch openstack baremetal node list
parameter_defaults: ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2 ComputeGpuNetworkConfigTemplate: /home/stack/templates/nic-configs/<gpu_net_top>.j2 ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2
ls /sys/class/mdev_bus/
ls /sys/class/mdev_bus/<mdev_device>/mdev_supported_types
ls /sys/class/mdev_bus/0000:84:00.0/mdev_supported_types: nvidia-35 nvidia-36 nvidia-37 nvidia-38 nvidia-39 nvidia-40 nvidia-41 nvidia-42 nvidia-43 nvidia-44 nvidia-45 ls /sys/class/mdev_bus/0000:85:00.0/mdev_supported_types: nvidia-35 nvidia-36 nvidia-37 nvidia-38 nvidia-39 nvidia-40 nvidia-41 nvidia-42 nvidia-43 nvidia-44 nvidia-45 ls /sys/class/mdev_bus/0000:86:00.0/mdev_supported_types: nvidia-35 nvidia-36 nvidia-37 nvidia-38 nvidia-39 nvidia-40 nvidia-41 nvidia-42 nvidia-43 nvidia-44 nvidia-45 ls /sys/class/mdev_bus/0000:87:00.0/mdev_supported_types: nvidia-35 nvidia-36 nvidia-37 nvidia-38 nvidia-39 nvidia-40 nvidia-41 nvidia-42 nvidia-43 nvidia-44 nvidia-45
parameter_defaults: ComputeGpuExtraConfig: nova::compute::vgpu::enabled_vgpu_types: - nvidia-35 - nvidia-36
parameter_defaults: ComputeGpuExtraConfig: nova::compute::vgpu::enabled_vgpu_types: - nvidia-35 - nvidia-36 NovaVGPUTypesDeviceAddressesMapping: {'vgpu_<vgpu_type>': ['<pci_address>', '<pci_address>'],'vgpu_<vgpu_type>': ['<pci_address>', '<pci_address>']}
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -r /home/stack/templates/roles_data_gpu.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/gpu.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml
(overcloud)USD openstack --os-placement-api-version 1.6 trait create CUSTOM_<TRAIT_NAME>
(overcloud)USD existing_traits=USD(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')
(overcloud)USD echo USDexisting_traits
(overcloud)USD openstack --os-placement-api-version 1.6 resource provider trait set USDexisting_traits --trait CUSTOM_<TRAIT_NAME> <host_uuid>
source ~/overcloudrc
(overcloud)USD openstack server create --flavor <flavor> --image <image> temp_vgpu_instance
(overcloud)USD openstack server image create --name vgpu_image temp_vgpu_instance
(overcloud)USD openstack flavor create --vcpus 6 --ram 8192 --disk 100 m1.small-gpu
(overcloud)USD openstack flavor set m1.small-gpu --property "resources:VGPU=1"
(overcloud)USD openstack flavor set m1.small-gpu --property trait:CUSTOM_NVIDIA_11=required
(overcloud)USD openstack server create --flavor m1.small-gpu --image vgpu_image --security-group web --nic net-id=internal0 --key-name lambda vgpu-instance
lspci -nn | grep <gpu_name>
lspci -nn | grep -i <gpu_name>
lspci -nn | grep -i nvidia 3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1) d8:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1db4] (rev a1)
lspci -v -s 3b:00.0 3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
parameter_defaults: NovaSchedulerEnabledFilters: - AvailabilityZoneFilter - ComputeFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter - ServerGroupAntiAffinityFilter - ServerGroupAffinityFilter - PciPassthroughFilter - NUMATopologyFilter
ControllerExtraConfig: nova::pci::aliases: - name: "t4" product_id: "1eb8" vendor_id: "10de" device_type: "type-PF" - name: "v100" product_id: "1db4" vendor_id: "10de" device_type: "type-PF"
ControllerExtraConfig: nova::pci::aliases: - name: "t4" product_id: "1eb8" vendor_id: "10de" - name: "v100" product_id: "1db4" vendor_id: "10de"
parameter_defaults: NovaPCIPassthrough: - vendor_id: "10de" product_id: "1eb8"
ComputeExtraConfig: nova::pci::aliases: - name: "t4" product_id: "1eb8" vendor_id: "10de" device_type: "type-PF" - name: "v100" product_id: "1db4" vendor_id: "10de" device_type: "type-PF"
ComputeExtraConfig: nova::pci::aliases: - name: "t4" product_id: "1eb8" vendor_id: "10de" - name: "v100" product_id: "1db4" vendor_id: "10de"
parameter_defaults: ComputeParameters: KernelArgs: "intel_iommu=on iommu=pt"
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/pci_passthru_controller.yaml -e /home/stack/templates/pci_passthru_compute.yaml
openstack flavor set m1.large --property "pci_passthrough:alias"="t4:2"
openstack server create --flavor m1.large --image <custom_gpu> --wait test-pci
lspci -nn | grep <gpu_name>
nvidia-smi
----------------------------------------------------------------------------- | NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 | |------------------------------- ---------------------- ----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |=============================== ====================== ======================| | 0 Tesla T4 Off | 00000000:01:00.0 Off | 0 | | N/A 43C P0 20W / 70W | 0MiB / 15109MiB | 0% Default | ------------------------------- ---------------------- ---------------------- ----------------------------------------------------------------------------- | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | -----------------------------------------------------------------------------
parameter_defaults: ComputeExtraConfig: nova::compute::force_config_drive: 'true'
parameter_defaults: ComputeExtraConfig: nova::compute::force_config_drive: 'true' nova::compute::config_drive_format: vfat
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml \
(overcloud)USD openstack server create --flavor m1.tiny --image cirros test-config-drive-instance
mkdir -p /mnt/config mount /dev/disk/by-label/config-2 /mnt/config
blkid -t LABEL="config-2" -odevice /dev/vdb mkdir -p /mnt/config mount /dev/vdb /mnt/config
parameter_defaults: ControllerExtraConfig: nova::vendordata::vendordata_providers: - DynamicJSON
parameter_defaults: ControllerExtraConfig: nova::vendordata::vendordata_providers: - DynamicJSON nova::vendordata::vendordata_dynamic_targets: "target1@http://127.0.0.1:125" nova::vendordata::vendordata_dynamic_targets: "target2@http://127.0.0.1:126"
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
[stack@director ~]USD source ~/stackrc
parameter_defaults: <Role>Parameters: KernelArgsDeferReboot: True
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/kernelargs_manual_reboot.yaml
(undercloud)USD source ~/overcloudrc (overcloud)USD openstack compute service list
(overcloud)USD openstack compute service set <node> nova-compute --disable
(overcloud)USD openstack server list --host <node_UUID> --all-projects
[tripleo-admin@overcloud-compute-0 ~]USD sudo reboot
(overcloud)USD openstack compute service set <node_UUID> nova-compute --enable
(overcloud)USD openstack compute service list
[stack@director ~]USD source ~/stackrc
parameter_defaults: NovaVNCProxySSLMinimumVersion: <version>
parameter_defaults: NovaVNCProxySSLCiphers: <ciphers>
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
parameter_defaults: ComputeParameters: NovaEnableVTPM: True
(undercloud)USD openstack overcloud deploy --templates -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(overcloud)USD openstack image create ... --property hw_tpm_version=2.0 vtpm-image
(overcloud)USD openstack image set --property hw_tpm_model=<tpm_model> vtpm-image
(overcloud)USD openstack server create --flavor m1.small --image vtpm-image vtpm-instance
dmesg | grep -i tpm
(overcloud)USD openstack flavor create --vcpus 1 --ram 512 --disk 2 --property hw:tpm_version=2.0 vtpm-flavor
(overcloud)USD openstack flavor set --property hw:tpm_model=<tpm_model> vtpm-flavor
(overcloud)USD openstack server create --flavor vtpm-flavor --image rhel-image vtpm-instance
dmesg | grep -i tpm
parameter_defaults: NovaCronArchiveDeleteRowsPurge: True
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
openstack port show <port_name/port_id>
(undercloud)USD source ~/overcloudrc (overcloud)USD openstack compute service list
(overcloud)USD openstack server list --host <source> --all-projects
(overcloud)USD openstack compute service set <source> nova-compute --disable
(overcloud)USD openstack server migrate <instance> --wait
(overcloud)USD openstack server list --all-projects
(overcloud)USD openstack server resize --confirm <instance>
(overcloud)USD openstack server resize --revert <instance>
(overcloud)USD openstack server start <instance>
(overcloud)USD openstack compute service set <source> nova-compute --enable
(overcloud)USD openstack server migrate <instance> --live-migration [--host <dest>] --wait
(overcloud)USD openstack server migrate <instance> --live-migration [--host <dest>] --wait --block-migration
(overcloud)USD openstack server show <instance> +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | ... | ... | | status | MIGRATING | | ... | ... | +----------------------+--------------------------------------+
(overcloud)USD openstack server list --host <dest> --all-projects
(overcloud)USD openstack compute service set <source> nova-compute --enable
openstack server migration list --server <instance> +----+-------------+----------- (...) | Id | Source Node | Dest Node | (...) +----+-------------+-----------+ (...) | 2 | - | - | (...) +----+-------------+-----------+ (...)
openstack server migration show <instance> <migration_id>
+------------------------+--------------------------------------+ | Property | Value | +------------------------+--------------------------------------+ | created_at | 2017-03-08T02:53:06.000000 | | dest_compute | controller | | dest_host | - | | dest_node | - | | disk_processed_bytes | 0 | | disk_remaining_bytes | 0 | | disk_total_bytes | 0 | | id | 2 | | memory_processed_bytes | 65502513 | | memory_remaining_bytes | 786427904 | | memory_total_bytes | 1091379200 | | server_uuid | d1df1b5a-70c4-4fed-98b7-423362f2c47c | | source_compute | compute2 | | source_node | - | | status | running | | updated_at | 2017-03-08T02:53:47.000000 | +------------------------+--------------------------------------+
(overcloud)USD openstack server list --host <node> --all-projects
(overcloud)USD openstack server show <instance> +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | ... | ... | | status | NONE | | ... | ... | +----------------------+--------------------------------------+
(overcloud)USD openstack baremetal node show <node>
(overcloud)USD openstack compute service set <node> nova-compute --disable --disable-reason <disable_host_reason>
(overcloud)USD openstack server evacuate [--host <dest>] [--password <password>] <instance>
(overcloud)[stack@director ~]USD openstack hypervisor list
(overcloud)USD openstack compute service set <node> nova-compute --enable
openstack server migration list --server <instance>
openstack server migration abort <instance> <migration_id>
openstack server migration list --server <instance>
openstack server migration force complete <instance> <migration_id>