Chapter 2. Running Tempest tests using Test Operator Use the Red Hat OpenStack Services on OpenShift (RHOSO) test-operator when working with Tempest tests. Prerequisites A deployed RHOSO environment. Ensure that you are in the OpenStack project: 2.1. Tempest Custom Resources configuration file This test-v1beta1-tempest.yaml file is an example of a Tempest custom resource (CR) that you can edit and use to execute Tempest tests with test-operator . Warning Use the privileged parameter with caution. A value of true can create security vulnerabilities. 2.2. Installing Test Operator using OperatorHub Install the test-operator in the openstack-operators project using OperatorHub . Procedure Log in to the Red Hat OpenShift Container Platform (RHOCP) web console as a user with cluster-admin permissions. Select Operators > OperatorHub . Optional: In the Filter by keyword field, begin typing OpenStack to filter the list of Operators. Select the OpenStack Test Operator tile with the Red Hat source label from the list of Operators. Read the information about the Operator and click Install . On the page with the heading Install Operator located at OperatorHub > Operator Installation , select Select a Namespace in the Installed Namespace section. Optional: In the Select Project field, begin typing openstack-operators to filter the list of Projects. Select openstack-operators from the list of Projects. Click Install to make the Operator available to the openstack-operators namespace. Wait for the Operator to install and then click View Operator to check that the Status of the OpenStack Test Operator is Succeeded . Verification When the test-operator-controller-manager pod successfully spawns and the pod is running, you can communicate with the operator using the custom resources (CRs) that the test-operator accepts: 2.3. Running Tempest tests Select and run an image to use for Tempest tests. The following file names are examples and might vary from your environment. Procedure Edit the Tempest test configuration file, for example with vim : Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Add the appropriate value for the containerImage parameter: The openstack-tempest-all:current-podified image in this example contains all the default supported plug-ins. Save and close the Tempest test configuration file. Create the pod and run your Tempest tests: Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Verification Check that the pod is running: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. 2.4. Finding Tempest logs You can access the Tempest logs, for example for a test that successfully completed, or to troubleshoot a pod that has failed. Caution The test pods that run in the openstack namespace have access to the secrets in that namespace, such as clouds.yaml . These secrets might propagate into test logs stored on persistent volume claims (PVCs) created by the test-operator . Procedure Get the name and status of the relevant pod: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Get the logs: Replace <pod_name> with the name of the pod that you got in the step. Verification View the logs. 2.5. Getting logs from inside the pod You can access the Tempest logs, for example, for a test that successfully completed, or to troubleshoot a pod that has failed. You can access specific and more detailed Tempest logs from inside the pod. Procedure Get the name and status of the relevant pod: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Access the pod: Replace <pod_name> with the name of the pod that you got in the step. View available log files inside the pod: View available log files in the required directory: Replace <tempest-tests> with the name of the relevant directory that you want to view logs in, for example tempest-tests . Verification View the logs. 2.6. Re-running Tempest tests Modify the Tempest configuration file and re-run the Tempest tests. Procedure Get the name and status of the relevant pod: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Note If the pod is still active, you can wait for the test to complete before proceeding to the following step. Get the name of the Tempest custom resource (CR): Delete the Tempest CR: Replace <tempest_cr> with the name of the Tempest CR that you got in the step. Verify that you deleted the pod: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Edit the Tempest test configuration file, for example with vim : Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Make the required edits to the Tempest test configuration file, for example you can modify the excludeList: parameter: Replace <excludeList_value> with the excludeList value that you want to test, such as tempest.api.identity.v3.* . Save and close the Tempest test configuration file. Create the pod for your Tempest tests: Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Verification Get the name of the pod that you created in the step: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Check in the logs for the expected changes: Replace <pod_name> with the name of the relevant pod that you got in the step and replace <excludeList_value> with the excludeList value that you added to the Tempest test configuration file, such as tempest.api.identity.v3.* . 2.7. Installing external plug-ins You can install external plug-ins, such as barbican-tempest-plugin . Note The barbican-tempest-plugin is included with the image registry.redhat.io/rhoso/openstack-tempest-all-rhel9:18.0 and is shown in the following procedure as an example. If you are using external plug-ins that are unsupported, ensure that you proceed with caution. Procedure Edit the Tempest test configuration file, for example with vim : Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Add externalPlugin option, or uncomment the relevant lines in your Tempest test configuration file: Save and close the Tempest test configuration file. Create the new pod for the Tempest tests: Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Verification Get the name and status of the pod that you created in the step: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. 2.8. Fixing pod in pending state You can use the following procedure to fix a pod that is in a Pending state caused by a lack of available persistent volumes. Procedure Get the name of the relevant pod and verify it has a status of Pending : Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Confirm that the Pending status is caused by a lack of available persistent volumes: Replace <pod_name> with the name of the pod that you got in the step. List all persistent volumes that are associated with Tempest: Edit one of the persistent volumes to change the claim reference value to null : Replace <name_of_volume> with the name of one of the Tempest volume that you got from the step. Verification Confirm that the volume that you edited has changed from Released to Bound : Confirm that status of the pod has changed from Pending : Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. 2.9. Using debug mode With debug mode, you can keep the pod running if the test finishes or in case of a failure, and use a remote shell to get more information and detail. Procedure Edit the Tempest test configuration file, for example with vim : Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Change the value of debug: parameter to true , or add the line debug: true to the configuration file: Save and close the Tempest test configuration file. Create the new pod for the Tempest tests: Replace <Tempest_config> with the name of your Tempest test configuration file, such as test_v1beta1_tempest.yaml . Verification Get the name of the pod that you created in the step: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Access the pod remotely: Replace <pod_name> with the name of the pod that you got in the step. Make changes or check errors in the running pod: 2.10. Using pudb to debug Tempest tests You can use pudb to create and customize breakpoints that you can use to debug your Tempest tests. Prerequisites You have configured debug mode. For more information about debug mode, see Section 2.9, "Using debug mode" . Procedure Get the name of the pod that you want to use pudb with: Replace <pod_name> with the name that you specified in your Tempest Custom Resources configuration file, for example tempest-tests , or you can just use USD oc get pods and search for the relevant pod. Access the pod remotely: Replace <pod_name> with the name of the pod that you got in the step. Navigate to the correct directory: Create a Python3 lightweight virtual environment: Activate the Python3 lightweight virtual environment: Download and install pudb in the Python3 lightweight virtual environment: Find the path to the file that you want to debug, for example test_networks.py : Open your chosen file for editing: Insert the line import pudb; pu.db into the file where you want to create a pudb breakpoint, and save and close the file. Change the ownership: Run the test with the pudb breakpoint: Verification The pudb interface opens. You can interact with the pudb interface before the test completes.

Commands:
oc project openstack Now using project "openstack" on server "https://api.crc.testing:6443".
apiVersion: test.openstack.org/v1beta1 kind: Tempest metadata: name: tempest-tests namespace: openstack spec: containerImage: "" # storageClass: local-storage # parallel: false # debug: false # configOverwrite # --------------- # An interface to overwrite default config files like e.g. logging.conf But can also # be used to add additional files. Those get added to the service config dir in # /etc/test_operator/<file> # # configOverwrite: # file.txt: | # content of the file # SSHKeySecretName # ---------------- # SSHKeySecretName is the name of the k8s secret that contains an ssh key. The key is # mounted to ~/.ssh/id_ecdsa in the tempest pod. Note, the test-operator looks for # the private key in ssh-privatekey field of the secret. # # SSHKeySecretName: secret_name # Privileged # ---------- # When the privileged parameter has the default value of false, the test pods spawn with # allowedPrivilegedEscalation: false and without NET_ADMIN, NET_RAW, and the default capabilities. # You must set the value of the privileged parameter to true for some test-operator functionalities, # such as extraRPMs in Tempest CR, or some set tobiko tests, but setting privileged: true can decrease security. # privileged: false tempestRun: # NOTE: All parameters have default values (use only when you want to override # the default behaviour) includeList: | # <-- Use | to preserve \n tempest.api.identity.v3.* concurrency: 8 # excludeList: | # <-- Use | to preserve \n # tempest.api.identity.v3.* # workerFile: | # <-- Use | to preserve \n # - worker: # - tempest.api.* # - neutron_tempest_tests # - worker: # - tempest.scenario.* # smoke: false # serial: false # parallel: true # externalPlugin: # - repository: "https://opendev.org/openstack/barbican-tempest-plugin.git" # - repository: "https://opendev.org/openstack/neutron-tempest-plugin.git" # changeRepository: "https://review.opendev.org/openstack/neutron-tempest-plugin" # changeRefspec: "refs/changes/97/896397/2" # extraImages: # - URL: https://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img # name: cirros-0.6.2-test-operator # flavor: # name: cirros-0.6.2-test-operator-flavor # RAM: 512 # disk: 20 # vcpus: 1 # extraRPMs: # ---------- # A list of URLs that point to RPMs that should be installed before # the execution of tempest. WARNING! This parameter has no efect when used # in combination with externalPlugin parameter. # extraRPMs: # - https://cbs.centos.org/kojifiles/packages/python-sshtunnel/0.4.0/12.el9s/noarch/python3-sshtunnel-0.4.0-12.el9s.noarch.rpm # - https://cbs.centos.org/kojifiles/packages/python-whitebox-tests-tempest/0.0.3/0.1.766ff04git.el9s/noarch/python3-whitebox-tests-tempest-0.0.3-0.1.766ff04git.el9s.noarch.rpm tempestconfRun: # NOTE: All parameters have default values (use only when you want to override # the default behaviour) # create: true # collectTiming: false # insecure: false # noDefaultDeployer: false # debug: false # verbose: false # nonAdmin: false # retryImage: false # convertToRaw: false # out: ./etc/tempest.conf # flavorMinMem: 128 # flavorMinDisk: 1 # timeout: 600 # imageDiskFormat: qcow2 # image: https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img # The following text will be mounted to the tempest pod # as /etc/test_operator/deployer_input.yaml # deployerInput: | # [section] # value1 = exmaple_value2 # value2 = example_value2 # The following text will be mounted to the tempest pod # as /etc/test_operator/accounts.yaml # testAccounts: | # - username: 'multi_role_user' # tenant_name: 'test_tenant_42' # password: 'test_password' # roles: # - 'fun_role' # - 'not_an_admin' # - 'an_admin' # The following text will be mounted to the tempest pod # as /etc/test_operator/profile.yaml # profile: | # collect_timing: false # create: false # create_accounts_file: null # createAccountsFile: /path/to/accounts.yaml # generateProfile: /path/to/profile.yaml # networkID: # append: | # <-- Use | to preserve \n # section1.name1 value1 # section1.name1 value2 # remove: | # <-- Use | to preserve \n # section1.name1 value1 # section1.name1 value2 # overrides: | # <-- Use | to preserve \n # overrides_section1.name1 value1 # overrides_section1.name1 value2 # Workflow # -------- # Workflow section can be utilized to spawn multiple test pods at the same time. # The commented out example spawns two test pods that are executed sequentially. # Each step inherits all configuration that is specified outside of the workflow # field. For each step you can overwrite values specified in the tempestRun and # tempestconfRun sections. # # workflow: # - stepName: firstStep # tempestRun: # includeList: | # tempest.api.* # - stepName: secondStep # tempestRun: # includeList: | # neutron_tempest_plugin.*
oc get pods -n openstack-operators
vim <Tempest_config>
registry.redhat.io/rhoso/openstack-tempest-all-rhel9:18.0
oc apply -f <Tempest_config>
oc get pods | grep -i <pod_name>
oc get pods | grep -i <pod_name>
oc logs <pod_name>
oc get pods | grep -i <pod_name>
oc debug <pod_name>
sh-5.1USD ls -lah /var/lib/tempest/external_files
sh-5.1USD ls -lah /var/lib/tempest/external_files/<tempest-tests>
oc get pods | grep -i <pod_name>
oc get tempest
oc delete tempest <tempest_cr>
oc get pods | grep -i <pod_name>
vim <Tempest_config>
excludeList: | # <-- Use | to preserve \n <excludeList_value>
oc apply -f <Tempest_config>
oc get pods | grep -i <pod_name>
oc logs <pod_name> | grep <excludeList_value> --context=4
vim <Tempest_config>
externalPlugin: - repository: "https://opendev.org/openstack/barbican-tempest-plugin.git"
oc apply -f <Tempest_config>
oc get pods | grep -i <pod_name>
oc get pods | grep -i <pod_name>
oc describe pod <pod_name>
oc get pv | grep -i tempest
oc patch pv <name_of_volume> -p '{"spec":{"claimRef":null}}'
oc get pv | grep -i tempest
oc get pods | grep -i <pod_name>
vim <Tempest_config>
apiVersion: test.openstack.org/v1beta1 kind: Tempest metadata: name: tempest-tests namespace: openstack spec: containerImage: registry.redhat.io/rhoso/openstack-tempest-all-rhel9:18.0 debug: true
oc apply -f <Tempest_config>
oc get pods | grep -i <pod_name>
oc rsh <pod_name>
sh-5.1USD ls -lah /var/lib/tempest
oc get pods | grep -i <pod_name>
oc rsh <pod_name>
sh-5.1USD cd /var/lib/tempest/openshift
sh-5.1USD python3 -m venv --system-site-packages .venv
sh-5.1USD . .venv/bin/activate
(.venv) sh-5.1USD pip install pudb
(.venv) sh-5.1USD find / -name test_networks.py 2> /dev/null
(.venv) sh-5.1USD sudo vi /usr/lib/python3.9/site-packages/tempest/api/network/test_networks.py
(.venv) sh-5.1 USD sudo chown -R tempest:tempest /var/lib/tempest/.config
(.venv) sh-5.1 USD python -m testtools.run tempest.api.network.test_networks.NetworksTest.test_list_networks