Chapter 13. Configuring the overcloud with Ansible Ansible is the main method to apply the overcloud configuration. This chapter provides information about how to interact with the overcloud Ansible configuration. Although director generates the Ansible playbooks automatically, it is a good idea to familiarize yourself with Ansible syntax. For more information about using Ansible, see https://docs.ansible.com/ . Note Ansible also uses the concept of roles, which are different to OpenStack Platform director roles. Ansible roles form reusable components of playbooks, whereas director roles contain mappings of OpenStack services to node types. 13.1. Ansible-based overcloud configuration (config-download) The config-download feature is the method that director uses to configure the overcloud. Director uses config-download in conjunction with OpenStack Orchestration (heat) and OpenStack Workflow Service (mistral) to generate the software configuration and apply the configuration to each overcloud node. Although heat creates all deployment data from SoftwareDeployment resources to perform the overcloud installation and configuration, heat does not apply any of the configuration. Heat only provides the configuration data through the heat API. When director creates the stack, a mistral workflow queries the heat API to obtain the configuration data, generate a set of Ansible playbooks, and applies the Ansible playbooks to the overcloud. As a result, when you run the openstack overcloud deploy command, the following process occurs: Director creates a new deployment plan based on openstack-tripleo-heat-templates and includes any environment files and parameters to customize the plan. Director uses heat to interpret the deployment plan and create the overcloud stack and all descendant resources. This includes provisioning nodes with the OpenStack Bare Metal service (ironic). Heat also creates the software configuration from the deployment plan. Director compiles the Ansible playbooks from this software configuration. Director generates a temporary user ( tripleo-admin ) on the overcloud nodes specifically for Ansible SSH access. Director downloads the heat software configuration and generates a set of Ansible playbooks using heat outputs. Director applies the Ansible playbooks to the overcloud nodes using ansible-playbook . 13.2. config-download working directory Director generates a set of Ansible playbooks for the config-download process. These playbooks are stored in a working directory in the /var/lib/mistral/ . This directory is named after the name of the overcloud, which is overcloud by default. The working directory contains a set of sub-directories named after each overcloud role. These sub-directories contain all tasks relevant to the configuration of the nodes in the overcloud role. These sub-directories also contain additional sub-directories named after each specific node. These sub-directories contain node-specific variables to apply to the overcloud role tasks. As a result, the overcloud roles within the working directory use the following structure: Each working directory is a local Git repository that records changes after each deployment operation. Use the local Git repositories to track configuration changes between each deployment. 13.3. Enabling access to config-download working directories The mistral user in the OpenStack Workflow service (mistral) containers own all files in the /var/lib/mistral/ working directories. You can grant the stack user on the undercloud access to all files in this directory. This helps with performing certain operations within the directory. Procedure Use the setfacl command to grant the stack user on the undercloud access to the files in the /var/lib/mistral directory: This command retains mistral user access to the directory. 13.4. Checking config-download log During the config-download process, Ansible creates a log file on the undercloud in the config-download working directory. Procedure View the log with the less command within the config-download working directory. The following example uses the overcloud working directory: 13.5. Performing Git operations on the working directory The config-download working directory is a local Git repository. Every time a deployment operation runs, director adds a Git commit to the working directory with the relevant changes. You can perform Git operations to view configuration for the deployment at different stages and compare the configuration with different deployments. Be aware of the limitations of the working directory. For example, if you use Git to revert to a version of the config-download working directory, this action affects only the configuration in the working directory. It does not affect the following configurations: The overcloud data schema: Applying a version of the working directory software configuration does not undo data migration and schema changes. The hardware layout of the overcloud: Reverting to software configuration does not undo changes related to overcloud hardware, such as scaling up or down. The heat stack: Reverting to earlier revisions of the working directory has no effect on the configuration stored in the heat stack. The heat stack creates a new version of the software configuration that applies to the overcloud. To make permanent changes to the overcloud, modify the environment files applied to the overcloud stack before you rerun the openstack overcloud deploy command. Complete the following steps to compare different commits of the config-download working directory. Procedure Change to the config-download working directory for your overcloud. In this example, the working directory is for the overcloud named overcloud : Run the git log command to list the commits in your working directory. You can also format the log output to show the date: By default, the most recent commit appears first. Run the git diff command against two commit hashes to see all changes between the deployments: 13.6. Deployment methods that use config-download There are four main methods that use config-download in the context of an overcloud deployment: Standard deployment Run the openstack overcloud deploy command to automatically run the configuration stage after the provisioning stage. This is the default method when you run the openstack overcloud deploy command. Separate provisioning and configuration Run the openstack overcloud deploy command with specific options to separate the provisioning and configuration stages. Run the ansible-playbook-command.sh script after a deployment Run the openstack overcloud deploy command with combined or separate provisioning and configuration stages, then run the ansible-playbook-command.sh script supplied in the config-download working directory to re-apply the configuration stage. Provision nodes, manually create config-download, and run Ansible Run the openstack overcloud deploy command with a specific option to provision nodes, then run the ansible-playbook command with the deploy_steps_playbook.yaml playbook. 13.7. Running config-download on a standard deployment The default method for executing config-download is to run the openstack overcloud deploy command. This method suits most environments. Prerequisites A successful undercloud installation. Overcloud nodes ready for deployment. Heat environment files that are relevant to your specific overcloud customization. Procedure Log in to the undercloud host as the stack user. Source the stackrc file: Run the deployment command. Include any environment files that you require for your overcloud: Wait until the deployment process completes. During the deployment process, director generates the config-download files in a /var/lib/mistral/ working directory. After the deployment process finishes, view the Ansible playbooks in the working directory to see the tasks director executed to configure the overcloud. 13.8. Running config-download with separate provisioning and configuration The openstack overcloud deploy command runs the heat-based provisioning process and then the config-download configuration process. You can also run the deployment command to execute each process individually. Use this method to provision your overcloud nodes as a distinct process so that you can perform any manual pre-configuration tasks on the nodes before you run the overcloud configuration process. Prerequisites A successful undercloud installation. Overcloud nodes ready for deployment. Heat environment files that are relevant to your specific overcloud customization. Procedure Log in to the undercloud host as the stack user. Source the stackrc file: Run the deployment command with the --stack-only option. Include any environment files you require for your overcloud: Wait until the provisioning process completes. Enable SSH access from the undercloud to the overcloud for the tripleo-admin user. The config-download process uses the tripleo-admin user to perform the Ansible-based configuration: Perform any manual pre-configuration tasks on nodes. If you use Ansible for configuration, use the tripleo-admin user to access the nodes. Run the deployment command with the --config-download-only option. Include any environment files required for your overcloud: Wait until the configuration process completes. During the configuration stage, director generates the config-download files in a /var/lib/mistral/ working directory. After the deployment process finishes, view the Ansible playbooks in the working directory to see the tasks director executed to configure the overcloud. 13.9. Running config-download with the ansible-playbook-command.sh script When you deploy the overcloud, either with the standard method or a separate provisioning and configuration process, director generates a working directory in /var/lib/mistral/ . This directory contains the playbooks and scripts necessary to run the configuration process again. Prerequisites An overcloud deployed with the one of the following methods: Standard method that combines provisioning and configuration process Separate provisioning and configuration processes Procedure Log in to the undercloud host as the stack user. Change to the directory of the Ansible playbook: Change the owner of /var/lib/mistral/.ssh directory to the stack user. Run the ansible-playbook-command.sh command to run the overcloud configuration: Change the owner of /var/lib/mistral/.ssh directory to the mistral user. This is required to ensure that the ansible-playbook command running inside the mistral_executor container is successful. Run the script again as the mistral user. You can pass additional Ansible arguments to this script, which are then passed unchanged to the ansible-playbook command. This means that you can use other Ansible features, such as check mode ( --check ), limiting hosts ( --limit ), or overriding variables ( -e ). For example: Warning When --limit is used to deploy at scale, only hosts included in the execution are added to the SSH known_hosts file across the nodes. Therefore, some operations, such as live migration, may not work across nodes that are not in the known_hosts file. Note To ensure that the /etc/hosts file, on all nodes, is up-to-date, run the following command as the root user: Wait until the configuration process completes. Additional information The working directory contains a playbook called deploy_steps_playbook.yaml , which manages the overcloud configuration tasks. To view this playbook, run the following command: The playbook uses various task files contained in the working directory. Some task files are common to all OpenStack Platform roles and some are specific to certain OpenStack Platform roles and servers. The working directory also contains sub-directories that correspond to each role that you define in your overcloud roles_data file. For example: Each OpenStack Platform role directory also contains sub-directories for individual servers of that role type. The directories use the composable role hostname format: The Ansible tasks in deploy_steps_playbook.yaml are tagged. To see the full list of tags, use the CLI option --list-tags with ansible-playbook : Then apply tagged configuration using the --tags , --skip-tags , or --start-at-task with the ansible-playbook-command.sh script: When you run the config-download playbooks against the overcloud, you might receive a message regarding the SSH fingerprint for each host. To avoid these messages, include --ssh-common-args="-o StrictHostKeyChecking=no" when you run the ansible-playbook-command.sh script: 13.10. Running config-download with manually created playbooks You can create your own config-download files outside of the standard workflow. For example, you can run the openstack overcloud deploy command with the --stack-only option to provision the nodes, and then manually apply the Ansible configuration separately. Prerequisites A successful undercloud installation. Overcloud nodes ready for deployment. Heat environment files that are relevant to your specific overcloud customization. Procedure Log in to the undercloud host as the stack user. Source the stackrc file: Run the deployment command with the --stack-only option. Include any environment files required for your overcloud: Wait until the provisioning process completes. Enable SSH access from the undercloud to the overcloud for the tripleo-admin user. The config-download process uses the tripleo-admin user to perform the Ansible-based configuration: Generate the config-download files: --name is the name of the overcloud that you want to use for the Ansible file export. --config-dir is the location where you want to save the config-download files. Change to the directory that contains your config-download files: Generate a static inventory file: Replace <overcloud> with the name of your overcloud. Use the config-download files and the static inventory file to perform a configuration. To execute the deployment playbook, run the ansible-playbook command: Wait until the configuration process completes. To generate an overcloudrc file manually from this configuration, run the following command: Manually set the deployment status to success: Replace <OVERCLOUD> with the name of your overcloud. Additional information The config-download directory contains a playbook called deploy_steps_playbook.yaml , which runs the overcloud configuration. To view this playbook, run the following command: The playbook uses various task files contained in the working directory. Some task files are common to all OpenStack Platform roles and some are specific to certain OpenStack Platform roles and servers. The config-download directory also contains sub-directories that correspond to each role that you define in your overcloud roles_data file. For example: Each OpenStack Platform role directory also contains sub-directories for individual servers of that role type. The directories use the composable role hostname format: The Ansible tasks in deploy_steps_playbook.yaml are tagged. To see the full list of tags, use the CLI option --list-tags with ansible-playbook : Then apply tagged configuration using the --tags , --skip-tags , or --start-at-task with the ansible-playbook-command.sh script: When you run the config-download playbooks against the overcloud, you might receive a message regarding the SSH fingerprint for each host. To avoid these messages, include --ssh-common-args="-o StrictHostKeyChecking=no" to your ansible-playbook command: 13.11. Limitations of config-download The config-download feature has some limitations: When you use ansible-playbook CLI arguments such as --tags , --skip-tags , or --start-at-task , do not run or apply deployment configuration out of order. These CLI arguments are a convenient way to rerun previously failed tasks or to iterate over an initial deployment. However, to guarantee a consistent deployment, you must run all tasks from deploy_steps_playbook.yaml in order. You can not use the --start-at-task arguments for certain tasks that use a variable in the task name. For example, the --start-at-task arguments does not work for the following Ansible task: If your overcloud deployment includes a director-deployed Ceph Storage cluster, you cannot skip step1 tasks when you use the --check option unless you also skip external_deploy_steps tasks. You can set the number of parallel Ansible tasks with the --forks option. However, the performance of config-download operations degrades after 25 parallel tasks. For this reason, do not exceed 25 with the --forks option. 13.12. config-download top level files The following file are important top level files within a config-download working directory. Ansible configuration and execution The following files are specific to configuring and executing Ansible within the config-download working directory. ansible.cfg Configuration file used when running ansible-playbook . ansible.log Log file from the last run of ansible-playbook . ansible-errors.json JSON structured file that contains any deployment errors. ansible-playbook-command.sh Executable script to rerun the ansible-playbook command from the last deployment operation. ssh_private_key Private SSH key that Ansible uses to access the overcloud nodes. tripleo-ansible-inventory.yaml Ansible inventory file that contains hosts and variables for all the overcloud nodes. overcloud-config.tar.gz Archive of the working directory. Playbooks The following files are playbooks within the config-download working directory. deploy_steps_playbook.yaml Main deployment steps. This playbook performs the main configuration operations for your overcloud. pre_upgrade_rolling_steps_playbook.yaml Pre upgrade steps for major upgrade upgrade_steps_playbook.yaml Major upgrade steps. post_upgrade_steps_playbook.yaml Post upgrade steps for major upgrade. update_steps_playbook.yaml Minor update steps. fast_forward_upgrade_playbook.yaml Fast forward upgrade tasks. Use this playbook only when you want to upgrade from one long-life version of Red Hat OpenStack Platform to the . 13.13. config-download tags The playbooks use tagged tasks to control the tasks that they apply to the overcloud. Use tags with the ansible-playbook CLI arguments --tags or --skip-tags to control which tasks to execute. The following list contains information about the tags that are enabled by default: facts Fact gathering operations. common_roles Ansible roles common to all nodes. overcloud All plays for overcloud deployment. pre_deploy_steps Deployments that happen before the deploy_steps operations. host_prep_steps Host preparation steps. deploy_steps Deployment steps. post_deploy_steps Steps that happen after the deploy_steps operations. external All external deployment tasks. external_deploy_steps External deployment tasks that run on the undercloud only. 13.14. config-download deployment steps The deploy_steps_playbook.yaml playbook configures the overcloud. This playbook applies all software configuration that is necessary to deploy a full overcloud based on the overcloud deployment plan. This section contains a summary of the different Ansible plays used within this playbook. The play names in this section are the same names that are used within the playbook and that are displayed in the ansible-playbook output. This section also contains information about the Ansible tags that are set on each play. Gather facts from undercloud Fact gathering for the undercloud node. Tags: facts Gather facts from overcloud Fact gathering for the overcloud nodes. Tags: facts Load global variables Loads all variables from global_vars.yaml . Tags: always Common roles for TripleO servers Applies common Ansible roles to all overcloud nodes, including tripleo-bootstrap for installing bootstrap packages, and tripleo-ssh-known-hosts for configuring ssh known hosts. Tags: common_roles Overcloud deploy step tasks for step 0 Applies tasks from the deploy_steps_tasks template interface. Tags: overcloud , deploy_steps Server deployments Applies server-specific heat deployments for configuration such as networking and hieradata. Includes NetworkDeployment, <Role>Deployment, <Role>AllNodesDeployment, etc. Tags: overcloud , pre_deploy_steps Host prep steps Applies tasks from the host_prep_steps template interface. Tags: overcloud , host_prep_steps External deployment step [1,2,3,4,5] Applies tasks from the external_deploy_steps_tasks template interface. Ansible runs these tasks only against the undercloud node. Tags: external , external_deploy_steps Overcloud deploy step tasks for [1,2,3,4,5] Applies tasks from the deploy_steps_tasks template interface. Tags: overcloud , deploy_steps Overcloud common deploy step tasks [1,2,3,4,5] Applies the common tasks performed at each step, including puppet host configuration, container-puppet.py , and paunch (container configuration). Tags: overcloud , deploy_steps Server Post Deployments Applies server specific heat deployments for configuration performed after the 5-step deployment process. Tags: overcloud , post_deploy_steps External deployment Post Deploy tasks Applies tasks from the external_post_deploy_steps_tasks template interface. Ansible runs these tasks only against the undercloud node. Tags: external , external_deploy_steps

Commands:
─ /var/lib/mistral/overcloud | ├── Controller │ ├── overcloud-controller-0 | ├── overcloud-controller-1 │ └── overcloud-controller-2 ├── Compute │ ├── overcloud-compute-0 | ├── overcloud-compute-1 │ └── overcloud-compute-2
sudo setfacl -R -m u:stack:rwx /var/lib/mistral sudo chmod -R og-rwx /var/lib/mistral/.ssh
less /var/lib/mistral/overcloud/ansible.log
cd /var/lib/mistral/overcloud
git log --format=format:"%h%x09%cd%x09" a7e9063 Mon Oct 8 21:17:52 2018 +1000 dfb9d12 Fri Oct 5 20:23:44 2018 +1000 d0a910b Wed Oct 3 19:30:16 2018 +1000
git diff a7e9063 dfb9d12
source ~/stackrc
openstack overcloud deploy --templates -e environment-file1.yaml -e environment-file2.yaml
source ~/stackrc
openstack overcloud deploy --templates -e environment-file1.yaml -e environment-file2.yaml --stack-only
openstack overcloud admin authorize
openstack overcloud deploy --templates -e environment-file1.yaml -e environment-file2.yaml --config-download-only
cd /var/lib/mistral/overcloud/
sudo chown stack. -R /var/lib/mistral/.ssh/
sudo ./ansible-playbook-command.sh
sudo chown 42430:42430 -R /var/lib/mistral/.ssh/
./ansible-playbook-command.sh --limit Controller
(undercloud)USD sudo -i (undercloud)USD cd /var/lib/mistral/overcloud (undercloud)USD ANSIBLE_REMOTE_USER="tripleo-admin" ansible allovercloud -i tripleo-ansible-inventory.yaml -m include_role -a name=tripleo-hosts-entries -e @global_vars.yaml
less deploy_steps_playbook.yaml
ls Controller/
ls Controller/overcloud-controller-0
ansible-playbook -i tripleo-ansible-inventory.yaml --list-tags deploy_steps_playbook.yaml
./ansible-playbook-command.sh --tags overcloud
./ansible-playbook-command.sh --tags overcloud --ssh-common-args="-o StrictHostKeyChecking=no"
source ~/stackrc
openstack overcloud deploy --templates -e environment-file1.yaml -e environment-file2.yaml --stack-only
openstack overcloud admin authorize
openstack overcloud config download --name overcloud --config-dir ~/config-download
cd ~/config-download
tripleo-ansible-inventory --stack <overcloud> --ansible_ssh_user heat-admin --static-yaml-inventory inventory.yaml
ansible-playbook -i inventory.yaml -e gather_facts=true -e @global_vars.yaml --private-key ~/.ssh/id_rsa --become ~/config-download/deploy_steps_playbook.yaml
openstack action execution run --save-result --run-sync tripleo.deployment.overcloudrc '{"container":"overcloud"}' | jq -r '.["result"]["overcloudrc.v3"]' > overcloudrc.v3
openstack workflow execution create tripleo.deployment.v1.set_deployment_status_success '{"plan": "<OVERCLOUD>"}'
less deploy_steps_playbook.yaml
ls Controller/
ls Controller/overcloud-controller-0
ansible-playbook -i tripleo-ansible-inventory.yaml --list-tags deploy_steps_playbook.yaml
ansible-playbook -i inventory.yaml -e gather_facts=true -e @global_vars.yaml --private-key ~/.ssh/id_rsa --become --tags overcloud ~/config-download/deploy_steps_playbook.yaml
ansible-playbook -i inventory.yaml -e gather_facts=true -e @global_vars.yaml --private-key ~/.ssh/id_rsa --ssh-common-args="-o StrictHostKeyChecking=no" --become --tags overcloud ~/config-download/deploy_steps_playbook.yaml
- name: Run puppet host configuration for step {{ step }}