Chapter 5. Securing Red Hat OpenStack deployments with TLS and PKI Red Hat OpenStack Platform consists of many networks and endpoints that handle sensitive or confidential data that you can secure. When you use Transport Layer Security (TLS), you secure traffic with symmetric key encryption. The key and cipher are negotiated in the TLS handshake, which requires validation of the server's identity through a shared trust in an intermediary called a Certificate Authority (CA). Public Key Infrastructure (PKI) is a framework for validating an entity through a certificate authority. 5.1. Components of Public Key Infrastructure (PKI) The core components of PKI are shown in the folling table: Table 5.1. Key Terms Term Definition End entity The user, process, or system that validates itself through the use of a digital certificate. Certificate Authority (CA) The CA is an entity that is trusted by both the end entity, and the relying party that validates the end entity. Relying party The relying party receives the digital certificate as validation of the end entity, and has the capability of verifying the digital certificate. Digital certificates Signed public key certificates have a verifiable entity and a public key, and are issued by a CA. When a CA signs a certificate, it creates a message digest from the certificate encrypted with its private key. You can verify the signature using the public key associated with CA. The X.509 standard is used to define the certificates. Registration Authority (RA) An RA is an optional dedicated authority that can perform management functions such as authenticating end entities before they are issued a certificate by a CA. The CA authenticates end entities if there is no RA. Certificate Revocation List (CRL) A CRL is a list of certificate serial numbers that have been revoked. End entities presenting certificates with revoked serial numbers are not trusted in a PKI model. CRL issuer An optional system to which a CA delegates the publication of certificate revocation lists. Certificate Repository The location where the end entity certificates and certificate revocation lists are stored and queried. 5.2. Certificate Authority requirements and recommendations You must get certificates signed by a widely recognized certificate authority (CA) for publicly available Red Hat OpenStack Platform Dashboards or publicly accessible APIs. You must give a DNS domain or subdomain to each endpoint that you secure with TLS. The domains you provide are used to create the certificates issued by a CA. Customers access the dashboard or the API using the DNS name so that the CA can validate the endpoint. Red Hat recommends using a separate and internally managed CA to secure internal traffic. This allows the cloud deployer to maintain control of their Private Key Infrastructure (PKI) implementation and makes requesting, signing and deploying certificates for internal systems easier. You can enable SSL/TLS on your overcloud endpoints. Due to the number of certificates required to configure TLS everywhere (TLS-e), director integrates with a Red Hat Identity Management (IdM) server to act as a certificate authority and manage the overcloud certificates. For more information on configuring TLS-e, see Implementing TLS-e with Ansible . To check the status of TLS support across the OpenStack components, refer to the TLS Enablement status matrix . If want to you use a SSL certificate with your own certificate authority, see Enabling SSL/TLS on overcloud public endpoints . Note This will configure Red Hat OpenStack Platform with SSL/TLS on publicly accessible endpoints only. 5.3. Identifying TLS versions in your environment Important TLS version 1.0 is is deprecated for Red Hat OpenStack platform. Additionally, you must at minimum use TLS 1.2 for NIST-approval. For more information, see Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations . You can use cipherscan to determine the versions of TLS being presented by your deployment. Cipherscan can be cloned from https://github.com/mozilla/cipherscan . This example output demonstrates results received from horizon : Note Run cipherscan from a non-production system, as it might install additional dependencies when you first run it. Procedure Run cipher scan against the accessible URL of the Dashboard service: When scanning a server, Cipherscan advertises support for a specific TLS version, which is the highest TLS version it is willing to negotiate. If the target server correctly follows TLS protocol, it will respond with the highest version that is mutually supported, which may be lower than what Cipherscan initially advertised. If the server does proceed to establish a connection with the client using that specific version, it is not considered to be intolerant to that protocol version. If it does not establish the connection (with the specified version, or any lower version), then intolerance for that version of protocol is considered to be present. For example: In this output, intolerance of TLS 1.0 and TLS 1.1 is reported as PRESENT , meaning that the connection could not be established, and that Cipherscan was unable to connect while advertising support for those TLS versions. As a result, it is reasonable to conclude that those (and any lower) versions of the protocol are not enabled on the scanned server. 5.4. Identity Management (IdM) server recommendations for OpenStack Red Hat provides the following information to help you integrate your IdM server and OpenStack environment. For information on preparing Red Hat Enterprise Linux for an IdM installation, see Installing Identity Management . Run the ipa-server-install command to install and configure IdM. You can use command parameters to skip interactive prompts. Use the following recommendations so that your IdM server can integrate with your Red Hat OpenStack Platform environment: Table 5.2. Parameter recommendations Option Recommendation --admin-password Note the value you provide. You will need this password when configuring Red Hat OpenStack Platform to work with IdM. --ip-address Note the value you provide. The undercloud and overcloud nodes require network access to this ip address. --setup-dns Use this option to install an integrated DNS service on the IdM server. The undercloud and overcloud nodes use the IdM server for domain name resolution. --auto-forwarders Use this option to use the addresses in /etc/resolv.conf as DNS forwarders. --auto-reverse Use this option to resolve reverse records and zones for the IdM server IP addresses. If neither reverse records or zones are resolvable, IdM creates the reverse zones. This simplifies the IdM deployment. --ntp-server , --ntp-pool You can use both or either of these options to configure your NTP source. Both the IdM server and your OpenStack environment must have correct and synchronized time. You must open the firewall ports required by IdM to enable communication with Red Hat OpenStack Platform nodes. For more information, see Opening the ports required by IdM . Additional resources Configuring and Managing Identity Management Red Hat Identity Management Documentation 5.5. Implementing TLS-e with Ansible You can use the new tripleo-ipa method to enable SSL/TLS on overcloud endpoints, called TLS everywhere (TLS-e). Due to the number of certificates required, Red Hat OpenStack Platform integrates with Red Hat Identity management (IdM). When you use tripleo-ipa to configure TLS-e, IdM is the certificate authority. Prerequisites Ensure that all configuration steps for the undercloud, such as the creation of the stack user, are complete. For more details, see Director Installation and Usage for more details The IP address of your DNS server is configured on the undercloud to the IP address of the IdM server. One of the following parameters must be configured in your undercloud.conf file: DEFAULT/undercloud_nameservers %SUBNET_SECTION%/dns_nameservers Procedure Use the following procedure to implement TLS-e on a new installation of Red Hat OpenStack Platform, or an existing deployment that you want to configure with TLS-e. You must use this method if you deploy Red Hat OpenStack Platform with TLS-e on pre-provisioned nodes. Note If you are implementing TLS-e for an existing environment, you are required to run commands such as openstack undercloud install , and openstack overcloud deploy . These procedures are idempotent and only adjust your existing deployment configuration to match updated templates and configuration files. Configure the /etc/resolv.conf file: Set the appropriate search domains and the nameserver on the undercloud in /etc/resolv.conf . For example, if the deployment domain is example.com , and the domain of the FreeIPA server is bigcorp.com , then add the following lines to /etc/resolv.conf: Install required software: Export environmental variables with values specific to your environment.: 1 2 The IdM user credentials are an administrative user that can add new hosts and services. 3 The value of the UNDERCLOUD_FQDN parameter matches the first hostname-to-IP address mapping in /etc/hosts . Run the undercloud-ipa-install.yaml ansible playbook on the undercloud: Add the following parameters to undercloud.conf [Optional] If your IPA realm does not match your IPA domain, set the value of the certmonger_krb_realm parameter: Set the value of the certmonger_krb_realm in /home/stack/hiera_override.yaml : Set the value of the custom_env_files parameter in undercloud.conf to /home/stack/hiera_override.yaml : Deploy the undercloud: Verification Verify that the undercloud was enrolled correctly by completing the following steps: List the hosts in IdM: Confirm that /etc/novajoin/krb5.keytab exists on the undercloud. Note The novajoin directory name is for legacy naming purposes only. Configuring TLS-e on the overcloud When you deploy the overcloud with TLS everywhere (TLS-e), IP addresses from the Undercloud and Overcloud will automatically be registered with IdM. Before deploying the overcloud, create a YAML file tls-parameters.yaml with contents similar to the following. The values you select will be specific for your environment: The shown value of the OS::TripleO::Services::IpaClient parameter overrides the default setting in the enable-internal-tls.yaml file. You must ensure the tls-parameters.yaml file follows enable-internal-tls.yaml in the openstack overcloud deploy command. For more information about the parameters that you use to implement TLS-e, see Parameters for tripleo-ipa . Deploy the overcloud. You will need to include the tls-parameters.yaml in the deployment command: Confirm each endpoint is using HTTPS by querying keystone for a list of endpoints: 5.6. Parameters for tripleo-ipa Use the fully qualified domain name (FQDN) of your cloud to define the cloud name and cloud domain parameters required for tripleo-ipa . For example, with an FQDN of overcloud.example.com , use the following values: CloudDomain: example.com CloudName: overcloud.example.com CloudNameCtlplane: overcloud.ctlplane.example.com CloudNameInternal: overcloud.internalapi.example.com CloudNameStorage: overcloud.storage.example.com CloudNameStorageManagement: overcloud.storagemgmt.example.com Set the following additional parameters based on the requirements of your environment: CertmongerKerberosRealm Set CertmongerKerberosRealm parameter to the value of the IPA realm. This is required if the IPA realm does not match the IPA domain. DnsSearchDomains The DnsSearchDomains parameter is a comma-separated list. If the domain of the IdM server is different than the cloud domain, include the domain of the IdM server in the DnsSearchDomains parameter. EnableEtcdInternalTLS If you deploy TLSe on a distributed compute node (DCN) architecture, you must add the EnableEtcdInternalTLS parameter with the value of True . IDMInstallClientPackages If you have preprovisioned your compute nodes, set the IDMInstallClientPackages parameter to a value of True . Otherwise, set the value to False . IDMModifyDNS Set the IDMModifyDNS parameter to false to disable automatic IP registration of the overcloud nodes on Red Hat Identity Server. IdmDomain Set the IdmDomain parameter to the domain portion of the FQDN of your Red Hat Identity server. The value that you specify is also used as the value of the IdM realm. If the IdM domain and IdM realm differ, set the realm explicitly using the CertmongerKerberosRealm parameter. IdmServer Set the IdmServer parameter to the FQDN of your Red Hat Identity server. If you use a replicated IdM environment, then set multiple values using a comma delimited list. For more information on IdM replicas, see Installing an IdM replica . 5.7. Encrypting memcached traffic under TLS everywhere (TLS-e) Important This feature is available in this release as a Technology Preview , and therefore is not fully supported by Red Hat. It should only be used for testing, and should not be deployed in a production environment. For more information about Technology Preview features, see Scope of Coverage Details . You can now encrypt memcached traffic with TLS-e. This feature works with both novajoin and tripleo-ipa: Create an environment file called memcached.yaml with the following contents to add TLS support for memcached: Include the memcached.yaml environment file in the overcloud deployment process: Additional Resources For more information about deploying TLSe with tripleo-ipa, see Implementing TLS-e with Ansible . 5.8. Increasing the size of private keys You can improve security by increasing the the size of private keys used to create certificates for encrypted service traffic. The default RHOSP private key size of 2048 bits matches the National Institute of Standards and Technology (NIST) recommended minimum. Use the CertificateKeySize parameter to change the size of private keys globally. Use a service specific parameter such as RedisCertificateKeySize , to modify a specific private key, or to override the global CertificateKeySize parameter. Use these parameters in an environment heat template, and include the template in your overcloud deployment command. If you have already deployed the overcloud, you must rerun the same openstack overcloud deploy command with the same templates you originally used, and include the new parameters for the changes to take effect. In the following example, the global value for private keys is 4096 . The private key for redis is 2048 because the RedisCertificateKeySize overrides the global parameter: Example 5.9. Replacing the IdM server for Red Hat OpenStack Platform with its replica When you replace an existing IPA server with its replica, you must update the necessary parameters. Failure to do so results in failed overcloud deployments when you update your cluster's configuration. Procedure On each overcloud node, edit the /etc/ipa/default.conf configuration file to ensure that the server and xmlrpc_uri parameters include the fully qualified domain name (FQDN) of the IdM server: On the undercloud, edit the /home/stack/templates/tls-parameters.yaml environment file and ensure that the IPA_SERVER_HOSTNAME parameter matches the FQDN that is shown in the xmlrpc_uri and server parameters in /etc/ipa/default.conf . Ensure that all parameters match your environment:

Commands:
./cipherscan https://openstack.lab.local ........................... Target: openstack.lab.local:443 prio ciphersuite protocols pfs curves 1 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1 2 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1 3 DHE-RSA-AES128-GCM-SHA256 TLSv1.2 DH,1024bits None 4 DHE-RSA-AES256-GCM-SHA384 TLSv1.2 DH,1024bits None 5 ECDHE-RSA-AES128-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1 6 ECDHE-RSA-AES256-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1 7 ECDHE-RSA-AES128-SHA TLSv1.2 ECDH,P-256,256bits prime256v1 8 ECDHE-RSA-AES256-SHA TLSv1.2 ECDH,P-256,256bits prime256v1 9 DHE-RSA-AES128-SHA256 TLSv1.2 DH,1024bits None 10 DHE-RSA-AES128-SHA TLSv1.2 DH,1024bits None 11 DHE-RSA-AES256-SHA256 TLSv1.2 DH,1024bits None 12 DHE-RSA-AES256-SHA TLSv1.2 DH,1024bits None 13 ECDHE-RSA-DES-CBC3-SHA TLSv1.2 ECDH,P-256,256bits prime256v1 14 EDH-RSA-DES-CBC3-SHA TLSv1.2 DH,1024bits None 15 AES128-GCM-SHA256 TLSv1.2 None None 16 AES256-GCM-SHA384 TLSv1.2 None None 17 AES128-SHA256 TLSv1.2 None None 18 AES256-SHA256 TLSv1.2 None None 19 AES128-SHA TLSv1.2 None None 20 AES256-SHA TLSv1.2 None None 21 DES-CBC3-SHA TLSv1.2 None None Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature TLS ticket lifetime hint: None NPN protocols: None OCSP stapling: not supported Cipher ordering: server Curves ordering: server - fallback: no Server supports secure renegotiation Server supported compression methods: NONE TLS Tolerance: yes Intolerance to: SSL 3.254 : absent TLS 1.0 : PRESENT TLS 1.1 : PRESENT TLS 1.2 : absent TLS 1.3 : absent TLS 1.4 : absent
Intolerance to: SSL 3.254 : absent TLS 1.0 : PRESENT TLS 1.1 : PRESENT TLS 1.2 : absent TLS 1.3 : absent TLS 1.4 : absent
search example.com bigcorp.com nameserver USDIDM_SERVER_IP_ADDR
sudo dnf install -y python3-ipalib python3-ipaclient krb5-devel
export IPA_DOMAIN=bigcorp.com export IPA_REALM=BIGCORP.COM export IPA_ADMIN_USER=USDIPA_USER 1 export IPA_ADMIN_PASSWORD=USDIPA_PASSWORD 2 export IPA_SERVER_HOSTNAME=ipa.bigcorp.com export UNDERCLOUD_FQDN=undercloud.example.com 3 export USER=stack export CLOUD_DOMAIN=example.com
ansible-playbook --ssh-extra-args "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" /usr/share/ansible/tripleo-playbooks/undercloud-ipa-install.yaml
undercloud_nameservers = USDIDM_SERVER_IP_ADDR overcloud_domain_name = example.com
parameter_defaults: certmonger_krb_realm = EXAMPLE.COMPANY.COM
custom_env_files = /home/stack/hiera_override.yaml
openstack undercloud install
kinit admin ipa host-find
ls /etc/novajoin/krb5.keytab
parameter_defaults: DnsSearchDomains: ["example.com"] CloudDomain: example.com CloudName: overcloud.example.com CloudNameInternal: overcloud.internalapi.example.com CloudNameStorage: overcloud.storage.example.com CloudNameStorageManagement: overcloud.storagemgmt.example.com CloudNameCtlplane: overcloud.ctlplane.example.com IdMServer: freeipa-0.redhat.local IdMDomain: redhat.local IdMInstallClientPackages: False resource_registry: OS::TripleO::Services::IpaClient: /usr/share/openstack-tripleo-heat-templates/deployment/ipa/ipaservices-baremetal-ansible.yaml
DEFAULT_TEMPLATES=/usr/share/openstack-tripleo-heat-templates/ CUSTOM_TEMPLATES=/home/stack/templates openstack overcloud deploy -e USD{DEFAULT_TEMPLATES}/environments/ssl/tls-everywhere-endpoints-dns.yaml -e USD{DEFAULT_TEMPLATES}/environments/services/haproxy-public-tls-certmonger.yaml -e USD{DEFAULT_TEMPLATES}/environments/ssl/enable-internal-tls.yaml -e USD{CUSTOM_TEMPLATES}/tls-parameters.yaml
openstack endpoint list
parameter_defaults: MemcachedTLS: true MemcachedPort: 11212
openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml -e /home/stack/memcached.yaml
parameter_defaults: CertificateKeySize: '4096' RedisCertificateKeySize: '2048'
#File modified by ipa-client-install [global] basedn = dc=redhat,dc=local realm = REDHAT.LOCAL domain = redhat.local server = freeipa-0.redhat.local host = undercloud-0.redhat.local xmlrpc_uri = https://freeipa-0.redhat.local/ipa/xml enable_ra = True
export IPA_DOMAIN=bigcorp.com export IPA_REALM=BIGCORP.COM export IPA_ADMIN_USER=USDIPA_USER export IPA_ADMIN_PASSWORD=USDIPA_PASSWORD export IPA_SERVER_HOSTNAME=ipa.bigcorp.com export UNDERCLOUD_FQDN=undercloud.example.com export USER=stack export CLOUD_DOMAIN=example.com