Chapter 5. Customizing the Image service (glance) When you have deployed the Image service (glance), you can customize the image import workflow and configure scalability and security features. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. 5.1. Configuring the image import workflow Users can upload their own images to the Image service (glance) by using the default glance-direct or web-download import methods. If you have multiple Red Hat Ceph Storage back ends for the Image service, you can also enable the copy-image import method. You can monitor uploaded images in a staging area before they go active in a storage back end, and you can configure the import workflow to run plugins to make user images discoverable, for example, the Inject Image Metadata plugin for metadata or the Image Conversion plugin for image formats. 5.1.1. Distributed image import Distributed image import works with the glance-direct image import method, and it is enabled by default in Red Hat OpenStack Services on OpenShift (RHOSO). When using the glance-direct image import method, users can upload local images to the Image service (glance) without any requirement for a shared storage area where API worker nodes (or replicas) stage images. Instead, staging is distributed because individual API workers have their own local and unshared staging directory. The API worker that owns the image data is the same API worker that performs the image import. When the image is created and staged, the Image service records the URL of the staging API worker in a database. With this URL, other API workers can proxy image import requests from clients to the worker that has the image data and can perform the import operation. This workflow allows API worker nodes to be isolated for High Availability (HA) and distributed geographically for a distributed compute node (DCN) environment. 5.1.2. URI allowlist and blocklist for web-download sources You can limit the sources of web-download image imports by specifying a URI allowlist and blocklist in the glance template in your OpenStackControlPlane custom resource (CR) file. You can allow or block image source URIs at three levels: scheme (allowed_schemes, disallowed_schemes) host (allowed_hosts, disallowed_hosts) port (allowed_ports, disallowed_ports) If you specify both an allowlist and a blocklist at any level, the allowlist is honored and the blocklist is ignored. For an example of a URI allowlist, see Configuring a URI allowlist for web-import sources . Decision logic for URI validation The Image service applies the following decision logic to validate image source URIs: The scheme is checked. Missing scheme: reject If there is an allowlist, and the scheme is not present in the allowlist: reject. Otherwise, skip C and continue on to 2. If there is a blocklist, and the scheme is present in the blocklist: reject. The host name is checked. Missing host name: reject If there is an allowlist, and the host name is not present in the allowlist: reject. Otherwise, skip C and continue on to 3. If there is a blocklist, and the host name is present in the blocklist: reject. If there is a port in the URI, the port is checked. If there is a allowlist, and the port is not present in the allowlist: reject. Otherwise, skip B and continue on to 4. If there is a blocklist, and the port is present in the blocklist: reject. The URI is accepted as valid. If you allow a scheme, either by adding it to an allowlist or by not adding it to a blocklist, any URI that uses the default port for that scheme by not including a port is allowed. If the URI does include a port, the URI is validated according to the default decision logic. 5.1.2.1. Default settings for the URI allowlist and blocklist The following allowlist and blocklist values are the default settings for the web-download image import method in your Red Hat OpenStack Services on OpenShift (RHOSO) deployment. allowed_schemes - [ http , https ] disallowed_schemes - empty list allowed_hosts - empty list disallowed_hosts - empty list allowed_ports - [80, 443] disallowed_ports - empty list If you use the default values, users can only access URIs by using the http or https scheme, and they can only specify ports 80 and 443 . Users do not have to specify a port, but if they do, it must be either 80 or 443 . 5.1.2.2. Configuring a URI allowlist for web-import sources You configure the sources of web-import image downloads by specifying URI allowlists and blocklists in the glance template in your OpenStackControlPlane custom resource (CR) file. In this example, you are using an FTP server for image upload. The default port for FTP is 21. Procedure Open your OpenStackControlPlane custom resource CR file, openstack_control_plane.yaml , and add the following parameters to the glance template: Example Because ftp is in the list for allowed_schemes , this URL to the image resource is allowed: ftp://example.org/some/resource . Because 21 is not in the list for allowed_ports , this URL to the same image resource is rejected: ftp://example.org:21/some/resource . Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: Tip Append the -w option to the end of the get command to track deployment progress. 5.1.3. Configuring the copy-image import method You can configure the Image service (glance) to copy existing images to multiple Red Hat Ceph Storage stores. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml , and add the following parameters to the customServiceConfig in the glance template: Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 5.1.4. Enabling or rejecting disk formats You can configure the Image service (glance) to enable or reject disk formats. For example, you can enable only RAW and ISO disk formats or reject images in QCOW2 disk format. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml , and add the following parameters to the glance template: Replace <raw> and <iso> with the formats you want to enable from the following supported supported disk formats: none, ami, ari, aki, vhd, vhdx, vmdk, raw, qcow2, vdi, iso, ploop. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 5.1.5. Enabling plugins for the image import workflow You can enable plugins for the image import workflow by configuring the image_import_plugins option in the glance-image-import.conf file. The plugins do not run in parallel; they run in the order in which they appear in the image_import_plugins list. Image conversion is enabled by default when you use Red Hat Ceph Storage as the back end for the Image service. Procedure Configure plugins in the glance-image-import.conf file. In this example, you convert the images to RAW format before you inject metadata properties. 5.1.5.1. Injecting metadata on image import to control where instances launch You can enable the Inject Image Metadata plugin to apply metadata properties to images that are imported by cloud users so that instances that are launched from the images are located on specific Compute nodes. The Inject Image Metadata plugin contains two parameters: ignore_user_roles is a comma-separated list of Identity service (keystone) roles that the plugin will ignore. If the user making the image import call has any of these roles, the plugin will not inject any properties into the image. inject is a comma-separated list of properties and values that will be injected into the image record for the imported image. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml , and add the following parameters to the glance template: Replace <admin> with the user roles you want the plugin to ignore. Replace <property1> , <value1> , <property2> , <value2> , and so on with the properties and values that you want to inject to the image. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 5.2. Configuring scalability and security You can improve the scalability of Image service (glance) operations by configuring image caching and sparse image upload. You can configure security features, such as image signing and verification, and metadata definition (metadef) APIs. You can use the Image service API ( glanceAPI ) caching mechanism to store copies of images on glanceAPI servers and retrieve them automatically to improve scalability by reducing the load on resources. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. 5.2.1. Improving scalability with image caching Use the Image service API ( glanceAPI ) caching mechanism to store copies of images on glanceAPI servers and retrieve them automatically. With image caching, the API does not need to retrieve the same image from back-end storage multiple times. The automatic retrieval of image copies improves scalability and performance by reducing the load on resources. You use two files to manage image cache: Use glance-api.conf to configure the server. Use glance-cache.conf to configure the utilities. The following options are in both configuration files. These options require the same values in both files. Table 5.1. Configuration options for image cache Option Description Default image_cache_dir Base directory where the Image service (glance) stores the cache data. None, must be set image_cache_sqlite_db Path to the sqlite file database to be used for cache management. This path is a relative path from the image_cache_dir directory. cache.db image_cache_driver Driver used for cache management. sqlite image_cache_max_size Cache size when the glance-cache-pruner removes the oldest images to reduce the number of bytes until it is below this value. 10 GB image_cache_stall_time Amount of time an incomplete image stays in the cache until it is deleted. 1 day You can use a cron job to run glance-cache-pruner to remove the oldest images at regular intervals. You can also use a cron job to run glance-cache-cleaner to clean up image files that are in stalled or invalid states because the image cache writes failed to complete or because image files were not written successfully to disk. The following values are specific to the glance-cache.conf file and are only required for the prefetcher to run correctly. filesystem_store_datadir : Use this parameter if you are using a filesystem store. The parameter points to where the data is kept. filesystem_store_datadirs : Use this parameter to point to multiple filesystem stores. In Red Hat OpenStack Services on OpenShift (RHOSO), you can queue images for immediate caching instead of using periodic jobs to prefetch queued images as in Red Hat OpenStack Platform versions. 5.2.1.1. Enabling image caching To enable image caching in the Image service (glance), you specify the size assigned to the cache by adding the Size parameter to the imageCache section of the glance template in your OpenStackControlPlane custom resource (CR) file. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml , and add the following parameters to the glance template: Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 5.2.1.2. Image caching command options You can use the glance client to manage cached images. Because image cache is local to each node, you must perform cache operations on each node locally. If your Red Hat OpenStack Services on OpenShift (RHOSO) deployment has high availability with three, five, or seven controllers, then you specify the host address by using the --os-image-url option when running cache operations. Example: Replace <image_url> with the URL for your image, for example "http://REPLICA""0.{REPLICA}""0.REPLICA""0.DOMAIN:9292" , where REPLICA is a glanceAPI replica that does not own the data and DOMAIN is the service hostname associated with a specific glanceAPI . If you have cachemanage middleware enabled, you can use the glance-cache-manage program to manage cached images. You can run the glance-cache-manage program from a different host than the host that contains the image cache. Table 5.2. Command options for image cache when using the glance client Command Description USD glance --os-image-url=<image_url> cache-queue <image_id> Queue an image for immediate caching. USD glance --os-image-url=<image_url> cache-list List all images that are currently cached. USD glance --os-image-url=<image_url> cache-delete <image_id> [<image_id> ...] Delete one or more cached images or images that are queued for caching from the cache directory. USD glance --os-image-url=<image_url> cache-clear Remove all cached images and images that are queued for caching from the cache directory. USD glance --os-image-url=<image_url> --target cached Remove all cached images from the cache directory. USD glance --os-image-url=<image_url> cache-clear --target queued Remove all images that are queued for caching from the cache directory. Table 5.3. Command options for image cache when using the glance-cache-manage program Command Description USD glance-cache-manage --host=<host> list-cached List all images that are currently cached on a specific host. USD glance-cache-manage --host=<host> list-queued List all images that are currently queued for caching on a specific host. USD glance-cache-manage --host=<host> queue-image Queue an image for caching on a specific host. USD glance-cache-manage --host=<host> delete-cached-image Delete an image from the cache or cache queue of a specific host. USD glance-cache-manage --host=<host> delete-all-cached-images Remove all images from the cache on a specific host. USD glance-cache-manage --host=<host> delete-queued-image Deleted an image from the cache queue on a specific host. USD glance-cache-manage --host=<host> delete-all-queued-images Deleted all images from the cache queue on a specific host. 5.2.2. Enabling sparse image upload When you use Red Hat Ceph Storage RADOS Block Device (RBD) as the storage back end for the Image service (glance), you can use sparse image upload to reduce network traffic and save storage space. This feature is useful in distributed compute node (DCN) environments. With a sparse image file, the Image service does not write null byte sequences. The Image service writes data with a given offset. Storage back ends interpret these offsets as null bytes that do not consume storage space. Prerequisites Your Red Hat OpenStack Services on OpenShift (RHOSO) deployment uses Ceph Storage RBD as the storage back end for the Image service. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml , and add the following parameters to the glance template. Enable sparse image upload by setting the rbd_thin_provisioning parameter to True : Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. Verification You can import an image and check its size to verify sparse image upload. Download the image file locally: Replace <file_location> with the location of the file. Replace <file_name> with the name of the file. For example: Check the disk size and the virtual size of the image to be uploaded: For example: Import the image: Record the image ID. It is required in a subsequent step. Verify that the image is imported and in an active state: Access the remote shell for the OpenStackClient pod from your workstation: View the image: Exit the openstackclient pod: From a Ceph Storage node, verify that the size of the image is less than the virtual size from the output of step 2: 5.2.3. Image signature verification You can use image signature verification to validate images that are uploaded to the Image service (glance) before storing the images in the configured back end. If validation fails for an image, then the upload is stopped and the image is deleted. To protect image integrity and authenticity, you can save the signatures and public key certificates as image properties. You store the secret for signature verification in the Key Manager service (barbican), and the Image service interacts with the Key Manager service through the internal endpoint provided by the Identity service (keystone): Other services, such as the Compute service (nova) can use the image properties to perform data validation when a user downloads the image from the Image service. Note Image signing and verification is not supported if the Compute service (nova) is using Ceph RADOS Block Device (RBD) to store virtual machines disks. For information about creating signed images, see Signing Image service (glance) images in Performing security operations . 5.2.4. Secure metadef APIs In Red Hat OpenStack Services on OpenShift (RHOSO), you can define key value pairs and tag metadata with metadata definition (metadef) APIs. There is no limit on the number of metadef namespaces, objects, properties, resources, or tags that you can create. Image service policies control metadef APIs. By default, only administrators can create, update, or delete (CUD) metadef APIs. This limitation prevents metadef APIs from exposing information to unauthorized users and mitigates the risk of a malicious user filling the Image service (glance) database with unlimited resources, which can create a Denial of Service (DoS) style attack. 5.3. Configuring image quotas In RHOSO 18, image quotas are not enabled by default in the Image service (glance). When you enable image quotas, some quota limits have default values, but you can override the default quota limits so that individual projects have different consumption limits. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. 5.3.1. Changing the project quotas for the Image service You can update the following image quotas for projects. Table 5.4. Image quota types and default limits Quota Description Default image_size_total The maximum amount of storage (in MiB) that the project has for active images. Images with multiple locations contribute to this count, for example, if you have a single 1GiB image stored in four locations, the usage is 4 GiB. Set the value to at least the size of the largest image you expect a user to use. 1,000 MiB image_stage_total The total amount of staging space that may be used. Set the value to at least the size of the largest image you expect a user to use, and allow or limit a user to import one or more images at the same time. 1,000 MiB image_count_total The maximum number of image objects that the user may have, regardless of the size of individual or collective image objects. 100 MiB image_count_uploading The number of parallel upload operations that can be in-progress at any single point in time. 100 MiB image_size_cap The global limit on the size of a single image. 1 TB 5.3.2. Configuring the Image service to use project quotas Project-specific quotas are not enabled by default in the Image service (glance). To use project-specific quotas in the Image service, you create the limits in the Identity service (keystone), and configure the Image service to access the Identity service and use the registered quotas in the glance-api.conf file. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Register quota limits for the Image service in the Identity service: Register a limit for image_size_total : Register a limit for image-stage_total : Register a limit for image_count_total : Register a limit for image_count_uploading : Configure the Image service to use the quotas from the Identity service in your glance-api.conf file. Configure access to the Identity service in the [oslo_limit] section of your glance-api.conf file. Replace <glance_pass> with the password you set for the glance user in the Identity service. Replace <endpoint_id> withe ID of the image endpoint. Run the following command to view the image endpoint: Ensure that the glance user has reader access to system-wide resources, such as limits: Enable project-specific quotas in the DEFAULT section of your glance-api.conf file: Exit the openstackclient pod:

Commands:
glance: template: customServiceConfig: | [DEFAULT] allowed_schemes = [http,https,ftp] disallowed_schemes = [] allowed_hosts = [] disallowed_hosts = [] allowed_ports = [80,443] disallowed_ports = [] glanceAPIs:
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane spec: glance: enabled: true template: customServiceConfig: | [DEFAULT] enabled_import_methods=web-download,glance-direct,copy-image glanceAPIs:
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane spec: glance: enabled: true template: databaseInstance: openstack databaseUser: glance customServiceConfig: | [image_format] disk_formats=<raw>,<iso> glanceAPIs:
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
[image_import_opts] image_import_plugins = ['image_conversion','inject_image_metadata'] [image_conversion] output_format = raw [inject_metadata_properties] ignore_user_roles = admin, inject = "property1":"value1","property2":"value2",
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane spec: glance: enabled: true template: databaseInstance: openstack databaseUser: glance customServiceConfig: | [DEFAULT] enabled_backends = default_backend:rbd enabled_import_methods=[web-download,glance-direct] [image_import_opts] image_import_plugins = ['inject_image_metadata'] [inject_metadata_properties] ignore_user_roles = <admin>, inject = "<property1>":"<value1>","<property2>":"<value2>",
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane spec: glance: template: databaseInstance: openstack databaseUser: glance customServiceConfig: | imageCache: size: 10Gi storage: storageRequest: 10Gi storageClass: "" glanceAPIs: default:
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
glance --os-image-url=<image_url> cache-list
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane spec: glance: template: databaseInstance: openstack databaseUser: glance customServiceConfig: | [DEFAULT] enabled_backends = default_backend:rbd enabled_import_methods=[web-download] [glance_store] default_backend = default_backend [default_backend] rbd_store_ceph_conf = /etc/ceph/ceph.conf store_description = "RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
wget <file_location>/<file_name>
wget https://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud-1508.qcow2
qemu-img info <file_name>
qemu-img info CentOS-6-x86_64-GenericCloud-1508.qcow2 image: CentOS-6-x86_64-GenericCloud-1508.qcow2 file format: qcow2 virtual size: 8 GiB (8589934592 bytes) disk size: 1.09 GiB cluster_size: 65536 Format specific information: compat: 0.10 refcount bits: 1
glance image-create-via-import --disk-format qcow2 --container-format bare --name centos_1 --file <file_name>
oc rsh -n openstack openstackclient
openstack image show <image_id>
exit
sudo rbd -p images diff <image_id> | awk '{ SUM += USD2 } END { print SUM/1024/1024/1024 " GB" }' 1.03906 GB
[key_manager] backend = barbican [barbican] auth_endpoint={{ .KeystoneInternalURL }} barbican_endpoint_type=internal
oc rsh -n openstack openstackclient
openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 1000 --region RegionOne image_size_total
openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 1000 --region RegionOne image_stage_total
openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 100 --region RegionOne image_count_total
openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 100 --region RegionOne image_count_uploading
[oslo_limit] auth_url = http://controller:5000 auth_type = password user_domain_id = default username = glance system_scope = all password = <glance_pass> endpoint_id = <endpoint_id> region_name = RegionOne
openstack endpoint list --service glance --region RegionOne
openstack role add --user glance --user-domain Default --system all reader
[DEFAULT] use_keystone_limits = True
exit