Chapter 5. Testing and troubleshooting autoscaling Use the Orchestration service (heat) to automatically scale instances up and down based on threshold definitions. To troubleshoot your environment, you can look for errors in the log files and history records. 5.1. Testing automatic scaling up of instances You can use the Orchestration service (heat) to scale instances automatically based on the cpu_alarm_high threshold definition. When the CPU use reaches a value defined in the threshold parameter, another instance starts up to balance the load. The threshold value in the template.yaml file is set to 80%. This procedure uses example values that might differ from your environment. Procedure Access the client pod: Get the cirros server IP address and log in to the instance. The default password for cirros images is "gocubsgo": Run multiple dd commands to generate the load: Exit from the running instance and return to the host. After you run the dd commands, you can expect to have 100% CPU use in the instance after some time. Verify that the alarm has been triggered: After approximately 60 seconds, Orchestration starts another instance and adds it to the group. To verify that an instance has been created, enter the following command: (Optional) After another short period of time, observe if the Orchestration service autoscales to three instances. The configuration is set to a maximum of three instances. Verify there are three instances: 5.2. Testing automatic scaling down of instances You can use the Orchestration service (heat) to automatically scale down instances based on the cpu_alarm_low threshold. In this example, the instances are scaled down when CPU usage is below 20%. Procedure From within the workload instance, terminate the running dd processes and observe Orchestration begin to scale the instances back down. Exit from the running instance and return to the host. When you stop the dd processes, this triggers the cpu_alarm_low event alarm. As a result, Orchestration begins to automatically scale down and remove the instances. Verify that the corresponding alarm has triggered: After a few minutes, Orchestration continually reduce the number of instances to the minimum value defined in the min_size parameter of the scaleup_group definition. In this scenario, the min_size parameter is set to 1 . 5.3. Troubleshooting for autoscaling If necessary, you can look for errors in the log files and history records. Procedure Access the openstack client pod: To retrieve information on state transitions, list the stack event records: Read the alarm history log: Exit the openstack client pod: To view the records of scale-out or scale-down operations that the Orchestration service (heat) collects for the existing stack, you can use the awk command to parse the heat pod logs. To view information about the Alarming service (aodh), examine the logs of the aodh-evaluator container. 5.4. Using CPU telemetry values for autoscaling threshold when using the rate function When using the OS::Heat::Autoscaling heat orchestration template (HOT) and setting a threshold value for CPU, the value is expressed in nanoseconds of CPU time, which is a dynamic value based on the number of virtual CPUs allocated to the instance workload. You can calculate and express the CPU nanosecond value as a percentage when using the rate function when querying prometheus. 5.4.1. Calculating CPU telemetry values as a percentage CPU telemetry is stored in Prometheus as CPU utilization in nanoseconds. When using CPU telemetry to define autoscaling thresholds, it is useful to express the values as a percentage of CPU utilization since that is more natural when defining the threshold values. When defining the scaling policies used as part of an autoscaling group, you can define thresholds as percentages and calculate the CPU utilization as a percentage, instead of in nanoseconds. Divide the rate of increase of the metric by 1,000,000,000. Divide the answer by the number of vCPUs. Multiply the answer by 100 to get a percentage. Use this formula to calculate this value in the PromQL queries: "(rate(ceilometer_cpu{server_group=~<stack_id>.*'}[150s])) / 1000000000 / <number_of_vCPUs> * 100". Replace <server_name_prefix> with the name of your server name prefix. Replace <number_of_vCPUs> with the the number of vCPUs. 5.4.2. Displaying instance workload vCPU as a percentage You can display the CPU telemetry data as a percentage rather than the nanosecond values for instances by using the openstack metric query command. Prerequisites Create a heat stack using the autoscaling group resource that results in an instance workload. Procedure Access the openstack client pod: Retrieve the stack_id of the autoscaling group heat stack: Return the metrics rate of increase with the value calculated as a percentage. The rate of increase is returned as a value of nanoseconds of CPU time by which the metric increased in our given window.You divide that number by 1000000000 to get the value in seconds. You then convert that value to a percentage by multiplying by 100. Finally, you divide the total value by the number of vCPU provided by the flavor assigned to the instance, in this example a value of 1 vCPU, providing a value expressed as a percentage of CPU time. Use stack_id for the value of the 'server_group' label in the query to get only the results from your stack: You can look at historical values by adding the offset keyword to the query. The following gives similar output as the step before, but with data 10 minutes in the past: 5.4.3. Retrieving available telemetry for an instance workload Retrieve the available telemetry for an instance workload and express the vCPU utilization as a percentage. Prerequisites Create a heat stack using the autoscaling group resource that results in an instance workload. Procedure Access the openstack client pod: Retrieve the ID of the autoscaling group heat stack: Set the value of the stack ID to an environment variable: Retrieve the ID of the workload instance that you want to return data for. You can use the server list long form and filter for instances that are part of the autoscaling group: Set the instance ID for one of the returned instance workload names: Verify that the metrics are stored for the instance resource ID. If no metrics are available, not enough time has elapsed since the instance was created. If enough time has elapsed, you can check the logs for the data collection service by executing podman logs ceilometer_agent_compute on the Compute node where the instance is running, and you can check the logs for the time-series database service Prometheus by executing oc logs prometheus-metric-storage-0 -c prometheus : Display gathered ceilometer metrics: Retrieve the number of vCPU cores applied to the workload instance by reviewing the configured flavor for the instance workload: Return the metrics rate of increase with the value calculated as a percentage. The rate of increase is returned as a value of nanoseconds of CPU time by which the metric increased in our given window. You divide that number by 1000000000 to get the value in seconds. That value is then converted to a percentage by multiplying by 100. Finally, you divide the total value by the number of vCPU provided by the flavor assigned to the instance, in this example a value of 1 vCPU, providing a value expressed as a percentage of CPU time: Add the offset keyword to the query to view historical results. The following command provides similar output to the step before, but with data 10 minutes in the past:

Commands:
oc rsh openstackclient
openstack server list | grep cirros ssh cirros@192.168.122.8
[instance ~]USD sudo dd if=/dev/zero of=/dev/null & [instance ~]USD sudo dd if=/dev/zero of=/dev/null & [instance ~]USD sudo dd if=/dev/zero of=/dev/null &
openstack alarm list +--------------------------------------+------------+------------------------------------+-------+----------+---------+ | alarm_id | type | name | state | severity | enabled | +--------------------------------------+------------+------------------------------------+-------+----------+---------+ | 3000766d-a7fc-4187-98ea-cafd1239c160 | prometheus | stack1-cpu_alarm_low-n3iy7oaffq4i | ok | low | True | | 899f8242-96dd-4bad-a7f9-0b9526ea6032 | prometheus | stack1-cpu_alarm_high-riua4sthkvcs | alarm | low | True | +--------------------------------------+------------+------------------------------------+-------+----------+---------+
openstack server list +--------------------------------------+-----------------------------------------------------------------------------------+--------+----------------------------------------+--------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------------------------------------------------------------------+--------+----------------------------------------+--------+----------+ | 53ffb3c5-a425-4929-b01d-50d653fe23f8 | autoscaling_server_stack1-autoscalinggroup-oqmxiukze6dg-qn7drottgjek-3oiecx4gkowi | ACTIVE | private=192.168.0.116, 192.168.122.204 | cirros | m1.small | | 9e2e9266-d69e-4de1-8b3a-e75f7111d1d9 | autoscaling_server_stack1-autoscalinggroup-oqmxiukze6dg-hdtuapbjxcow-q366nn4k5utr | ACTIVE | private=192.168.0.209, 192.168.122.205 | cirros | m1.small | +--------------------------------------+-----------------------------------------------------------------------------------+--------+----------------------------------------+--------+----------+
openstack server list +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------------------------+ | 477ee1af-096c-477c-9a3f-b95b0e2d4ab5 | ex-3gax-4urpikl5koff-yrxk3zxzfmpf-server-2hde4tp4trnk | ACTIVE | - | Running | internal1=10.10.10.13, 192.168.122.17 | | e1524f65-5be6-49e4-8501-e5e5d812c612 | ex-3gax-5f3a4og5cwn2-png47w3u2vjd-server-vaajhuv4mj3j | ACTIVE | - | Running | internal1=10.10.10.9, 192.168.122.8 | | 6c88179e-c368-453d-a01a-555eae8cd77a | ex-3gax-fvxz3tr63j4o-36fhftuja3bw-server-rhl4sqkjuy5p | ACTIVE | - | Running | internal1=10.10.10.5, 192.168.122.5 | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------------------------+
sudo killall dd
openstack alarm list +--------------------------------------+------------+------------------------------------+-------+----------+---------+ | alarm_id | type | name | state | severity | enabled | +--------------------------------------+------------+------------------------------------+-------+----------+---------+ | 98dbbf20-44ec-4aa9-bb42-14f75b184dad | prometheus | stack1-cpu_alarm_high-xed3bp5mrscw | ok | low | True | | d3dd58e9-b802-4b52-9c00-7932e59a49c2 | prometheus | stack1-cpu_alarm_low-bjql75566bkn | alarm | low | True | +--------------------------------------+------------+------------------------------------+-------+----------+---------+
oc rsh openstackclient
openstack stack event list example 2017-03-06 11:12:43Z [example]: CREATE_IN_PROGRESS Stack CREATE started 2017-03-06 11:12:43Z [example.scaleup_group]: CREATE_IN_PROGRESS state changed 2017-03-06 11:13:04Z [example.scaleup_group]: CREATE_COMPLETE state changed 2017-03-06 11:13:04Z [example.scaledown_policy]: CREATE_IN_PROGRESS state changed 2017-03-06 11:13:05Z [example.scaleup_policy]: CREATE_IN_PROGRESS state changed 2017-03-06 11:13:05Z [example.scaledown_policy]: CREATE_COMPLETE state changed 2017-03-06 11:13:05Z [example.scaleup_policy]: CREATE_COMPLETE state changed 2017-03-06 11:13:05Z [example.cpu_alarm_low]: CREATE_IN_PROGRESS state changed 2017-03-06 11:13:05Z [example.cpu_alarm_high]: CREATE_IN_PROGRESS state changed 2017-03-06 11:13:06Z [example.cpu_alarm_low]: CREATE_COMPLETE state changed 2017-03-06 11:13:07Z [example.cpu_alarm_high]: CREATE_COMPLETE state changed 2017-03-06 11:13:07Z [example]: CREATE_COMPLETE Stack CREATE completed successfully 2017-03-06 11:19:34Z [example.scaleup_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 95.4080102993) 2017-03-06 11:25:43Z [example.scaleup_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 95.8869217299) 2017-03-06 11:33:25Z [example.scaledown_policy]: SIGNAL_COMPLETE alarm state changed from ok to alarm (Transition to alarm due to 1 samples outside threshold, most recent: 2.73931707966) 2017-03-06 11:39:15Z [example.scaledown_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 2.78110858552)
openstack alarm-history show 022f707d-46cc-4d39-a0b2-afd2fc7ab86a +----------------------------+------------------+-----------------------------------------------------------------------------------------------------+--------------------------------------+ | timestamp | type | detail | event_id | +----------------------------+------------------+-----------------------------------------------------------------------------------------------------+--------------------------------------+ | 2017-03-06T11:32:35.510000 | state transition | {"transition_reason": "Transition to ok due to 1 samples inside threshold, most recent: | 25e0e70b-3eda-466e-abac-42d9cf67e704 | | | | 2.73931707966", "state": "ok"} | | | 2017-03-06T11:17:35.403000 | state transition | {"transition_reason": "Transition to alarm due to 1 samples outside threshold, most recent: | 8322f62c-0d0a-4dc0-9279-435510f81039 | | | | 95.0964497325", "state": "alarm"} | | | 2017-03-06T11:15:35.723000 | state transition | {"transition_reason": "Transition to ok due to 1 samples inside threshold, most recent: | 1503bd81-7eba-474e-b74e-ded8a7b630a1 | | | | 3.59330523447", "state": "ok"} | | | 2017-03-06T11:13:06.413000 | creation | {"alarm_actions": ["trust+http://fca6e27e3d524ed68abdc0fd576aa848:delete@192.168.122.126:8004/v1/fd | 224f15c0-b6f1-4690-9a22-0c1d236e65f6 | | | | 1c345135be4ee587fef424c241719d/stacks/example/d9ef59ed-b8f8-4e90-bd9b- | | | | | ae87e73ef6e2/resources/scaleup_policy/signal"], "user_id": "a85f83b7f7784025b6acdc06ef0a8fd8", | | | | | "name": "example-cpu_alarm_high-odj77qpbld7j", "state": "insufficient data", "timestamp": | | | | | "2017-03-06T11:13:06.413455", "description": "Scale up if CPU > 80%", "enabled": true, | | | | | "state_timestamp": "2017-03-06T11:13:06.413455", "rule": {"evaluation_periods": 1, "metric": | | | | | "cpu_util", "aggregation_method": "mean", "granularity": 300, "threshold": 80.0, "query": "{\"=\": | | | | | {\"server_group\": \"d9ef59ed-b8f8-4e90-bd9b-ae87e73ef6e2\"}}", "comparison_operator": "gt", | | | | | "resource_type": "instance"}, "alarm_id": "022f707d-46cc-4d39-a0b2-afd2fc7ab86a", | | | | | "time_constraints": [], "insufficient_data_actions": null, "repeat_actions": true, "ok_actions": | | | | | null, "project_id": "fd1c345135be4ee587fef424c241719d", "type": | | | | | "gnocchi_aggregation_by_resources_threshold", "severity": "low"} | | +----------------------------+------------------+-----------------------------------------------------------------------------------------------------+-------------------------------------
exit
oc logs -l component=engine,service=heat --tail=-1 | awk '/Updating stack/,/Stack CREATE completed successfully/ {print USD0}'
oc logs aodh-0 -c aodh-evaluator | grep -i alarm | grep -i transition | grep -v DEBUG
oc rsh openstackclient
openstack stack show stack1 -c id +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 1ae77e51-ba09-452c-bc09-385f0791a366 | +-------+--------------------------------------+
openstack metric query "rate(ceilometer_cpu{server_group='1ae77e51-ba09-452c-bc09-385f0791a366'}[150s])/1000000000*100/1" -c resource_name -c value +-------------------------------------------------------------------------+-------+ | resource_name | value | +-------------------------------------------------------------------------+-------+ | st-ih4tya6-bf63mo4vvyqp-pvdwdmvzov5c-vnf-x4f535xcvjyp:instance-00000003 | 0.525 | +-------------------------------------------------------------------------+-------+
openstack metric query "rate(ceilometer_cpu{server_group='1ae77e51-ba09-452c-bc09-385f0791a366'}[150s] offset 10m)/1000000000*100/1" -c resource_name -c value +-------------------------------------------------------------------------+-------------------+ | resource_name | value | +-------------------------------------------------------------------------+-------------------+ | st-ih4tya6-3ul7w6w66st7-mmvkxuwghjxl-vnf-dj7k7665csuv:instance-00000002 | 38.40833333333333 | +-------------------------------------------------------------------------+-------------------+
oc rsh openstackclient
openstack stack show stack1 -c id -c stack_status +--------------+--------------------------------------+ | Field | Value | +--------------+--------------------------------------+ | id | 0d6e19a9-2e1c-4728-b9ed-320da60c3c9d | | stack_status | CREATE_COMPLETE | +--------------+--------------------------------------+
export STACK_ID=USD(openstack stack show stack1 -c id -f value)
openstack server list --long --fit-width -c ID -c Name -c Properties | grep "metering.server_group='USDSTACK_ID'" | 94a776a6-ab74-41ec-9c21-26d3ca0a0138 | autoscaling_server_stack1-autoscalinggroup-l3qthrzpad67-nonm7balfxrn-sx6oebhyi3qo | metering.server_group='168364ae-afe2-4275-af28-aae01ee63b00' |
INSTANCE_ID=94a776a6-ab74-41ec-9c21-26d3ca0a0138
openstack metric query "ceilometer_cpu{resource='USDINSTANCE_ID'}" -c resource -c resource_name -c value -c unit +--------------------------------------+-----------------------------------------------------------------------------------------------------+------+--------------+ | resource | resource_name | unit | value | +--------------------------------------+-----------------------------------------------------------------------------------------------------+------+--------------+ | a19f11ba-7834-4143-bf83-766af8199f04 | autoscaling_server_stack1-autoscalinggroup-5yzjn6ow4x6v-gmu2apozwluw-t7nqx63tybxh:instance-0000003c | ns | 204290000000 | +--------------------------------------+-----------------------------------------------------------------------------------------------------+------+--------------+
openstack metric list | grep ceilometer | ceilometer_cpu | | ceilometer_disk_device_allocation | | ceilometer_disk_device_capacity | | ceilometer_disk_device_read_bytes | | ceilometer_disk_device_read_latency | | ceilometer_disk_device_read_requests | | ceilometer_disk_device_usage | | ceilometer_disk_device_write_bytes | | ceilometer_disk_device_write_latency | | ceilometer_disk_device_write_requests | | ceilometer_image_size | | ceilometer_memory_usage | | ceilometer_network_incoming_bytes | | ceilometer_network_incoming_bytes_delta | | ceilometer_network_incoming_packets | | ceilometer_network_incoming_packets_drop | | ceilometer_network_incoming_packets_error | | ceilometer_network_outgoing_bytes | | ceilometer_network_outgoing_bytes_delta | | ceilometer_network_outgoing_packets | | ceilometer_network_outgoing_packets_drop | | ceilometer_network_outgoing_packets_error |
openstack server show USDINSTANCE_ID -cflavor -f value m1.small (692533fe-0912-417e-b706-5d085449db53) openstack flavor show 692533fe-0912-417e-b706-5d085449db53 -c vcpus -f value 1
openstack metric query "rate(ceilometer_cpu{resource=~'USDINSTANCE_ID'}[150s])/1000000000*100/1" -c resource_name -c value -c resource +--------------------------------------+-----------------------------------------------------------------------------------------------------+--------------------+ | resource | resource_name | value | +--------------------------------------+-----------------------------------------------------------------------------------------------------+--------------------+ | a19f11ba-7834-4143-bf83-766af8199f04 | autoscaling_server_stack1-autoscalinggroup-5yzjn6ow4x6v-gmu2apozwluw-t7nqx63tybxh:instance-0000003c | 0.6166666666666667 | +--------------------------------------+-----------------------------------------------------------------------------------------------------+--------------------+
openstack metric query "rate(ceilometer_cpu{resource=~'USDINSTANCE_ID'}[150s] offset 10m)/1000000000*100/1" -c resource_name -c value -c resource +--------------------------------------+-----------------------------------------------------------------------------------------------------+--------------------+ | resource | resource_name | value | +--------------------------------------+-----------------------------------------------------------------------------------------------------+--------------------+ | a19f11ba-7834-4143-bf83-766af8199f04 | autoscaling_server_stack1-autoscalinggroup-5yzjn6ow4x6v-gmu2apozwluw-t7nqx63tybxh:instance-0000003c | 0.6083333333333333 | +--------------------------------------+-----------------------------------------------------------------------------------------------------+--------------------+