Chapter 4. Managing Load-balancing service instance logs Load-balancing service (octavia) instances (amphorae) generate administrative logs and tenant flow logs. The amphorae offload these logs to central locations on syslog receivers using a set of containers or to other syslog receivers at endpoints that you can choose. This log offload feature enables administrators to go to one location for logs, and to retain logs when the amphorae are rotated. Even though log offloading is enabled by default, amphorae still continue to write administrative and tenant flow logs to the disk inside the amphorae. You can, however, disable logging locally if you choose. When you use the TCP syslog protocol, you can specify one or more secondary endpoints for administrative and tenant log offloading in the event that the primary endpoint fails. You can control a range of other logging features such as setting the syslog facility value, changing the tenant flow log format, and widening the scope of administrative logging to include logs from sources like the kernel and from cron . Section 4.1, "Configuration parameters for Load-balancing service instance logging" Section 4.2, "Load-balancing service instance tenant flow log format" Section 4.3, "Disabling Load-balancing service instance tenant flow logging" Section 4.4, "Disabling Load-balancing service instance local log storage" 4.1. Configuration parameters for Load-balancing service instance logging To modify the Load-balancing service (octavia) instance (amphora) logging configuration, set values for one or more configuration parameters that control logging and apply the OpenStackControlPlane custom resource (CR) for the Load-balancing service. These configuration parameters for amphora logging enable you to control features such as turning off log offloading, defining custom endpoints to offload logs to, setting the syslog facility value for logs, and so on. The octavia Operator automatically enables log offloading. Global logging parameters To set the configuration parameters for all logs, you must add a specific section to OpenStackControlerPlane CR for each of the octavia services: housekeeping, health manager, and worker. Add the configuration parameters for all logs underneath the customServiceConfig.[amphora_agent] parameters. Usage example disable_local_log_storage=false When true , instances do not store logs on the instance host filesystem. This includes all kernel, system, and security logs. Default: false . forward_all_logs=true When true , instances forward all log messages to the administrative log endpoints, including non-load balancing related logs such as the cron and kernel logs. Default: true . Administrative logging parameters To set the configuration parameters for administrative logging, you must add a specific section to OpenStackControlerPlane CR for each of the octavia services: housekeeping, health manager, and worker. With the exception of adminLogTargets , you add the configuration parameters for administrative logging underneath the customServiceConfig.[amphora_agent] parameters. Usage example adminLogTargets A list of objects describing syslog endpoints to receive administrative log messages: host : <host> port : <port> protocol : <protocol> An endpoint can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port. Default: The default value is automatically set by the octavia Operator. You add adminLogTargets underneath the octaviaRsyslog parameter. administrative_log_facility=<number> A number between 0 and 7 that is the syslog LOG_LOCAL facility to use for the administrative log messages. Default: 1 . Tenant flow logging parameters To set the configuration parameters for tenant flow logging, you must add a specific section to OpenStackControlerPlane CR for each of the octavia services: housekeeping, health manager, and worker. With the exception of tenantLogTargets , you add the configuration parameters for tenant flow logging underneath the customServiceConfig.[amphora_agent] parameters. For an example of how to set these parameters, see Section 4.3, "Disabling Load-balancing service instance tenant flow logging" . Usage example connection_login=true | false When true , tenant connection flows are logged. Default: true . tenantLogTargets A list of objects describing syslog endpoints to receive tenant traffic flow log messages: host : <host> port : <port> protocol : <protocol> These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port. Default: The default value is automatically set by the octavia Operator. You add tenantLogTargets underneath the octaviaRsyslog parameter. user_log_facility=<number> A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the tenant traffic flow log messages. Default: 0 . user_log_format="<value>" The format for the tenant traffic flow log. Default: "{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc" . The alphanumerics represent specific octavia fields, and the curly braces ({}) are substitution variables. 4.2. Load-balancing service instance tenant flow log format Tenant flow logs for Load-balancing service instances (amphorae) use the HAProxy log format. The two exceptions are the project_id and lb_id variables whose values are provided by the amphora provider driver. Example Here is an example log entry with rsyslog as the syslog receiver: Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]: 5408b89aa45b48c69a53dca1aaec58db fd8f23df-960b-4b12-ba62-2b1dff661ee7 261ecfc2-9e8e-4bba-9ec2-3c903459a895 172.24.4.1 41152 12/Jun/2019:00:44:13.030 "GET / HTTP/1.1" 200 76 73 - "" e37e0e04-68a3-435b-876c-cffe4f2138a4 6f2720b3-27dc-4496-9039-1aafe2fee105 4 -- Notes A hyphen (-) indicates any field that is unknown or not applicable to the connection. The prefix in the earlier sample log entry originates from the rsyslog receiver, and is not part of the syslog message from the amphora: Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]:" Default The default amphora tenant flow log format is: `"{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc"` The following table describes the log file format details. Table 4.1. Data variables for tenant flow logs format variable definitions. Variable Type Field name {{project_id}} UUID Project ID (substitution variable from the amphora provider driver) {{lb_id}} UUID Load balancer ID (substitution variable from the amphora provider driver) %f string frontend_name %ci IP address client_ip %cp numeric client_port %t date date_time %ST numeric status_code %B numeric bytes_read %U numeric bytes_uploaded %ssl_c_verify Boolean client_certificate_verify (0 or 1) %ssl_c_s_dn string client_certificate_distinguised_name %b string pool_id %s string member_id %Tt numeric processing_time (milliseconds) %tsc string termination_state (with cookie status) Additional resources Custom log format in HAProxy Documentation 4.3. Disabling Load-balancing service instance tenant flow logging Tenant flow log offloading for Load-balancing service instances (amphorae) is enabled by default. To disable tenant flow logging without disabling administrative log offloading, you must override the [amphora_agent].tenant_log_targets` in the customServiceConfig` field of each Load-balancing service component in the OpenstackControlPlane custom resource (CR) file. When the OctaviaConnectionLogging parameter is false , the amphorae do not write tenant flow logs to the disk inside the amphorae, nor offload any logs to syslog receivers listening elsewhere. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml , on your workstation. Add the following configuration to the octavia service configuration: Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. Confirm that the control plane is deployed by reviewing the pods in the openstack namespace: The control plane is deployed when all the pods are either completed or running. 4.4. Disabling Load-balancing service instance local log storage Even when you configure Load-balancing service instances (amphorae) to offload administrative and tenant flow logs, the amphorae continue to write these logs to the disk inside the amphorae. To improve the performance of the load balancer, you can stop logging locally. Important If you disable logging locally, you also disable all log storage in the amphora, including kernel, system, and security logging. Note If you disable local log storage and the OctaviaLogOffload parameter is set to false, ensure that you set OctaviaConnectionLogging to false for improved load balancing performance. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. Procedure Open your OpenStackControlPlane` custom resource (CR) file, openstack_control_plane.yaml , on your workstation. Add the following configuration to the octavia service configuration: Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. Confirm that the control plane is deployed by reviewing the pods in the openstack namespace: The control plane is deployed when all the pods are either completed or running.

Commands:
octavia: template: octaviaHousekeeping: customServiceConfig: | [amphora_agent] <log configuration parameters go here> octaviaHealthManager: customServiceConfig: | [amphora_agent] <log configuration parameters go here> octaviaWorker: customServiceConfig: | [amphora_agent] <log configuration parameters go here>
octavia: template: octaviaRsyslog: adminLogTargets: - host: 192.168.1.1 port: 1514 protocol: udp octaviaHousekeeping: customServiceConfig: | [amphora_agent] <administrative logging parameters go here> octaviaHealthManager: customServiceConfig: | [amphora_agent] <administrative logging parameters go here> octaviaWorker: customServiceConfig: | [amphora_agent] <administrative logging parameters go here>
octavia: template: octaviaRsyslog tenantLogTargets: - host: 192.168.1.1 port: 1514 protocol: udp octaviaHousekeeping: customServiceConfig: | [amphora_agent] <tenant flow logging parameters go here> [haproxy_amphora] connection_login=true octaviaHealthManager: customServiceConfig: | [amphora_agent] <tenant flow logging parameters go here> [haproxy_amphora] connection_login=true octaviaWorker: customServiceConfig: | [amphora_agent] <tenant flow logging go here> [haproxy_amphora] connection_login=true
Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]: 5408b89aa45b48c69a53dca1aaec58db fd8f23df-960b-4b12-ba62-2b1dff661ee7 261ecfc2-9e8e-4bba-9ec2-3c903459a895 172.24.4.1 41152 12/Jun/2019:00:44:13.030 "GET / HTTP/1.1" 200 76 73 - "" e37e0e04-68a3-435b-876c-cffe4f2138a4 6f2720b3-27dc-4496-9039-1aafe2fee105 4 --
Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]:"
`"{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc"`
octavia: template: octaviaHousekeeping: customServiceConfig: | [amphora_agent] tenant_log_targets = octaviaHealthManager: customServiceConfig: | [amphora_agent] tenant_log_targets = octaviaWorker: customServiceConfig: | [amphora_agent] tenant_log_targets =
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack NAME STATUS MESSAGE openstack-control-plane Unknown Setup started
oc get pods -n openstack
octavia: template: octaviaHousekeeping: customServiceConfig: | [amphora_agent] disable_local_log_storage=true octaviaHealthManager: customServiceConfig: | [amphora_agent] disable_local_log_storage=true octaviaWorker: customServiceConfig: | [amphora_agent] disable_local_log_storage=true
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack NAME STATUS MESSAGE openstack-control-plane Unknown Setup started
oc get pods -n openstack