Chapter 4. Installing and configuring the logs service Red Hat OpenStack Platform (RHOSP) writes informational messages to specific log files; you can use these messages for troubleshooting and monitoring system events. The log collection agent Rsyslog collects logs on the client side and sends these logs to an instance of Rsyslog that is running on the server side. The server-side Rsyslog instance redirects log records to Elasticsearch for storage. Note You do not need to attach the individual log files to your support cases manually. The sosreport utility gathers the required logs automatically. 4.1. The centralized log system architecture and components Monitoring tools use a client-server model with the client deployed onto the Red Hat OpenStack Platform (RHOSP) overcloud nodes. The Rsyslog service provides client-side centralized logging (CL). All RHOSP services generate and update log files. These log files record actions, errors, warnings, and other events. In a distributed environment like OpenStack, collecting these logs in a central location simplifies debugging and administration. With centralized logging, there is one central place to view logs across your entire RHOSP environment. These logs come from the operating system, such as syslog and audit log files, infrastructure components, such as RabbitMQ and MariaDB, and OpenStack services such as Identity, Compute, and others. The centralized logging toolchain consists of the following components: Log Collection Agent (Rsyslog) Data Store (ElasticSearch) API/Presentation Layer (Grafana) Note Red Hat OpenStack Platform director does not deploy the server-side components for centralized logging. Red Hat does not support the server-side components, including the Elasticsearch database and Grafana. 4.2. Enabling centralized logging with Elasticsearch To enable centralized logging, you must specify the implementation of the OS::TripleO::Services::Rsyslog composable service. Note The Rsyslog service uses only Elasticsearch as a data store for centralized logging. Prerequisites Elasticsearch is installed on the server side. Procedure Add the file path of the logging environment file to the overcloud deployment command with any other environment files that are relevant to your environment and deploy, as shown in the following example: Replace <existing_overcloud_environment_files> with the list of environment files that are part of your existing deployment. 4.3. Configuring logging features To configure logging features, modify the RsyslogElasticsearchSetting parameter in the logging-environment-rsyslog.yaml file. Procedure Copy the tripleo-heat-templates/environments/logging-environment-rsyslog.yaml file to your home directory. Create entries in the RsyslogElasticsearchSetting parameter to suit your environment. The following snippet is an example configuration of the RsyslogElasticsearchSetting parameter: Additional resources For more information about the configurable parameters, see Section 4.3.1, "Configurable logging parameters" . 4.3.1. Configurable logging parameters This table contains descriptions of logging parameters that you use to configure logging features in Red Hat OpenStack Platform (RHOSP). You can find these parameters in the tripleo-heat-templates/deployment/logging/rsyslog-container-puppet.yaml file. Table 4.1. Configurable logging parameters Parameter Description RsyslogElasticsearchSetting Configuration for rsyslog-elasticsearch plugin. For more information, see https://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html . RsyslogElasticsearchTlsCACert Contains the content of the CA cert for the CA that issued the Elasticsearch server cert. RsyslogElasticsearchTlsClientCert Contains the content of the client cert for doing client cert authorization against Elasticsearch. RsyslogElasticsearchTlsClientKey Contains the content of the private key corresponding to the cert RsyslogElasticsearchTlsClientCert . 4.4. Managing logs The containerized service log files are stored in /var/log/containers/<service> , for example /var/log/containers/cinder . Log files from services running inside containers are stored locally. The available logs can be different based on enabled and disabled services. The following example forces the logrotate task to create a new log file when it reaches 10 megabytes and retains the log file for 14 days. To customize log rotation parameters include these parameter_defaults in the environment template, then deploy the overcloud. Verification: On any overcloud node, ensure that logrotate_crond is updated: In the following example, that the nova-compute.log file has been rotated once. The log file rotation process occurs in the logrotate_crond container. The /var/spool/cron/root configuration file is read and the configuration sent to the process. Verification: Ensure that the configuration exists on any controller node: The /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf file is bound to /etc/logrotate-crond.conf inside the logrotate_crond container. Note Old configuration files are left in place for historical reasons, but they are not used. 4.5. Overriding the default path for a log file If you modify the default containers and the modification includes the path to the service log file, you must also modify the default log file path. Every composable service has a <service_name>LoggingSource parameter. For example, for the nova-compute service, the parameter is NovaComputeLoggingSource . Procedure To override the default path for the nova-compute service, add the path to the NovaComputeLoggingSource parameter in your configuration file: Note For each service, define the tag and file . Other values are derived by default. You can modify the format for a specific service. This passes directly to the Rsyslog configuration. The default format for the LoggingDefaultFormat parameter is /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<message>.*)USD/ Use the following syntax: The following snippet is an example of a more complex transformation: 4.6. Modifying the format of a log record You can modify the format of the start of the log record for a specific service. This passes directly to the Rsyslog configuration. The default format for the Red Hat OpenStack Platform (RHOSP) log record is ('^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) '). Procedure To add a different regular expression for parsing the start of log records, add startmsg.regex to the configuration: 4.7. Testing the connection between Rsyslog and Elasticsearch On the client side, you can verify communication between Rsyslog and Elasticsearch. Procedure Navigate to the Elasticsearch connection log file, /var/log/rsyslog/omelasticsearch.log in the Rsyslog container or /var/log/containers/rsyslog/omelasticsearch.log on the host. If this log file does not exist or if the log file exists but does not contain logs, there is no connection problem. If the log file is present and contains logs, Rsyslog has not connected successfully. Note To test the connection from the server side, view the Elasticsearch logs for connection issues. 4.8. Server-side logging If you have an Elasticsearch cluster running, you must configure the RsyslogElasticsearchSetting parameter in the logging-environment-rsyslog.yaml file to connect Rsyslog that is running on overcloud nodes. To configure the RsyslogElasticsearchSetting parameter, see https://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html 4.9. Tracebacks When you encounter an issue and you start troubleshooting, you can use a traceback log to diagnose the issue. In log files, tracebacks usually have several lines of information, all relating to the same issue. Rsyslog provides a regular expression to define how a log record starts. Each log record usually starts with a timestamp and the first line of the traceback is the only line that contains this information. Rsyslog bundles the indented records with the first line and sends them as one log record. For that behaviour configuration option startmsg.regex in <Service>LoggingSource is used. The following regular expression is the default value for all <service>LoggingSource parameters in director: When this default does not match log records of your added or modified LoggingSource , you must change startmsg.regex accordingly. 4.10. Location of log files for OpenStack services Each OpenStack component has a separate logging directory containing files specific to a running service. 4.10.1. Bare Metal Provisioning (ironic) log files Service Service name Log path OpenStack Ironic API openstack-ironic-api.service /var/log/containers/ironic/ironic-api.log OpenStack Ironic Conductor openstack-ironic-conductor.service /var/log/containers/ironic/ironic-conductor.log 4.10.2. Block Storage (cinder) log files Service Service name Log path Block Storage API openstack-cinder-api.service /var/log/containers/cinder-api.log Block Storage Backup openstack-cinder-backup.service /var/log/containers/cinder/backup.log Informational messages The cinder-manage command /var/log/containers/cinder/cinder-manage.log Block Storage Scheduler openstack-cinder-scheduler.service /var/log/containers/cinder/scheduler.log Block Storage Volume openstack-cinder-volume.service /var/log/containers/cinder/volume.log 4.10.3. Compute (nova) log files Service Service name Log path OpenStack Compute API service openstack-nova-api.service /var/log/containers/nova/nova-api.log OpenStack Compute certificate server openstack-nova-cert.service /var/log/containers/nova/nova-cert.log OpenStack Compute service openstack-nova-compute.service /var/log/containers/nova/nova-compute.log OpenStack Compute Conductor service openstack-nova-conductor.service /var/log/containers/nova/nova-conductor.log OpenStack Compute VNC console authentication server openstack-nova-consoleauth.service /var/log/containers/nova/nova-consoleauth.log Informational messages nova-manage command /var/log/containers/nova/nova-manage.log OpenStack Compute NoVNC Proxy service openstack-nova-novncproxy.service /var/log/containers/nova/nova-novncproxy.log OpenStack Compute Scheduler service openstack-nova-scheduler.service /var/log/containers/nova/nova-scheduler.log 4.10.4. Dashboard (horizon) log files Service Service name Log path Log of certain user interactions Dashboard interface /var/log/containers/horizon/horizon.log The Apache HTTP server uses several additional log files for the Dashboard web interface, which you can access by using a web browser or command-line client, for example, keystone and nova. The log files in the following table can be helpful in tracking the use of the Dashboard and diagnosing faults: Purpose Log path All processed HTTP requests /var/log/containers/httpd/horizon_access.log HTTP errors /var/log/containers/httpd/horizon_error.log Admin-role API requests /var/log/containers/httpd/keystone_wsgi_admin_access.log Admin-role API errors /var/log/containers/httpd/keystone_wsgi_admin_error.log Member-role API requests /var/log/containers/httpd/keystone_wsgi_main_access.log Member-role API errors /var/log/containers/httpd/keystone_wsgi_main_error.log Note There is also /var/log/containers/httpd/default_error.log , which stores errors reported by other web services that are running on the same host. 4.10.5. Identity Service (keystone) log files Service Service name Log Path OpenStack Identity Service openstack-keystone.service /var/log/containers/keystone/keystone.log 4.10.6. Image Service (glance) log files Service Service name Log path OpenStack Image Service API server openstack-glance-api.service /var/log/containers/glance/api.log OpenStack Image Service Registry server openstack-glance-registry.service /var/log/containers/glance/registry.log 4.10.7. Networking (neutron) log files Service Service name Log path OpenStack Neutron DHCP Agent neutron-dhcp-agent.service /var/log/containers/neutron/dhcp-agent.log OpenStack Networking Layer 3 Agent neutron-l3-agent.service /var/log/containers/neutron/l3-agent.log Metadata agent service neutron-metadata-agent.service /var/log/containers/neutron/metadata-agent.log Metadata namespace proxy n/a /var/log/containers/neutron/neutron-ns-metadata-proxy- UUID .log Open vSwitch agent neutron-openvswitch-agent.service /var/log/containers/neutron/openvswitch-agent.log OpenStack Networking service neutron-server.service /var/log/containers/neutron/server.log 4.10.8. Object Storage (swift) log files OpenStack Object Storage sends logs to the system logging facility only. Note By default, all Object Storage log files go to /var/log/containers/swift/swift.log , using the local0, local1, and local2 syslog facilities. The log messages of Object Storage are classified into two broad categories: those by REST API services and those by background daemons. The API service messages contain one line per API request, in a manner similar to popular HTTP servers; both the frontend (Proxy) and backend (Account, Container, Object) services post such messages. The daemon messages are less structured and typically contain human-readable information about daemons performing their periodic tasks. However, regardless of which part of Object Storage produces the message, the source identity is always at the beginning of the line. Here is an example of a proxy message: Here is an example of ad-hoc messages from background daemons: 4.10.9. Orchestration (heat) log files Service Service name Log path OpenStack Heat API Service openstack-heat-api.service /var/log/containers/heat/heat-api.log OpenStack Heat Engine Service openstack-heat-engine.service /var/log/containers/heat/heat-engine.log Orchestration service events n/a /var/log/containers/heat/heat-manage.log 4.10.10. Shared Filesystem Service (manila) log files Service Service name Log path OpenStack Manila API Server openstack-manila-api.service /var/log/containers/manila/api.log OpenStack Manila Scheduler openstack-manila-scheduler.service /var/log/containers/manila/scheduler.log OpenStack Manila Share Service openstack-manila-share.service /var/log/containers/manila/share.log Note Some information from the Manila Python library can also be logged in /var/log/containers/manila/manila-manage.log . 4.10.11. Telemetry (ceilometer) log files Service Service name Log path OpenStack ceilometer notification agent ceilometer_agent_notification /var/log/containers/ceilometer/agent-notification.log OpenStack ceilometer central agent ceilometer_agent_central /var/log/containers/ceilometer/central.log OpenStack ceilometer collection openstack-ceilometer-collector.service /var/log/containers/ceilometer/collector.log OpenStack ceilometer compute agent ceilometer_agent_compute /var/log/containers/ceilometer/compute.log 4.10.12. Log files for supporting services The following services are used by the core OpenStack components and have their own log directories and files. Service Service name Log path Message broker (RabbitMQ) rabbitmq-server.service /var/log/rabbitmq/rabbit@ short_hostname .log /var/log/rabbitmq/rabbit@ short_hostname -sasl.log (for Simple Authentication and Security Layer related log messages) Database server (MariaDB) mariadb.service /var/log/mariadb/mariadb.log Virtual network switch (Open vSwitch) openvswitch-nonetwork.service /var/log/openvswitch/ovsdb-server.log /var/log/openvswitch/ovs-vswitchd.log 4.10.13. aodh (alarming service) log files Service Container name Log path Alarming API aodh_api /var/log/containers/httpd/aodh-api/aodh_wsgi_access.log Alarm evaluator log aodh_evaluator /var/log/containers/aodh/aodh-evaluator.log Alarm listener aodh_listener /var/log/containers/aodh/aodh-listener.log Alarm notification aodh_notifier /var/log/containers/aodh/aodh-notifier.log 4.10.14. gnocchi (metric storage) log files Service Container name Log path Gnocchi API gnocchi_api /var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_access.log Gnocchi metricd gnocchi_metricd /var/log/containers/gnocchi/gnocchi-metricd.log Gnocchi statsd gnocchi_statsd /var/log/containers/gnocchi/gnocchi-statsd.log

Commands:
openstack overcloud deploy <existing_overcloud_environment_files> -e /usr/share/openstack-tripleo-heat-templates/environments/logging-environment-rsyslog.yaml
parameter_defaults: RsyslogElasticsearchSetting: uid: "elastic" pwd: "yourownpassword" skipverifyhost: "on" allowunsignedcerts: "on" server: "https://log-store-service-telemetry.apps.stfcloudops1.lab.upshift.rdu2.redhat.com" serverport: 443
parameter_defaults LogrotateRotate: '14' LogrotatePurgeAfterDays: '14' LogrotateMaxsize: '10M'
openstack overcloud deploy --timeout 100 --templates /usr/share/openstack-tripleo-heat-templates ... -e /home/stack/templates/rotate.yaml --log-file overcloud_deployment_90.log
podman exec -it logrotate_crond cat /etc/logrotate-crond.conf /var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err { daily rotate 14 maxage 14 # minsize 1 is required for GDPR compliance, all files in # /var/log/containers not managed with logrotate will be purged! minsize 1 # Do not use size as it's not compatible with time-based rotation rules # required for GDPR compliance. maxsize 10M missingok notifempty copytruncate delaycompress compress }
ls -lah /var/log/containers/nova/ total 48K drwxr-x---. 2 42436 42436 79 May 12 09:01 . drwxr-x---. 7 root root 82 Jan 21 2021 .. -rw-r--r--. 1 42436 42436 12K May 12 14:00 nova-compute.log -rw-r--r--. 1 42436 42436 33K May 12 09:01 nova-compute.log.1 -rw-r--r--. 1 42436 42436 0 Jan 21 2021 nova-manage.log
podman exec -it logrotate_crond /bin/sh ()[root@9410925fded9 /]USD cat /var/spool/cron/root HEADER: This file was autogenerated at 2021-01-21 16:47:27 +0000 by puppet. HEADER: While it can still be managed manually, it is definitely not recommended. HEADER: Note particularly that the comments starting with 'Puppet Name' should HEADER: not be deleted, as doing so could cause duplicate cron jobs. Puppet Name: logrotate-crond PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 0 * * * * sleep `expr USD{RANDOM} \% 90`; /usr/sbin/logrotate -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf 2>&1|logger -t logrotate-crond
NovaComputeLoggingSource: tag: openstack.nova.compute file: /some/other/path/nova-compute.log
<service_name>LoggingSource: tag: <service_name>.tag path: <service_name>.path format: <service_name>.format
ServiceLoggingSource: tag: openstack.Service path: /var/log/containers/service/service.log format: multiline format_firstline: '/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} \d+ \S+ \S+ \[(req-\S+ \S+ \S+ \S+ \S+ \S+|-)\]/' format1: '/^(?<Timestamp>\S+ \S+) (?<Pid>\d+) (?<log_level>\S+) (?<python_module>\S+) (\[(req-(?<request_id>\S+) (?<user_id>\S+) (?<tenant_id>\S+) (?<domain_id>\S+) (?<user_domain>\S+) (?<project_domain>\S+)|-)\])? (?<Payload>.*)?USD/'
NovaComputeLoggingSource: tag: openstack.nova.compute file: /some/other/path/nova-compute.log startmsg.regex: "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ \\+[0-9]+)? [A-Z]+ \\([a-z]+\\)
startmsg.regex='^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) '
Apr 20 15:20:34 rhev-a24c-01 proxy-server: 127.0.0.1 127.0.0.1 20/Apr/2015/19/20/34 GET /v1/AUTH_zaitcev%3Fformat%3Djson%26marker%3Dtestcont HTTP/1.0 200 - python-swiftclient-2.1.0 AUTH_tk737d6... - 2 - txc454fa8ea4844d909820a-0055355182 - 0.0162 - - 1429557634.806570053 1429557634.822791100
Apr 27 17:08:15 rhev-a24c-02 object-auditor: Object audit (ZBF). Since Mon Apr 27 21:08:15 2015: Locally: 1 passed, 0 quarantined, 0 errors files/sec: 4.34 , bytes/sec: 0.00, Total time: 0.23, Auditing time: 0.00, Rate: 0.00 Apr 27 17:08:16 rhev-a24c-02 object-auditor: Object audit (ZBF) "forever" mode completed: 0.56s. Total quarantined: 0, Total errors: 0, Total files/sec: 14.31, Total bytes/sec: 0.00, Auditing time: 0.02, Rate: 0.04 Apr 27 17:08:16 rhev-a24c-02 account-replicator: Beginning replication run Apr 27 17:08:16 rhev-a24c-02 account-replicator: Replication run OVER Apr 27 17:08:16 rhev-a24c-02 account-replicator: Attempted to replicate 5 dbs in 0.12589 seconds (39.71876/s) Apr 27 17:08:16 rhev-a24c-02 account-replicator: Removed 0 dbs Apr 27 17:08:16 rhev-a24c-02 account-replicator: 10 successes, 0 failures