Chapter 13. Logging and Reporting Problems This chapter provides information on how to log and report problems in Satellite, including information on relevant log files, how to enable debug logging, how to open a support case and attach the relevant log tar files, and how to access support cases within the Satellite web UI. You can use the log files and other information described in this chapter to do your own troubleshooting, or you can capture these and many more files, as well as diagnostic and configuration information, to send to Red Hat Support if you need further assistance. For more information about Satellite logging settings, use satellite-installer with the --full-help option: 13.1. Enabling Debug Logging Debug logging provides the most detailed log information and can help with troubleshooting issues that can arise with Satellite and its components. In the Satellite CLI, enable debug logging to log detailed debugging information for Satellite. Procedure To enable debug logging, enter the following command: After you complete debugging, reset the logging level to the default value: 13.2. Increasing the Logging Levels to Help with Debugging By default, Satellite comes with :INFO level logging enabled. You can increase or decrease the log levels on your Satellite. Enabling debug level logging on all components Enabling debug level logging for a specific component Reverting debug level logging to INFO Listing all components and changed configuration files 13.2.1. Increasing the Logging Level For Hammer You can find the log for Hammer in ~/.hammer/log/hammer.log . Edit /etc/hammer/cli_config.yml and set the :log_level: : 13.2.2. Increasing the Logging Level On Capsule You can find the log for Capsule in /var/log/foreman-proxy/proxy.log . Uncomment the DEBUG line in /etc/foreman-proxy/settings.yml : Ensure to restart the foreman-proxy service afterwards: Caution Running the installer will revert this change back. 13.2.3. Increasing the Logging Level For Candlepin You can find the log for Candlepin in /var/log/candlepin/candlepin.log . Errors are also logged to a separate file for easier debugging /var/log/candlepin/error.log . Extend /etc/candlepin/candlepin.conf : Ensure to restart the tomcat service afterwards: If the candlepin log files are too verbose, you can decrease the default debug level: 13.2.4. Increasing the Logging Level On Satellite You can find the log for Satellite in /var/log/foreman/production.log . Satellite stores logs for Apache in: /var/log/httpd/foreman_error.log /var/log/httpd/foreman_access.log /var/log/httpd/foreman_ssl_error.log /var/log/httpd/foreman_ssl_access.log Procedure Set the logging level in /etc/foreman/settings.yaml : Enable selected loggers in /etc/foreman/settings.yaml : Note that to see logging from some area, debug logging has to be set. Restart Satellite services: You can find the complete list of loggers with their default values in /usr/share/foreman/config/application.rb in the Foreman::Logging.add_loggers command. 13.2.5. Increasing the Logging Level For Qpid Dispatch Router Qpid logs to syslog and can be viewed in /var/log/messages or with journalctl . Enable debug logging in /etc/qpid-dispatch/qdrouterd.conf : Ensure to restart the Qpid Dispatch Router afterwards: Caution Running the installer will revert this change back. 13.2.6. Increasing the Logging Level For Qpid Broker Qpid logs to syslog and can be viewed in /var/log/messages or with journalctl . Set the log level in /etc/qpid/qpidd.conf : Ensure to restart the Qpid Broker afterwards: Caution Running the installer will revert this change. 13.2.7. Increasing the Logging Level For Redis You can find the log for Redis in /var/log/redis/redis.log . Set the log level in /etc/opt/rh/rh-redis5/redis.conf : Ensure to restart the Redis service afterwards: 13.2.8. Increasing the Logging Level For Postgres You can find the log for Postgres in /var/opt/rh/rh-postgresql12/lib/pgsql/data/log/ . Uncomment the log_statement in /var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf : Ensure to restart Satellite services afterwards: Caution Based on the size of your Satellite installation, this can cause disk space to fill up very quickly. Only turn this on if absolutely needed. For more debug log settings, refer to the Postgresql documentation . 13.2.9. Increasing the Logging Level For Satellite Installer You can find the log files in /var/log/foreman-installer/ . To increase the log level of the Satellite Installer during an install: 13.2.10. Increasing the Logging Level For Pulp By default, Pulp logs to syslog and can be viewed in /var/log/messages or with journalctl . Add the following config to the /etc/pulp/settings.py file: Ensure to restart the Pulp services afterwards: 13.2.11. Increasing the Logging Level For Puppet Agent You can increase the logging level for Puppet agent on your Satellite Server. Procedure Add the following line to the [agent] block in the /etc/puppetlabs/puppet/puppet.conf file: You can find the logs in /var/log/puppetlabs/puppet/ 13.2.12. Increasing the Logging Level For Puppet Server You can increase the logging level for Puppet server on your Satellite Server. Prerequisite Puppet must be enabled in your Satellite. For more information, see Enabling Puppet Integration with Satellite in Managing Configurations Using Puppet Integration in Red Hat Satellite . Procedure Add the following line to the [master] block in /etc/puppetlabs/puppet/puppet.conf file: Restart the Puppet server: You can find the logs in /var/log/puppetlabs/puppetserver/ . 13.3. Retrieving the Status of Services Procedure In the Satellite web UI, navigate to Administer > About . On the Smart Proxies tab, you can view the status of all Capsules. On the Compute Resources tab, you can view the status of attached compute resource providers. In the Backend System Status table, you can view the status of all back-end services. CLI procedure Run hammer ping to get information from the database and Satellite services: Use satellite-maintain to check the status of the services running in systemd: Use satellite-maintain to perform a health check: 13.4. Restarting Services Satellite uses a set of back-end services to perform tasks. You you experience an issue with your Satellite, check the status of Satellite services. Procedure Use satellite-maintain to restart Satellite services: Tip Run foreman-maintain --help for more information. 13.5. Enabling Individual Loggers You can enable individual loggers for selective logging. Satellite uses the following loggers: app Logs web requests and all general application messages. Default value: true. audit Logs additional fact statistics, numbers of added, updated, and removed facts. Default value: true. background Logs information from the background processing component. blob Logs contents of rendered templates for auditing purposes. Important The blob logger might contain sensitive data. dynflow Logs information from the Dynflow process. ldap Logs high level LDAP queries and LDAP operations. Default value: false. notifications Logs information from the notifications component. permissions Logs queries to user roles, filters, and permissions when loading pages. Default value: false. sql Logs SQL queries made through Rails ActiveRecord. Default value: false. telemetry Logs debugging information from telemetry. templates Logs information from the template renderer component. Procedure Enable the individual loggers that you want. For example, to enable sql and ldap loggers, enter the following command: Optional: To reset loggers to their default values, enter the following command: 13.6. Configuring Logging to Journal or File-based Logging Satellite uses file-based logging by default. You can use the satellite-installer command to reconfigure logging. Procedure for configuring logging with Journal Enter the following satellite-installer command to configure logging to the journald service: Optional: To inspect the log messages, use the journalctl utility. For example: journalctl --unit foreman and journalctl --unit foreman-proxy show messages for the foreman and foreman-proxy units journalctl REQUEST= request_ID shows messages for a specified request Procedure for configuring file-based logging Enter the following satellite-installer command to configure file-based logging: Optional: To inspect the log messages, view these files: /var/log/foreman/production.log /var/log/foreman-proxy.log Additional resources For more information about Journal, see Viewing logs using the command line in the Red Hat Enterprise Linux 8 Configuring Basic System Settings Guide . 13.7. Log File Directories Provided by Satellite Red Hat Satellite provides system information in the form of notifications and log files. Table 13.1. Log File Directories for Reporting and Troubleshooting Log File Directories Description of Log File Content /var/log/candlepin Subscription management /var/log/foreman-installer Installer /var/log/foreman-maintain Foreman maintain /var/log/foreman-proxy Foreman proxy /var/log/foreman Foreman /var/log/httpd Apache HTTP server /var/log/messages Various other log messages /var/log/puppetlabs/puppet Configuration management /var/log/rhsm Subscription management /var/log/tomcat Candlepin webservice logs You can also use the foreman-tail command to follow many of the log files related to Satellite. You can run foreman-tail -l to list the processes and services that it follows. 13.8. Utilities for Collecting Log Information You can collect information from log files to troubleshoot Satellite. sosreport The sosreport command collects configuration and diagnostic information from a Linux system, such as the running Kernel version, loaded modules, running services, and system and service configuration files. This output is stored in a tar file located at /var/tmp/ sosreport-XXX-20171002230919.tar.xz . For more information, run sosreport --help or see What is a sosreport and how can I create one? . Important The collection process removes security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Satellite Server. Red Hat recommends that you send this information directly to the intended recipient and not to a public target. 13.9. System Journal Metadata The following table lists metadata that the journald service uses in Satellite. You can use this metadata to filter your queries. Table 13.2. System Journal Metadata Name Description AUDIT_ACTION Audit action performed Example: Create, update, or delete AUDIT_TYPE Audit resource type Example: Host, Subnet, or ContentView AUDIT_ID Audit resource database ID as a number AUDIT_ATTRIBUTE Audit resource field or an updated database column AUDIT_FIELD_OLD Old audit value of an update action AUDIT_FIELD_NEW New audit value of an update action AUDIT_ID Record database ID of the audit subject AUDIT_ATTRIBUTE Attribute name or column on which an action was performed Example: Name or description EXCEPTION_MESSAGE Exception message when error is logged EXCEPTION_CLASS Exception Ruby class when error is logged EXCEPTION_BACKTRACE Exception backtrace as a multiline string when error is logged LOC_ID Location database ID LOC_NAME Location name LOC_LABEL Location label LOGGER Logger name To see the current list of loggers enabled by default, enter this command: ORG_ID Organization database ID ORG_NAME Organization name ORG_LABEL Organization label REMOTE_IP Remote IP address of a client REQUEST Request ID generated by the Action Dispatch module SESSION Random ID generated per session or a request for a sessionless request TEMPLATE_NAME Template name TEMPLATE_DIGEST Digest (SHA256) of rendered template contents TEMPLATE_HOST_NAME Host name for a rendered template if present TEMPLATE_HOST_ID Host database ID for a rendered template if present USER_LOGIN User login name

Commands:
satellite-installer --full-help | grep logging
satellite-installer --foreman-logging-level debug
satellite-installer --reset-foreman-logging-level
hammer admin logging --all --level-debug satellite-maintain service restart
hammer admin logging --components " Component " --level-debug
hammer admin logging --all --level-production satellite-maintain service restart
hammer admin logging --list -----------|-------------------------------------|------------------------------------- COMPONENT | AUTO-DETECTED BY EXISTENCE OF | DESTINATIONS -----------|-------------------------------------|------------------------------------- dhcpd | /etc/dhcp/dhcpd.conf | syslog /var/log/dhcpd-debug.log postgresql | /var/lib/pgsql/data/postgresql.conf | syslog /var/lib/pgsql/data/pg_log/ proxy | /etc/foreman-proxy/settings.yml | /var/log/foreman-proxy/proxy.log qpidd | /etc/qpid/qpidd.conf | syslog rails | /etc/foreman/settings.yaml | /var/log/foreman/production.log tomcat | /etc/candlepin/candlepin.conf | /var/log/candlepin/ /var/log/tomcat/ virt-who | /etc/sysconfig/virt-who | syslog -----------|-------------------------------------|-------------------------------------
:log_level: 'debug'
:log_level: DEBUG
systemctl restart foreman-proxy
log4j.logger.org.candlepin=DEBUG
systemctl restart tomcat
log4j.logger.org.candlepin.resource.ConsumerResource=WARN log4j.logger.org.candlepin.resource.HypervisorResource=WARN
:logging: :production: :type: file :layout: pattern :level: debug
:loggers: :ldap: :enabled: true :permissions: :enabled: true :sql: :enabled: true
satellite-maintain service restart
enable: debug+
systemctl restart qdrouterd
log-enable=debug+
systemctl restart qpidd
loglevel debug
systemctl restart rh-redis5-redis
log_statement = 'all'
satellite-maintain service restart
satellite-installer --verbose-log-level debug
LOGGING = {"dynaconf_merge": True, "loggers": {'': {'handlers': ['console'], 'level': 'DEBUG'}}}
systemctl restart pulpcore-api pulpcore-content pulpcore-resource-manager pulpcore-worker@1 pulpcore-worker@2 rh-redis5-redis
[agent] log_level = debug
[master] log_level = debug
satellite-maintain service restart --only puppetserver
hammer ping
satellite-maintain service status
satellite-maintain health check
satellite-maintain service restart
satellite-installer --foreman-loggers ldap:true --foreman-loggers sql:true
satellite-installer --reset-foreman-loggers
satellite-installer --foreman-logging-layout pattern --foreman-logging-type journald --foreman-proxy-log JOURNAL
satellite-installer --reset-foreman-logging-layout --reset-foreman-logging-type --reset-foreman-proxy-log
awk '/add_loggers/,/^USD/' /usr/share/foreman/config/application.rb