Chapter 16. Logging and reporting problems The following sections describe the most commonly used log files and debugging tools available in Satellite. 16.1. Increasing logging levels to help with debugging You can increase the logging level of Satellite components to troubleshoot Satellite. Increasing the logging level to debug provides the most detailed information. The default logging level is info . 16.1.1. Increasing the logging level for the Foreman application To increase logging for the Foreman application: Set the logging level to debug : After you complete debugging, reset the logging level to the default value: For more information about Satellite logging settings, use satellite-installer with the --full-help option: 16.1.2. Increasing the logging level for Hammer You can find the log in ~/.hammer/log/hammer.log . Procedure In /etc/hammer/cli_config.yml , set the :log_level: option to debug : 16.1.3. Increasing the logging level for Capsule You can find the log in /var/log/foreman-proxy/proxy.log . To increase logging for Capsule: Set the logging level to debug : After you complete debugging, reset the logging level to the default value: Note Running satellite-installer will revert the setting back to default. 16.1.4. Increasing the logging level for Candlepin You can find the logs in /var/log/candlepin/candlepin.log and /var/log/candlepin/error.log . To increase logging for Candlepin: Set the logging level to DEBUG : If the candlepin log files are too verbose, you can decrease the default debug level: After you complete debugging, reset the logging level to the default value: 16.1.5. Increasing the logging level for Redis You can find the log for Redis in /var/log/redis/redis.log . To increase logging for Redis: In /etc/redis.conf , set the logging level to debug : Restart the Redis service: Note Running satellite-installer will revert the setting to default. 16.1.6. Increasing the logging level for Satellite installer You can find the log files in /var/log/foreman-installer/ . To increase the logging level of the Satellite installer, add the --verbose-log-level debug option during the installation: 16.1.7. Increasing the logging level for Pulp By default, Pulp logs to syslog . You can view the log in /var/log/messages or with journalctl . To increase logging for Pulp: In /etc/pulp/settings.py , set the logging level to DEBUG : Restart the Pulp services: 16.1.8. Increasing the logging level for Puppet agent You can find the logs in /var/log/puppetlabs/puppet/ . To increase logging for Puppet agent: Ensure Puppet is enabled in your Satellite. For more information, see Enabling Puppet Integration with Satellite in Managing configurations by using Puppet integration . Set the logging level to debug : 16.1.9. Increasing the logging level for Puppet server You can find the logs in /var/log/puppetlabs/puppetserver/ . To increase logging for Puppet server: Ensure Puppet is enabled in your Satellite. For more information, see Enabling Puppet Integration with Satellite in Managing configurations by using Puppet integration . Set the logging level to debug : Restart the Puppet server: 16.2. Configuring logging levels with Hammer CLI You can use Hammer to log debugging information for various Satellite components. The default logging level is INFO . To list the components whose logging level you can change by using hammer : To set DEBUG level logging for all components: To set PRODUCTION level logging for all components: To set DEBUG level logging for a specific component: To list all available logging options: Additional resources Using the Hammer CLI tool 16.3. Configuring logging type and layout By default, Satellite uses file-based logging. You can use satellite-installer to change the logging type and logging layout. Procedure for configuring logging with journal Enter the following satellite-installer command to configure logging to the journald service: Optional: To inspect the log messages, use the journalctl utility. For example: journalctl --unit foreman and journalctl --unit foreman-proxy show messages for the foreman and foreman-proxy units journalctl REQUEST= request_ID shows messages for a specified request Procedure for configuring file-based logging Enter the following satellite-installer command to configure file-based logging: Optional: To inspect the log messages, view these files: /var/log/foreman/production.log /var/log/foreman-proxy.log Procedure for configuring logging to JSON output On your Satellite Server, enable logging to JSON output for satellite-installer : Optional: Inspect the log messages using jq : Additional resources For more information about Journal, see Viewing logs using the command line in Red Hat Enterprise Linux 8 Configuring basic system settings . 16.4. Selective debugging with individual loggers You can enable individual loggers for selective debugging. Procedure Enable the required individual loggers. For example, to enable sql and ldap loggers, enter the following command: Optional: Reset all loggers to their default values: 16.5. Overview of individual loggers The following list provides an overview of selected loggers. You can find the complete list of loggers with their default values in /usr/share/foreman/config/application.rb under Foreman::Logging.add_loggers . app Logs web requests and all general application messages. Default value: true. audit Logs additional fact statistics, numbers of added, updated, and removed facts. Default value: true. background Logs information from the background processing component. blob Logs contents of rendered templates for auditing purposes. Important The blob logger might contain sensitive data. dynflow Logs information from the Dynflow process. ldap Logs high level LDAP queries and LDAP operations. Default value: false. notifications Logs information from the notifications component. permissions Logs queries to user roles, filters, and permissions when loading pages. Default value: false. sql Logs SQL queries made through Rails ActiveRecord. Default value: false. telemetry Logs debugging information from telemetry. templates Logs information from the template renderer component. 16.6. Retrieving the status of services Satellite uses a set of back-end services. When troubleshooting, you can check the status of Satellite services. Procedure In the Satellite web UI, navigate to Administer > About . On the Smart Proxies tab, view the status of all Capsules. On the Compute Resources tab, view the status of attached compute resource providers. In the Backend System Status table, view the status of all back-end services. CLI procedure Get information from the database and Satellite services: Check the status of the services running in systemd: Run satellite-maintain service --help for more information. Perform a health check: Run satellite-maintain health --help for more information. 16.7. Restarting services Satellite uses a set of back-end services. When troubleshooting, you can restart the services if needed. Procedure Restart Satellite services: 16.8. Utilities for processing log information You can collect information from log files to troubleshoot Satellite. sosreport The sos report command collects configuration and diagnostic information from a Linux system, such as the running kernel version, loaded modules, running services, and system and service configuration files. Additionally, it collects information about Red Hat Satellite, such as its back-end services and tasks. This output is stored in a tar file located at /var/tmp/ sosreport-XXX-20171002230919.tar.xz . For more information, run sos report --help or see What is a sosreport and how can I create one? in the Red Hat Knowledgebase . Important The sos report command removes security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Satellite Server. Send the tar files directly to the intended recipient and not to a public target. foreman-tail The foreman-tail command displays Satellite logs in real time. For more information, see the foreman-tail(8) man page. 16.9. Log file directories The following table provides an overview of selected log directories on Satellite Server: Log file directory Description of log file content /var/log/candlepin/ Subscription management /var/log/foreman-installer/ Installer /var/log/foreman-maintain/ Foreman maintain /var/log/foreman-proxy/ Foreman proxy /var/log/foreman/ Foreman /var/log/httpd/ Apache HTTP server /var/log/messages/ Various other log messages /var/log/puppetlabs/puppet/ Puppet /var/log/tomcat/ Candlepin web service logs 16.10. System journal metadata The following table lists metadata that the journald service uses in Satellite. You can use this metadata to filter your queries. AUDIT_ACTION Audit action performed Example: Create, update, or delete AUDIT_TYPE Audit resource type Example: Host, Subnet, or ContentView AUDIT_ID Audit resource database ID as a number AUDIT_ATTRIBUTE Audit resource field or an updated database column AUDIT_FIELD_OLD Old audit value of an update action AUDIT_FIELD_NEW New audit value of an update action AUDIT_ID Record database ID of the audit subject AUDIT_ATTRIBUTE Attribute name or column on which an action was performed Example: Name or description EXCEPTION_MESSAGE Exception message when error is logged EXCEPTION_CLASS Exception Ruby class when error is logged EXCEPTION_BACKTRACE Exception backtrace as a multiline string when error is logged LOC_ID Location database ID LOC_NAME Location name LOC_LABEL Location label LOGGER Logger name To see the current list of loggers enabled by default, enter this command: ORG_ID Organization database ID ORG_NAME Organization name ORG_LABEL Organization label REMOTE_IP Remote IP address of a client REQUEST Request ID generated by the Action Dispatch module SESSION Random ID generated per session or a request for a sessionless request TEMPLATE_NAME Template name TEMPLATE_DIGEST Digest (SHA256) of rendered template contents TEMPLATE_HOST_NAME Host name for a rendered template if present TEMPLATE_HOST_ID Host database ID for a rendered template if present USER_LOGIN User login name

Commands:
satellite-installer --foreman-logging-level debug
satellite-installer --reset-foreman-logging-level
satellite-installer --full-help | grep logging
:log_level: 'debug'
satellite-installer --foreman-proxy-log-level DEBUG
satellite-installer --reset-foreman-proxy-log-level
satellite-installer --katello-candlepin-loggers log4j.logger.org.candlepin:DEBUG
satellite-installer --katello-candlepin-loggers log4j.logger.org.candlepin:DEBUG --katello-candlepin-loggers log4j.logger.org.candlepin.resource.ConsumerResource:WARN --katello-candlepin-loggers log4j.logger.org.candlepin.resource.HypervisorResource:WARN
satellite-installer --reset-katello-candlepin-loggers
loglevel debug
systemctl restart redis
satellite-installer --verbose-log-level debug
LOGGING = {"dynaconf_merge": True, "loggers": {'': {'handlers': ['console'], 'level': 'DEBUG'}}}
systemctl restart pulpcore-api pulpcore-content pulpcore-resource-manager pulpcore-worker@1 pulpcore-worker@2 redis
satellite-installer --puppet-agent-additional-settings log_level:debug
satellite-installer --puppet-server-additional-settings log_level:debug
satellite-maintain service restart --only puppetserver
hammer admin logging --list
hammer admin logging --all --level-debug satellite-maintain service restart
hammer admin logging --all --level-production satellite-maintain service restart
hammer admin logging --components My_Component --level-debug satellite-maintain service restart
hammer admin logging --help
satellite-installer --foreman-logging-type journald --foreman-proxy-log JOURNAL
satellite-installer --reset-foreman-logging-type --reset-foreman-proxy-log
satellite-installer --foreman-logging-layout json --foreman-logging-type file
cat /var/log/foreman/production.log | jq
satellite-installer --foreman-loggers ldap:true --foreman-loggers sql:true
satellite-installer --reset-foreman-loggers
hammer ping
satellite-maintain service status
satellite-maintain health check
satellite-maintain service restart
awk '/add_loggers/,/^USD/' /usr/share/foreman/config/application.rb