Chapter 5. Troubleshooting problems related to SELinux If you plan to enable SELinux on systems where it has been previously disabled or if you run a service in a non-standard configuration, you might need to troubleshoot situations potentially blocked by SELinux. Note that in most cases, SELinux denials are signs of misconfiguration. 5.1. Identifying SELinux denials Follow only the necessary steps from this procedure; in most cases, you need to perform just step 1. Procedure When your scenario is blocked by SELinux, the /var/log/audit/audit.log file is the first place to check for more information about a denial. To query Audit logs, use the ausearch tool. Because the SELinux decisions, such as allowing or disallowing access, are cached and this cache is known as the Access Vector Cache (AVC), use the AVC and USER_AVC values for the message type parameter, for example: If there are no matches, check if the Audit daemon is running. If it does not, repeat the denied scenario after you start auditd and check the Audit log again. In case auditd is running, but there are no matches in the output of ausearch , check messages provided by the systemd Journal: If SELinux is active and the Audit daemon is not running on your system, then search for certain SELinux messages in the output of the dmesg command: Even after the three checks, it is still possible that you have not found anything. In this case, AVC denials can be silenced because of dontaudit rules. To temporarily disable dontaudit rules, allowing all denials to be logged: After re-running your denied scenario and finding denial messages using the steps, the following command enables dontaudit rules in the policy again: If you apply all four steps, and the problem still remains unidentified, consider if SELinux really blocks your scenario: Switch to permissive mode: Repeat your scenario. If the problem still occurs, something different than SELinux is blocking your scenario. 5.2. Analyzing SELinux denial messages After identifying that SELinux is blocking your scenario, you might need to analyze the root cause before you choose a fix. Prerequisites The policycoreutils-python-utils and setroubleshoot-server packages are installed on your system. Procedure List more details about a logged denial using the sealert command, for example: If the output obtained in the step does not contain clear suggestions: Enable full-path auditing to see full paths to accessed objects and to make additional Linux Audit event fields visible: Clear the setroubleshoot cache: Reproduce the problem. Repeat step 1. After you finish the process, disable full-path auditing: If sealert returns only catchall suggestions or suggests adding a new rule using the audit2allow tool, match your problem with examples listed and explained in SELinux denials in the Audit log . Additional resources sealert(8) man page on your system 5.3. Fixing analyzed SELinux denials In most cases, suggestions provided by the sealert tool give you the right guidance about how to fix problems related to the SELinux policy. See Analyzing SELinux denial messages for information how to use sealert to analyze SELinux denials. Be careful when the tool suggests using the audit2allow tool for configuration changes. You should not use audit2allow to generate a local policy module as your first option when you see an SELinux denial. Troubleshooting should start with a check if there is a labeling problem. The second most often case is that you have changed a process configuration, and you forgot to tell SELinux about it. Labeling problems A common cause of labeling problems is when a non-standard directory is used for a service. For example, instead of using /var/www/html/ for a website, an administrator might want to use /srv/myweb/ . On Red Hat Enterprise Linux, the /srv directory is labeled with the var_t type. Files and directories created in /srv inherit this type. Also, newly-created objects in top-level directories, such as /myserver , can be labeled with the default_t type. SELinux prevents the Apache HTTP Server ( httpd ) from accessing both of these types. To allow access, SELinux must know that the files in /srv/myweb/ are to be accessible by httpd : This semanage command adds the context for the /srv/myweb/ directory and all files and directories under it to the SELinux file-context configuration. The semanage utility does not change the context. As root, use the restorecon utility to apply the changes: Incorrect context The matchpathcon utility checks the context of a file path and compares it to the default label for that path. The following example demonstrates the use of matchpathcon on a directory that contains incorrectly labeled files: USD matchpathcon -V /var/www/html/* /var/www/html/index.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0 /var/www/html/page1.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0 In this example, the index.html and page1.html files are labeled with the user_home_t type. This type is used for files in user home directories. Using the mv command to move files from your home directory may result in files being labeled with the user_home_t type. This type should not exist outside of home directories. Use the restorecon utility to restore such files to their correct type: # restorecon -v /var/www/html/index.html restorecon reset /var/www/html/index.html context unconfined_u:object_r:user_home_t:s0->system_u:object_r:httpd_sys_content_t:s0 To restore the context for all files under a directory, use the -R option: # restorecon -R -v /var/www/html/ restorecon reset /var/www/html/page1.html context unconfined_u:object_r:samba_share_t:s0->system_u:object_r:httpd_sys_content_t:s0 restorecon reset /var/www/html/index.html context unconfined_u:object_r:samba_share_t:s0->system_u:object_r:httpd_sys_content_t:s0 Confined applications configured in non-standard ways Services can be run in a variety of ways. To account for that, you need to specify how you run your services. You can achieve this through SELinux booleans that allow parts of SELinux policy to be changed at runtime. This enables changes, such as allowing services access to NFS volumes, without reloading or recompiling SELinux policy. Also, running services on non-default port numbers requires policy configuration to be updated using the semanage command. For example, to allow the Apache HTTP Server to communicate with MariaDB, enable the httpd_can_network_connect_db boolean: Note that the -P option makes the setting persistent across reboots of the system. If access is denied for a particular service, use the getsebool and grep utilities to see if any booleans are available to allow access. For example, use the getsebool -a | grep ftp command to search for FTP related booleans: To get a list of booleans and to find out if they are enabled or disabled, use the getsebool -a command. To get a list of booleans including their meaning, and to find out if they are enabled or disabled, install the selinux-policy-devel package and use the semanage boolean -l command as root. Port numbers Depending on policy configuration, services can only be allowed to run on certain port numbers. Attempting to change the port a service runs on without changing policy may result in the service failing to start. For example, run the semanage port -l | grep http command as root to list http related ports: # semanage port -l | grep http http_cache_port_t tcp 3128, 8080, 8118 http_cache_port_t udp 3130 http_port_t tcp 80, 443, 488, 8008, 8009, 8443 pegasus_http_port_t tcp 5988 pegasus_https_port_t tcp 5989 The http_port_t port type defines the ports Apache HTTP Server can listen on, which in this case, are TCP ports 80, 443, 488, 8008, 8009, and 8443. If an administrator configures httpd.conf so that httpd listens on port 9876 ( Listen 9876 ), but policy is not updated to reflect this, the following command fails: # systemctl start httpd.service Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details. # systemctl status httpd.service httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled) Active: failed (Result: exit-code) since Thu 2013-08-15 09:57:05 CEST; 59s ago Process: 16874 ExecStop=/usr/sbin/httpd USDOPTIONS -k graceful-stop (code=exited, status=0/SUCCESS) Process: 16870 ExecStart=/usr/sbin/httpd USDOPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) An SELinux denial message similar to the following is logged to /var/log/audit/audit.log : To allow httpd to listen on a port that is not listed for the http_port_t port type, use the semanage port command to assign a different label to the port: The -a option adds a new record; the -t option defines a type; and the -p option defines a protocol. The last argument is the port number to add. Corner cases, evolving or broken applications, and compromised systems Applications may contain bugs, causing SELinux to deny access. Also, SELinux rules are evolving - SELinux may not have seen an application running in a certain way, possibly causing it to deny access, even though the application is working as expected. For example, if a new version of PostgreSQL is released, it may perform actions the current policy does not account for, causing access to be denied, even though access should be allowed. For these situations, after access is denied, use the audit2allow utility to create a custom policy module to allow access. You can report missing rules in the SELinux policy in Red Hat Bugzilla . For Red Hat Enterprise Linux 8, create bugs against the Red Hat Enterprise Linux 8 product, and select the selinux-policy component. Include the output of the audit2allow -w -a and audit2allow -a commands in such bug reports. If an application asks for major security privileges, it could be a signal that the application is compromised. Use intrusion detection tools to inspect such suspicious behavior. The Solution Engine on the Red Hat Customer Portal can also provide guidance in the form of an article containing a possible solution for the same or very similar problem you have. Select the relevant product and version and use SELinux-related keywords, such as selinux or avc , together with the name of your blocked service or application, for example: selinux samba . 5.4. Creating a local SELinux policy module Adding specific SELinux policy modules to an active SELinux policy can fix certain problems with the SELinux policy. You can use this procedure to fix a specific Known Issue described in Red Hat release notes , or to implement a specific Red Hat Solution . Warning Use only rules provided by Red Hat. Red Hat does not support creating SELinux policy modules with custom rules, because this falls outside of the Production Support Scope of Coverage . If you are not an expert, contact your Red Hat sales representative and request consulting services. Prerequisites The setools-console and audit packages for verification. Procedure Open a new .cil file with a text editor, for example: To keep your local modules better organized, use the local_ prefix in the names of local SELinux policy modules. Insert the custom rules from a Known Issue or a Red Hat Solution. Important Do not write your own rules. Use only the rules provided in a specific Known Issue or Red Hat Solution. For example, to implement the SELinux denies cups-lpd read access to cups.sock in RHEL solution, insert the following rule: The example solution has been fixed permanently for RHEL in RHBA-2021:4420 . Therefore, the parts of this procedure specific to this solution have no effect on updated RHEL 8 and 9 systems, and are included only as examples of syntax. You can use either of the two SELinux rule syntaxes, Common Intermediate Language (CIL) and m4. For example, (allow cupsd_lpd_t cupsd_var_run_t (sock_file (read))) in CIL is equivalent to the following in m4: Save and close the file. Install the policy module: If you want to remove a local policy module which you created by using semodule -i , refer to the module name without the .cil suffix. To remove a local policy module, use semodule -r <local_module> . Restart any services related to the rules: Verification List the local modules installed in your SELinux policy: Because local modules have priority 400 , you can filter them from the list also by using that value, for example, by using the semodule -lfull | grep -v ^100 command. Search the SELinux policy for the relevant allow rules: Where <SOURCENAME> is the source SELinux type, <TARGETNAME> is the target SELinux type, <CLASSNAME> is the security class or object class name, and <P1> and <P2> are the specific permissions of the rule. For example, for the SELinux denies cups-lpd read access to cups.sock in RHEL solution: The last line should now include the read operation. Verify that the relevant service runs confined by SELinux: Identify the process related to the relevant service: Check the SELinux context of the process listed in the output of the command: Verify that the service does not cause any SELinux denials: The -i option interprets the numeric values into human-readable text. Additional resources How to create custom SELinux policy module wisely Knowledgebase article 5.5. SELinux denials in the Audit log The Linux Audit system stores log entries in the /var/log/audit/audit.log file by default. To list only SELinux-related records, use the ausearch command with the message type parameter set to AVC and AVC_USER at a minimum, for example: An SELinux denial entry in the Audit log file can look as follows: The most important parts of this entry are: avc: denied - the action performed by SELinux and recorded in Access Vector Cache (AVC) { read } - the denied action pid=6591 - the process identifier of the subject that tried to perform the denied action comm="httpd" - the name of the command that was used to invoke the analyzed process httpd_t - the SELinux type of the process nfs_t - the SELinux type of the object affected by the process action tclass=dir - the target object class The log entry can be translated to: SELinux denied the httpd process with PID 6591 and the httpd_t type to read from a directory with the nfs_t type. The following SELinux denial message occurs when the Apache HTTP Server attempts to access a directory labeled with a type for the Samba suite: { getattr } - the getattr entry indicates the source process was trying to read the target file's status information. This occurs before reading files. SELinux denies this action because the process accesses the file and it does not have an appropriate label. Commonly seen permissions include getattr , read , and write . path="/var/www/html/file1" - the path to the object (target) the process attempted to access. scontext="unconfined_u:system_r:httpd_t:s0" - the SELinux context of the process (source) that attempted the denied action. In this case, it is the SELinux context of the Apache HTTP Server, which is running with the httpd_t type. tcontext="unconfined_u:object_r:samba_share_t:s0" - the SELinux context of the object (target) the process attempted to access. In this case, it is the SELinux context of file1 . This SELinux denial can be translated to: SELinux denied the httpd process with PID 2465 to access the /var/www/html/file1 file with the samba_share_t type, which is not accessible to processes running in the httpd_t domain unless configured otherwise. Additional resources auditd(8) and ausearch(8) man pages on your system 5.6. Additional resources Basic SELinux Troubleshooting in CLI What is SELinux trying to tell me? The 4 key causes of SELinux errors

Commands:
ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERR -ts recent
journalctl -t setroubleshoot
dmesg | grep -i -e type=1300 -e type=1400
semodule -DB
semodule -B
setenforce 0 getenforce Permissive
sealert -l "*" SELinux is preventing /usr/bin/passwd from write access on the file /root/test. ***** Plugin leaks (86.2 confidence) suggests ***************************** If you want to ignore passwd trying to write access the test file, because you believe it should not need this access. Then you should report this as a bug. You can generate a local policy module to dontaudit this access. Do ausearch -x /usr/bin/passwd --raw | audit2allow -D -M my-passwd semodule -X 300 -i my-passwd.pp ***** Plugin catchall (14.7 confidence) suggests ************************** Raw Audit Messages type=AVC msg=audit(1553609555.619:127): avc: denied { write } for pid=4097 comm="passwd" path="/root/test" dev="dm-0" ino=17142697 scontext=unconfined_u:unconfined_r:passwd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0 Hash: passwd,passwd_t,admin_home_t,file,write
auditctl -w /etc/shadow -p w -k shadow-write
rm -f /var/lib/setroubleshoot/setroubleshoot.xml
auditctl -W /etc/shadow -p w -k shadow-write
semanage fcontext -a -t httpd_sys_content_t "/srv/myweb(/.*)?"
restorecon -R -v /srv/myweb
USD matchpathcon -V /var/www/html/* /var/www/html/index.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0 /var/www/html/page1.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0
# restorecon -v /var/www/html/index.html restorecon reset /var/www/html/index.html context unconfined_u:object_r:user_home_t:s0->system_u:object_r:httpd_sys_content_t:s0
# restorecon -R -v /var/www/html/ restorecon reset /var/www/html/page1.html context unconfined_u:object_r:samba_share_t:s0->system_u:object_r:httpd_sys_content_t:s0 restorecon reset /var/www/html/index.html context unconfined_u:object_r:samba_share_t:s0->system_u:object_r:httpd_sys_content_t:s0
# setsebool -P httpd_can_network_connect_db on
USD getsebool -a | grep ftp ftpd_anon_write --> off ftpd_full_access --> off ftpd_use_cifs --> off ftpd_use_nfs --> off ftpd_connect_db --> off httpd_enable_ftp_server --> off tftp_anon_write --> off
# semanage port -l | grep http http_cache_port_t tcp 3128, 8080, 8118 http_cache_port_t udp 3130 http_port_t tcp 80, 443, 488, 8008, 8009, 8443 pegasus_http_port_t tcp 5988 pegasus_https_port_t tcp 5989
# systemctl start httpd.service Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details. # systemctl status httpd.service httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled) Active: failed (Result: exit-code) since Thu 2013-08-15 09:57:05 CEST; 59s ago Process: 16874 ExecStop=/usr/sbin/httpd USDOPTIONS -k graceful-stop (code=exited, status=0/SUCCESS) Process: 16870 ExecStart=/usr/sbin/httpd USDOPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
type=AVC msg=audit(1225948455.061:294): avc: denied { name_bind } for pid=4997 comm="httpd" src=9876 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket
# semanage port -a -t http_port_t -p tcp 9876
vim <local_module> .cil
(allow cupsd_lpd_t cupsd_var_run_t (sock_file (read)))
module local_cupslpd-read-cupssock 1.0; require { type cupsd_var_run_t; type cupsd_lpd_t; class sock_file read; } #============= cupsd_lpd_t ============== allow cupsd_lpd_t cupsd_var_run_t:sock_file read;
semodule -i <local_module> .cil
systemctl restart <service-name>
semodule -lfull | grep "local_" 400 local_module cil
sesearch -A --source= <SOURCENAME> --target= <TARGETNAME> --class= <CLASSNAME> --perm= <P1> , <P2>
sesearch -A --source=cupsd_lpd_t --target=cupsd_var_run_t --class=sock_file --perm=read allow cupsd_lpd_t cupsd_var_run_t:sock_file { append getattr open read write };
systemctl status <service-name>
ps -efZ | grep <process-name>
ausearch -m AVC -i -ts recent <no matches>
ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERR
type=AVC msg=audit(1395177286.929:1638): avc: denied { read } for pid=6591 comm="httpd" name="webpages" dev="0:37" ino=2112 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=dir
type=AVC msg=audit(1226874073.147:96): avc: denied { getattr } for pid=2465 comm="httpd" path="/var/www/html/file1" dev=dm-0 ino=284133 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:samba_share_t:s0 tclass=file