Chapter 7. Security Profiles Operator 7.1. Security Profiles Operator overview OpenShift Container Platform Security Profiles Operator (SPO) provides a way to define secure computing ( seccomp ) profiles and SELinux profiles as custom resources, synchronizing profiles to every node in a given namespace. For the latest updates, see the release notes . The SPO can distribute custom resources to each node while a reconciliation loop ensures that the profiles stay up-to-date. See Understanding the Security Profiles Operator . The SPO manages SELinux policies and seccomp profiles for namespaced workloads. For more information, see Enabling the Security Profiles Operator . You can create seccomp and SELinux profiles, bind policies to pods, record workloads, and synchronize all worker nodes in a namespace. Use Advanced Security Profile Operator tasks to enable the log enricher, configure webhooks and metrics, or restrict profiles to a single namespace. Troubleshoot the Security Profiles Operator as needed, or engage Red Hat support . You can Uninstall the Security Profiles Operator by removing the profiles before removing the Operator. 7.2. Security Profiles Operator release notes The Security Profiles Operator provides a way to define secure computing ( seccomp ) and SELinux profiles as custom resources, synchronizing profiles to every node in a given namespace. These release notes track the development of the Security Profiles Operator in OpenShift Container Platform. For an overview of the Security Profiles Operator, see xref:[Security Profiles Operator Overview]. 7.2.1. Security Profiles Operator 0.8.6 The following advisory is available for the Security Profiles Operator 0.8.6: RHBA-2024:10380 - OpenShift Security Profiles Operator update This update includes upgraded dependencies in underlying base images. 7.2.2. Security Profiles Operator 0.8.5 The following advisory is available for the Security Profiles Operator 0.8.5: RHBA-2024:5016 - OpenShift Security Profiles Operator bug fix update 7.2.2.1. Bug fixes When attempting to install the Security Profile Operator from the web console, the option to enable Operator-recommended cluster monitoring was unavailable for the namespace. With this update, you can now enabled Operator-recommend cluster monitoring in the namespace. ( OCPBUGS-37794 ) Previously, the Security Profiles Operator would intermittently be not visible in the OperatorHub, which caused limited access to install the Operator via the web console. With this update, the Security Profiles Operator is present in the OperatorHub. 7.2.3. Security Profiles Operator 0.8.4 The following advisory is available for the Security Profiles Operator 0.8.4: RHBA-2024:4781 - OpenShift Security Profiles Operator bug fix update This update addresses CVEs in underlying dependencies. 7.2.3.1. New features and enhancements You can now specify a default security profile in the image attribute of a ProfileBinding object by setting a wildcard. For more information, see Binding workloads to profiles with ProfileBindings (SELinux) and Binding workloads to profiles with ProfileBindings (Seccomp) . 7.2.4. Security Profiles Operator 0.8.2 The following advisory is available for the Security Profiles Operator 0.8.2: RHBA-2023:5958 - OpenShift Security Profiles Operator bug fix update 7.2.4.1. Bug fixes Previously, SELinuxProfile objects did not inherit custom attributes from the same namespace. With this update, the issue has now been resolved and SELinuxProfile object attributes are inherited from the same namespace as expected. ( OCPBUGS-17164 ) Previously, RawSELinuxProfiles would hang during the creation process and would not reach an Installed state. With this update, the issue has been resolved and RawSELinuxProfiles are created successfully. ( OCPBUGS-19744 ) Previously, patching the enableLogEnricher to true would cause the seccompProfile log-enricher-trace pods to be stuck in a Pending state. With this update, log-enricher-trace pods reach an Installed state as expected. ( OCPBUGS-22182 ) Previously, the Security Profiles Operator generated high cardinality metrics, causing Prometheus pods using high amounts of memory. With this update, the following metrics will no longer apply in the Security Profiles Operator namespace: rest_client_request_duration_seconds rest_client_request_size_bytes rest_client_response_size_bytes ( OCPBUGS-22406 ) 7.2.5. Security Profiles Operator 0.8.0 The following advisory is available for the Security Profiles Operator 0.8.0: RHBA-2023:4689 - OpenShift Security Profiles Operator bug fix update 7.2.5.1. Bug fixes Previously, while trying to install Security Profiles Operator in a disconnected cluster, the secure hashes provided were incorrect due to a SHA relabeling issue. With this update, the SHAs provided work consistently with disconnected environments. ( OCPBUGS-14404 ) 7.2.6. Security Profiles Operator 0.7.1 The following advisory is available for the Security Profiles Operator 0.7.1: RHSA-2023:2029 - OpenShift Security Profiles Operator bug fix update 7.2.6.1. New features and enhancements Security Profiles Operator (SPO) now automatically selects the appropriate selinuxd image for RHEL 8- and 9-based RHCOS systems. Important Users that mirror images for disconnected environments must mirror both selinuxd images provided by the Security Profiles Operator. You can now enable memory optimization inside of an spod daemon. For more information, see Enabling memory optimization in the spod daemon . Note SPO memory optimization is not enabled by default. The daemon resource requirements are now configurable. For more information, see Customizing daemon resource requirements . The priority class name is now configurable in the spod configuration. For more information, see Setting a custom priority class name for the spod daemon pod . 7.2.6.2. Deprecated and removed features The default nginx-1.19.1 seccomp profile is now removed from the Security Profiles Operator deployment. 7.2.6.3. Bug fixes Previously, a Security Profiles Operator (SPO) SELinux policy did not inherit low-level policy definitions from the container template. If you selected another template, such as net_container, the policy would not work because it required low-level policy definitions that only existed in the container template. This issue occurred when the SPO SELinux policy attempted to translate SELinux policies from the SPO custom format to the Common Intermediate Language (CIL) format. With this update, the container template appends to any SELinux policies that require translation from SPO to CIL. Additionally, the SPO SELinux policy can inherit low-level policy definitions from any supported policy template. ( OCPBUGS-12879 ) Known issue When uninstalling the Security Profiles Operator, the MutatingWebhookConfiguration object is not deleted and must be manually removed. As a workaround, delete the MutatingWebhookConfiguration object after uninstalling the Security Profiles Operator. These steps are defined in Uninstalling the Security Profiles Operator . ( OCPBUGS-4687 ) 7.2.7. Security Profiles Operator 0.5.2 The following advisory is available for the Security Profiles Operator 0.5.2: RHBA-2023:0788 - OpenShift Security Profiles Operator bug fix update This update addresses a CVE in an underlying dependency. Known issue When uninstalling the Security Profiles Operator, the MutatingWebhookConfiguration object is not deleted and must be manually removed. As a workaround, delete the MutatingWebhookConfiguration object after uninstalling the Security Profiles Operator. These steps are defined in Uninstalling the Security Profiles Operator . ( OCPBUGS-4687 ) 7.2.8. Security Profiles Operator 0.5.0 The following advisory is available for the Security Profiles Operator 0.5.0: RHBA-2022:8762 - OpenShift Security Profiles Operator bug fix update Known issue When uninstalling the Security Profiles Operator, the MutatingWebhookConfiguration object is not deleted and must be manually removed. As a workaround, delete the MutatingWebhookConfiguration object after uninstalling the Security Profiles Operator. These steps are defined in Uninstalling the Security Profiles Operator . ( OCPBUGS-4687 ) 7.3. Security Profiles Operator support 7.3.1. Security Profiles Operator lifecycle The Security Profiles Operator is a "Rolling Stream" Operator, meaning updates are available asynchronously of OpenShift Container Platform releases. For more information, see OpenShift Operator Life Cycles on the Red Hat Customer Portal. 7.3.2. Getting support If you experience difficulty with a procedure described in this documentation, or with OpenShift Container Platform in general, visit the Red Hat Customer Portal . From the Customer Portal, you can: Search or browse through the Red Hat Knowledgebase of articles and solutions relating to Red Hat products. Submit a support case to Red Hat Support. Access other product documentation. To identify issues with your cluster, you can use Insights in OpenShift Cluster Manager . Insights provides details about issues and, if available, information on how to solve a problem. If you have a suggestion for improving this documentation or have found an error, submit a Jira issue for the most relevant documentation component. Please provide specific details, such as the section name and OpenShift Container Platform version. 7.4. Understanding the Security Profiles Operator OpenShift Container Platform administrators can use the Security Profiles Operator to define increased security measures in clusters. Important The Security Profiles Operator supports only Red Hat Enterprise Linux CoreOS (RHCOS) worker nodes. Red Hat Enterprise Linux (RHEL) nodes are not supported. 7.4.1. About Security Profiles Security profiles can increase security at the container level in your cluster. Seccomp security profiles list the syscalls a process can make. Permissions are broader than SELinux, enabling users to restrict operations system-wide, such as write . SELinux security profiles provide a label-based system that restricts the access and usage of processes, applications, or files in a system. All files in an environment have labels that define permissions. SELinux profiles can define access within a given structure, such as directories. 7.5. Enabling the Security Profiles Operator Before you can use the Security Profiles Operator, you must ensure the Operator is deployed in the cluster. Important The Security Profiles Operator supports only Red Hat Enterprise Linux CoreOS (RHCOS) worker nodes. Red Hat Enterprise Linux (RHEL) nodes are not supported. Important The Security Profiles Operator only supports x86_64 architecture. 7.5.1. Installing the Security Profiles Operator Prerequisites You must have admin privileges. Procedure In the OpenShift Container Platform web console, navigate to Operators OperatorHub . Search for the Security Profiles Operator, then click Install . Keep the default selection of Installation mode and namespace to ensure that the Operator will be installed to the openshift-security-profiles namespace. Click Install . Verification To confirm that the installation is successful: Navigate to the Operators Installed Operators page. Check that the Security Profiles Operator is installed in the openshift-security-profiles namespace and its status is Succeeded . If the Operator is not installed successfully: Navigate to the Operators Installed Operators page and inspect the Status column for any errors or failures. Navigate to the Workloads Pods page and check the logs in any pods in the openshift-security-profiles project that are reporting issues. 7.5.2. Installing the Security Profiles Operator using the CLI Prerequisites You must have admin privileges. Procedure Define a Namespace object: Example namespace-object.yaml apiVersion: v1 kind: Namespace metadata: name: openshift-security-profiles labels: openshift.io/cluster-monitoring: "true" Create the Namespace object: USD oc create -f namespace-object.yaml Define an OperatorGroup object: Example operator-group-object.yaml apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: security-profiles-operator namespace: openshift-security-profiles Create the OperatorGroup object: USD oc create -f operator-group-object.yaml Define a Subscription object: Example subscription-object.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: security-profiles-operator-sub namespace: openshift-security-profiles spec: channel: release-alpha-rhel-8 installPlanApproval: Automatic name: security-profiles-operator source: redhat-operators sourceNamespace: openshift-marketplace Create the Subscription object: USD oc create -f subscription-object.yaml Note If you are setting the global scheduler feature and enable defaultNodeSelector , you must create the namespace manually and update the annotations of the openshift-security-profiles namespace, or the namespace where the Security Profiles Operator was installed, with openshift.io/node-selector: "" . This removes the default node selector and prevents deployment failures. Verification Verify the installation succeeded by inspecting the following CSV file: USD oc get csv -n openshift-security-profiles Verify that the Security Profiles Operator is operational by running the following command: USD oc get deploy -n openshift-security-profiles 7.5.3. Configuring logging verbosity The Security Profiles Operator supports the default logging verbosity of 0 and an enhanced verbosity of 1 . Procedure To enable enhanced logging verbosity, patch the spod configuration and adjust the value by running the following command: USD oc -n openshift-security-profiles patch spod \ spod --type=merge -p '{"spec":{"verbosity":1}}' Example output securityprofilesoperatordaemon.security-profiles-operator.x-k8s.io/spod patched 7.6. Managing seccomp profiles Create and manage seccomp profiles and bind them to workloads. Important The Security Profiles Operator supports only Red Hat Enterprise Linux CoreOS (RHCOS) worker nodes. Red Hat Enterprise Linux (RHEL) nodes are not supported. 7.6.1. Creating seccomp profiles Use the SeccompProfile object to create profiles. SeccompProfile objects can restrict syscalls within a container, limiting the access of your application. Procedure Create a project by running the following command: USD oc new-project my-namespace Create the SeccompProfile object: apiVersion: security-profiles-operator.x-k8s.io/v1beta1 kind: SeccompProfile metadata: namespace: my-namespace name: profile1 spec: defaultAction: SCMP_ACT_LOG The seccomp profile will be saved in /var/lib/kubelet/seccomp/operator/<namespace>/<name>.json . An init container creates the root directory of the Security Profiles Operator to run the Operator without root group or user ID privileges. A symbolic link is created from the rootless profile storage /var/lib/openshift-security-profiles to the default seccomp root path inside of the kubelet root /var/lib/kubelet/seccomp/operator . 7.6.2. Applying seccomp profiles to a pod Create a pod to apply one of the created profiles. Procedure Create a pod object that defines a securityContext : apiVersion: v1 kind: Pod metadata: name: test-pod spec: securityContext: runAsNonRoot: true seccompProfile: type: Localhost localhostProfile: operator/my-namespace/profile1.json containers: - name: test-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] View the profile path of the seccompProfile.localhostProfile attribute by running the following command: USD oc -n my-namespace get seccompprofile profile1 --output wide Example output NAME STATUS AGE SECCOMPPROFILE.LOCALHOSTPROFILE profile1 Installed 14s operator/my-namespace/profile1.json View the path to the localhost profile by running the following command: USD oc get sp profile1 --output=jsonpath='{.status.localhostProfile}' Example output operator/my-namespace/profile1.json Apply the localhostProfile output to the patch file: spec: template: spec: securityContext: seccompProfile: type: Localhost localhostProfile: operator/my-namespace/profile1.json Apply the profile to any other workload, such as a Deployment object, by running the following command: USD oc -n my-namespace patch deployment myapp --patch-file patch.yaml --type=merge Example output deployment.apps/myapp patched Verification Confirm the profile was applied correctly by running the following command: USD oc -n my-namespace get deployment myapp --output=jsonpath='{.spec.template.spec.securityContext}' | jq . Example output { "seccompProfile": { "localhostProfile": "operator/my-namespace/profile1.json", "type": "localhost" } } 7.6.2.1. Binding workloads to profiles with ProfileBindings You can use the ProfileBinding resource to bind a security profile to the SecurityContext of a container. Procedure To bind a pod that uses a quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 image to the example SeccompProfile profile, create a ProfileBinding object in the same namespace with the pod and the SeccompProfile objects: apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileBinding metadata: namespace: my-namespace name: nginx-binding spec: profileRef: kind: SeccompProfile 1 name: profile 2 image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 3 1 The kind: variable refers to the kind of the profile. 2 The name: variable refers to the name of the profile. 3 You can enable a default security profile by using a wildcard in the image attribute: image: "*" Important Using the image: "*" wildcard attribute binds all new pods with a default security profile in a given namespace. Label the namespace with enable-binding=true by running the following command: USD oc label ns my-namespace spo.x-k8s.io/enable-binding=true Define a pod named test-pod.yaml : apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 Create the pod: USD oc create -f test-pod.yaml Note If the pod already exists, you must re-create the pod for the binding to work properly. Verification Confirm the pod inherits the ProfileBinding by running the following command: USD oc get pod test-pod -o jsonpath='{.spec.containers[*].securityContext.seccompProfile}' Example output {"localhostProfile":"operator/my-namespace/profile.json","type":"Localhost"} 7.6.3. Recording profiles from workloads The Security Profiles Operator can record system calls with ProfileRecording objects, making it easier to create baseline profiles for applications. When using the log enricher for recording seccomp profiles, verify the log enricher feature is enabled. See Additional resources for more information. Note A container with privileged: true security context restraints prevents log-based recording. Privileged containers are not subject to seccomp policies, and log-based recording makes use of a special seccomp profile to record events. Procedure Create a project by running the following command: USD oc new-project my-namespace Label the namespace with enable-recording=true by running the following command: USD oc label ns my-namespace spo.x-k8s.io/enable-recording=true Create a ProfileRecording object containing a recorder: logs variable: apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: namespace: my-namespace name: test-recording spec: kind: SeccompProfile recorder: logs podSelector: matchLabels: app: my-app Create a workload to record: apiVersion: v1 kind: Pod metadata: namespace: my-namespace name: my-pod labels: app: my-app spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: nginx image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] - name: redis image: quay.io/security-profiles-operator/redis:6.2.1 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] Confirm the pod is in a Running state by entering the following command: USD oc -n my-namespace get pods Example output NAME READY STATUS RESTARTS AGE my-pod 2/2 Running 0 18s Confirm the enricher indicates that it receives audit logs for those containers: USD oc -n openshift-security-profiles logs --since=1m --selector name=spod -c log-enricher Example output I0523 14:19:08.747313 430694 enricher.go:445] log-enricher "msg"="audit" "container"="redis" "executable"="/usr/local/bin/redis-server" "namespace"="my-namespace" "node"="xiyuan-23-5g2q9-worker-eastus2-6rpgf" "pid"=656802 "pod"="my-pod" "syscallID"=0 "syscallName"="read" "timestamp"="1684851548.745:207179" "type"="seccomp" Verification Remove the pod: USD oc -n my-namepace delete pod my-pod Confirm the Security Profiles Operator reconciles the two seccomp profiles: USD oc get seccompprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace Example output for seccompprofile NAME STATUS AGE test-recording-nginx Installed 2m48s test-recording-redis Installed 2m48s 7.6.3.1. Merging per-container profile instances By default, each container instance records into a separate profile. The Security Profiles Operator can merge the per-container profiles into a single profile. Merging profiles is useful when deploying applications using ReplicaSet or Deployment objects. Procedure Edit a ProfileRecording object to include a mergeStrategy: containers variable: apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: # The name of the Recording is the same as the resulting SeccompProfile CRD # after reconciliation. name: test-recording namespace: my-namespace spec: kind: SeccompProfile recorder: logs mergeStrategy: containers podSelector: matchLabels: app: sp-record Label the namespace by running the following command: USD oc label ns my-namespace security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite=true Create the workload with the following YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deploy namespace: my-namespace spec: replicas: 3 selector: matchLabels: app: sp-record template: metadata: labels: app: sp-record spec: serviceAccountName: spo-record-sa containers: - name: nginx-record image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 To record the individual profiles, delete the deployment by running the following command: USD oc delete deployment nginx-deploy -n my-namespace To merge the profiles, delete the profile recording by running the following command: USD oc delete profilerecording test-recording -n my-namespace To start the merge operation and generate the results profile, run the following command: USD oc get seccompprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace Example output for seccompprofiles NAME STATUS AGE test-recording-nginx-record Installed 55s To view the permissions used by any of the containers, run the following command: USD oc get seccompprofiles test-recording-nginx-record -o yaml Additional resources Managing security context constraints Managing SCCs in OpenShift Using the log enricher About security profiles 7.7. Managing SELinux profiles Create and manage SELinux profiles and bind them to workloads. Important The Security Profiles Operator supports only Red Hat Enterprise Linux CoreOS (RHCOS) worker nodes. Red Hat Enterprise Linux (RHEL) nodes are not supported. 7.7.1. Creating SELinux profiles Use the SelinuxProfile object to create profiles. The SelinuxProfile object has several features that allow for better security hardening and readability: Restricts the profiles to inherit from to the current namespace or a system-wide profile. Because there are typically many profiles installed on the system, but only a subset should be used by cluster workloads, the inheritable system profiles are listed in the spod instance in spec.selinuxOptions.allowedSystemProfiles . Performs basic validation of the permissions, classes and labels. Adds a new keyword @self that describes the process using the policy. This allows reusing a policy between workloads and namespaces easily, as the usage of the policy is based on the name and namespace. Adds features for better security hardening and readability compared to writing a profile directly in the SELinux CIL language. Procedure Create a project by running the following command: USD oc new-project nginx-deploy Create a policy that can be used with a non-privileged workload by creating the following SelinuxProfile object: apiVersion: security-profiles-operator.x-k8s.io/v1alpha2 kind: SelinuxProfile metadata: name: nginx-secure namespace: nginx-deploy spec: allow: '@self': tcp_socket: - listen http_cache_port_t: tcp_socket: - name_bind node_t: tcp_socket: - node_bind inherit: - kind: System name: container Wait for selinuxd to install the policy by running the following command: USD oc wait --for=condition=ready -n nginx-deploy selinuxprofile nginx-secure Example output selinuxprofile.security-profiles-operator.x-k8s.io/nginx-secure condition met The policies are placed into an emptyDir in the container owned by the Security Profiles Operator. The policies are saved in Common Intermediate Language (CIL) format in /etc/selinux.d/<name>_<namespace>.cil . Access the pod by running the following command: USD oc -n openshift-security-profiles rsh -c selinuxd ds/spod Verification View the file contents with cat by running the following command: USD cat /etc/selinux.d/nginx-secure_nginx-deploy.cil Example output (block nginx-secure_nginx-deploy (blockinherit container) (allow process nginx-secure_nginx-deploy.process ( tcp_socket ( listen ))) (allow process http_cache_port_t ( tcp_socket ( name_bind ))) (allow process node_t ( tcp_socket ( node_bind ))) ) Verify that a policy has been installed by running the following command: USD semodule -l | grep nginx-secure Example output nginx-secure_nginx-deploy 7.7.2. Applying SELinux profiles to a pod Create a pod to apply one of the created profiles. For SELinux profiles, the namespace must be labelled to allow privileged workloads. Procedure Apply the scc.podSecurityLabelSync=false label to the nginx-deploy namespace by running the following command: USD oc label ns nginx-deploy security.openshift.io/scc.podSecurityLabelSync=false Apply the privileged label to the nginx-deploy namespace by running the following command: USD oc label ns nginx-deploy --overwrite=true pod-security.kubernetes.io/enforce=privileged Obtain the SELinux profile usage string by running the following command: USD oc get selinuxprofile.security-profiles-operator.x-k8s.io/nginx-secure -n nginx-deploy -ojsonpath='{.status.usage}' Example output nginx-secure_nginx-deploy.process Apply the output string in the workload manifest in the .spec.containers[].securityContext.seLinuxOptions attribute: apiVersion: v1 kind: Pod metadata: name: nginx-secure namespace: nginx-deploy spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - image: nginxinc/nginx-unprivileged:1.21 name: nginx securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] seLinuxOptions: # NOTE: This uses an appropriate SELinux type type: nginx-secure_nginx-deploy.process Important The SELinux type must exist before creating the workload. 7.7.2.1. Applying SELinux log policies To log policy violations or AVC denials, set the SElinuxProfile profile to permissive . Important This procedure defines logging policies. It does not set enforcement policies. Procedure Add permissive: true to an SElinuxProfile : apiVersion: security-profiles-operator.x-k8s.io/v1alpha2 kind: SelinuxProfile metadata: name: nginx-secure namespace: nginx-deploy spec: permissive: true 7.7.2.2. Binding workloads to profiles with ProfileBindings You can use the ProfileBinding resource to bind a security profile to the SecurityContext of a container. Procedure To bind a pod that uses a quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 image to the example SelinuxProfile profile, create a ProfileBinding object in the same namespace with the pod and the SelinuxProfile objects: apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileBinding metadata: namespace: my-namespace name: nginx-binding spec: profileRef: kind: SelinuxProfile 1 name: profile 2 image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 3 1 The kind: variable refers to the kind of the profile. 2 The name: variable refers to the name of the profile. 3 You can enable a default security profile by using a wildcard in the image attribute: image: "*" Important Using the image: "*" wildcard attribute binds all new pods with a default security profile in a given namespace. Label the namespace with enable-binding=true by running the following command: USD oc label ns my-namespace spo.x-k8s.io/enable-binding=true Define a pod named test-pod.yaml : apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 Create the pod: USD oc create -f test-pod.yaml Note If the pod already exists, you must re-create the pod for the binding to work properly. Verification Confirm the pod inherits the ProfileBinding by running the following command: USD oc get pod test-pod -o jsonpath='{.spec.containers[*].securityContext.seLinuxOptions.type}' Example output profile_nginx-binding.process 7.7.2.3. Replicating controllers and SecurityContextConstraints When you deploy SELinux policies for replicating controllers, such as deployments or daemon sets, note that the Pod objects spawned by the controllers are not running with the identity of the user who creates the workload. Unless a ServiceAccount is selected, the pods might revert to using a restricted SecurityContextConstraints (SCC) which does not allow use of custom security policies. Procedure Create a project by running the following command: USD oc new-project nginx-secure Create the following RoleBinding object to allow SELinux policies to be used in the nginx-secure namespace: kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: spo-nginx namespace: nginx-secure subjects: - kind: ServiceAccount name: spo-deploy-test roleRef: kind: Role name: spo-nginx apiGroup: rbac.authorization.k8s.io Create the Role object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: spo-nginx namespace: nginx-secure rules: - apiGroups: - security.openshift.io resources: - securitycontextconstraints resourceNames: - privileged verbs: - use Create the ServiceAccount object: apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: spo-deploy-test namespace: nginx-secure Create the Deployment object: apiVersion: apps/v1 kind: Deployment metadata: name: selinux-test namespace: nginx-secure metadata: labels: app: selinux-test spec: replicas: 3 selector: matchLabels: app: selinux-test template: metadata: labels: app: selinux-test spec: serviceAccountName: spo-deploy-test securityContext: seLinuxOptions: type: nginx-secure_nginx-secure.process 1 containers: - name: nginx-unpriv image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 1 The .seLinuxOptions.type must exist before the Deployment is created. Note The SELinux type is not specified in the workload and is handled by the SCC. When the pods are created by the deployment and the ReplicaSet , the pods will run with the appropriate profile. Ensure that your SCC is usable by only the correct service account. Refer to Additional resources for more information. 7.7.3. Recording profiles from workloads The Security Profiles Operator can record system calls with ProfileRecording objects, making it easier to create baseline profiles for applications. When using the log enricher for recording SELinux profiles, verify the log enricher feature is enabled. See Additional resources for more information. Note A container with privileged: true security context restraints prevents log-based recording. Privileged containers are not subject to SELinux policies, and log-based recording makes use of a special SELinux profile to record events. Procedure Create a project by running the following command: USD oc new-project my-namespace Label the namespace with enable-recording=true by running the following command: USD oc label ns my-namespace spo.x-k8s.io/enable-recording=true Create a ProfileRecording object containing a recorder: logs variable: apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: namespace: my-namespace name: test-recording spec: kind: SelinuxProfile recorder: logs podSelector: matchLabels: app: my-app Create a workload to record: apiVersion: v1 kind: Pod metadata: namespace: my-namespace name: my-pod labels: app: my-app spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: nginx image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] - name: redis image: quay.io/security-profiles-operator/redis:6.2.1 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] Confirm the pod is in a Running state by entering the following command: USD oc -n my-namespace get pods Example output NAME READY STATUS RESTARTS AGE my-pod 2/2 Running 0 18s Confirm the enricher indicates that it receives audit logs for those containers: USD oc -n openshift-security-profiles logs --since=1m --selector name=spod -c log-enricher Example output I0517 13:55:36.383187 348295 enricher.go:376] log-enricher "msg"="audit" "container"="redis" "namespace"="my-namespace" "node"="ip-10-0-189-53.us-east-2.compute.internal" "perm"="name_bind" "pod"="my-pod" "profile"="test-recording_redis_6kmrb_1684331729" "scontext"="system_u:system_r:selinuxrecording.process:s0:c4,c27" "tclass"="tcp_socket" "tcontext"="system_u:object_r:redis_port_t:s0" "timestamp"="1684331735.105:273965" "type"="selinux" Verification Remove the pod: USD oc -n my-namepace delete pod my-pod Confirm the Security Profiles Operator reconciles the two SELinux profiles: USD oc get selinuxprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace Example output for selinuxprofile NAME USAGE STATE test-recording-nginx test-recording-nginx_my-namespace.process Installed test-recording-redis test-recording-redis_my-namespace.process Installed 7.7.3.1. Merging per-container profile instances By default, each container instance records into a separate profile. The Security Profiles Operator can merge the per-container profiles into a single profile. Merging profiles is useful when deploying applications using ReplicaSet or Deployment objects. Procedure Edit a ProfileRecording object to include a mergeStrategy: containers variable: apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: # The name of the Recording is the same as the resulting SelinuxProfile CRD # after reconciliation. name: test-recording namespace: my-namespace spec: kind: SelinuxProfile recorder: logs mergeStrategy: containers podSelector: matchLabels: app: sp-record Label the namespace by running the following command: USD oc label ns my-namespace security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite=true Create the workload with the following YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deploy namespace: my-namespace spec: replicas: 3 selector: matchLabels: app: sp-record template: metadata: labels: app: sp-record spec: serviceAccountName: spo-record-sa containers: - name: nginx-record image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 To record the individual profiles, delete the deployment by running the following command: USD oc delete deployment nginx-deploy -n my-namespace To merge the profiles, delete the profile recording by running the following command: USD oc delete profilerecording test-recording -n my-namespace To start the merge operation and generate the results profile, run the following command: USD oc get selinuxprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace Example output for selinuxprofiles NAME USAGE STATE test-recording-nginx-record test-recording-nginx-record_my-namespace.process Installed To view the permissions used by any of the containers, run the following command: USD oc get selinuxprofiles test-recording-nginx-record -o yaml 7.7.3.2. About seLinuxContext: RunAsAny Recording of SELinux policies is implemented with a webhook that injects a special SELinux type to the pods being recorded. The SELinux type makes the pod run in permissive mode, logging all the AVC denials into audit.log . By default, a workload is not allowed to run with a custom SELinux policy, but uses an auto-generated type. To record a workload, the workload must use a service account that has permissions to use an SCC that allows the webhook to inject the permissive SELinux type. The privileged SCC contains seLinuxContext: RunAsAny . In addition, the namespace must be labeled with pod-security.kubernetes.io/enforce: privileged if your cluster enables the Pod Security Admission because only the privileged Pod Security Standard allows using a custom SELinux policy. Additional resources Managing security context constraints Managing SCCs in OpenShift Using the log enricher About security profiles 7.8. Advanced Security Profiles Operator tasks Use advanced tasks to enable metrics, configure webhooks, or restrict syscalls. 7.8.1. Restrict the allowed syscalls in seccomp profiles The Security Profiles Operator does not restrict syscalls in seccomp profiles by default. You can define the list of allowed syscalls in the spod configuration. Procedure To define the list of allowedSyscalls , adjust the spec parameter by running the following command: USD oc -n openshift-security-profiles patch spod spod --type merge \ -p '{"spec":{"allowedSyscalls": ["exit", "exit_group", "futex", "nanosleep"]}}' Important The Operator will install only the seccomp profiles, which have a subset of syscalls defined into the allowed list. All profiles not complying with this ruleset are rejected. When the list of allowed syscalls is modified in the spod configuration, the Operator will identify the already installed profiles which are non-compliant and remove them automatically. 7.8.2. Base syscalls for a container runtime You can use the baseProfileName attribute to establish the minimum required syscalls for a given runtime to start a container. Procedure Edit the SeccompProfile kind object and add baseProfileName: runc-v1.0.0 to the spec field: apiVersion: security-profiles-operator.x-k8s.io/v1beta1 kind: SeccompProfile metadata: namespace: my-namespace name: example-name spec: defaultAction: SCMP_ACT_ERRNO baseProfileName: runc-v1.0.0 syscalls: - action: SCMP_ACT_ALLOW names: - exit_group 7.8.3. Enabling memory optimization in the spod daemon The controller running inside of spod daemon process watches all pods available in the cluster when profile recording is enabled. This can lead to very high memory usage in large clusters, resulting in the spod daemon running out of memory or crashing. To prevent crashes, the spod daemon can be configured to only load the pods labeled for profile recording into the cache memory. Note SPO memory optimization is not enabled by default. Procedure Enable memory optimization by running the following command: USD oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"enableMemoryOptimization":true}}' To record a security profile for a pod, the pod must be labeled with spo.x-k8s.io/enable-recording: "true" : apiVersion: v1 kind: Pod metadata: name: my-recording-pod labels: spo.x-k8s.io/enable-recording: "true" # ... 7.8.4. Customizing daemon resource requirements The default resource requirements of the daemon container can be adjusted by using the field daemonResourceRequirements from the spod configuration. Procedure To specify the memory and cpu requests and limits of the daemon container, run the following command: USD oc -n openshift-security-profiles patch spod spod --type merge -p \ '{"spec":{"daemonResourceRequirements": { \ "requests": {"memory": "256Mi", "cpu": "250m"}, \ "limits": {"memory": "512Mi", "cpu": "500m"}}}}' 7.8.5. Setting a custom priority class name for the spod daemon pod The default priority class name of the spod daemon pod is set to system-node-critical . A custom priority class name can be configured in the spod configuration by setting a value in the priorityClassName field. Procedure Configure the priority class name by running the following command: USD oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"priorityClassName":"my-priority-class"}}' Example output securityprofilesoperatordaemon.openshift-security-profiles.x-k8s.io/spod patched 7.8.6. Using metrics The openshift-security-profiles namespace provides metrics endpoints, which are secured by the kube-rbac-proxy container. All metrics are exposed by the metrics service within the openshift-security-profiles namespace. The Security Profiles Operator includes a cluster role and corresponding binding spo-metrics-client to retrieve the metrics from within the cluster. There are two metrics paths available: metrics.openshift-security-profiles/metrics : for controller runtime metrics metrics.openshift-security-profiles/metrics-spod : for the Operator daemon metrics Procedure To view the status of the metrics service, run the following command: USD oc get svc/metrics -n openshift-security-profiles Example output NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE metrics ClusterIP 10.0.0.228 <none> 443/TCP 43s To retrieve the metrics, query the service endpoint using the default ServiceAccount token in the openshift-security-profiles namespace by running the following command: USD oc run --rm -i --restart=Never --image=registry.fedoraproject.org/fedora-minimal:latest \ -n openshift-security-profiles metrics-test -- bash -c \ 'curl -ks -H "Authorization: Bearer USD(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-security-profiles/metrics-spod' Example output # HELP security_profiles_operator_seccomp_profile_total Counter about seccomp profile operations. # TYPE security_profiles_operator_seccomp_profile_total counter security_profiles_operator_seccomp_profile_total{operation="delete"} 1 security_profiles_operator_seccomp_profile_total{operation="update"} 2 To retrieve metrics from a different namespace, link the ServiceAccount to the spo-metrics-client ClusterRoleBinding by running the following command: USD oc get clusterrolebinding spo-metrics-client -o wide Example output NAME ROLE AGE USERS GROUPS SERVICEACCOUNTS spo-metrics-client ClusterRole/spo-metrics-client 35m openshift-security-profiles/default 7.8.6.1. controller-runtime metrics The controller-runtime metrics and the DaemonSet endpoint metrics-spod provide a set of default metrics. Additional metrics are provided by the daemon, which are always prefixed with security_profiles_operator_ . Table 7.1. Available controller-runtime metrics Metric key Possible labels Type Purpose seccomp_profile_total operation={delete,update} Counter Amount of seccomp profile operations. seccomp_profile_audit_total node , namespace , pod , container , executable , syscall Counter Amount of seccomp profile audit operations. Requires the log enricher to be enabled. seccomp_profile_bpf_total node , mount_namespace , profile Counter Amount of seccomp profile bpf operations. Requires the bpf recorder to be enabled. seccomp_profile_error_total reason={ SeccompNotSupportedOnNode, InvalidSeccompProfile, CannotSaveSeccompProfile, CannotRemoveSeccompProfile, CannotUpdateSeccompProfile, CannotUpdateNodeStatus } Counter Amount of seccomp profile errors. selinux_profile_total operation={delete,update} Counter Amount of SELinux profile operations. selinux_profile_audit_total node , namespace , pod , container , executable , scontext , tcontext Counter Amount of SELinux profile audit operations. Requires the log enricher to be enabled. selinux_profile_error_total reason={ CannotSaveSelinuxPolicy, CannotUpdatePolicyStatus, CannotRemoveSelinuxPolicy, CannotContactSelinuxd, CannotWritePolicyFile, CannotGetPolicyStatus } Counter Amount of SELinux profile errors. 7.8.7. Using the log enricher The Security Profiles Operator contains a log enrichment feature, which is disabled by default. The log enricher container runs with privileged permissions to read the audit logs from the local node. The log enricher runs within the host PID namespace, hostPID . Important The log enricher must have permissions to read the host processes. Procedure Patch the spod configuration to enable the log enricher by running the following command: USD oc -n openshift-security-profiles patch spod spod \ --type=merge -p '{"spec":{"enableLogEnricher":true}}' Example output securityprofilesoperatordaemon.security-profiles-operator.x-k8s.io/spod patched Note The Security Profiles Operator will re-deploy the spod daemon set automatically. View the audit logs by running the following command: USD oc -n openshift-security-profiles logs -f ds/spod log-enricher Example output I0623 12:51:04.257814 1854764 deleg.go:130] setup "msg"="starting component: log-enricher" "buildDate"="1980-01-01T00:00:00Z" "compiler"="gc" "gitCommit"="unknown" "gitTreeState"="clean" "goVersion"="go1.16.2" "platform"="linux/amd64" "version"="0.4.0-dev" I0623 12:51:04.257890 1854764 enricher.go:44] log-enricher "msg"="Starting log-enricher on node: 127.0.0.1" I0623 12:51:04.257898 1854764 enricher.go:46] log-enricher "msg"="Connecting to local GRPC server" I0623 12:51:04.258061 1854764 enricher.go:69] log-enricher "msg"="Reading from file /var/log/audit/audit.log" 2021/06/23 12:51:04 Seeked /var/log/audit/audit.log - &{Offset:0 Whence:2} 7.8.7.1. Using the log enricher to trace an application You can use the Security Profiles Operator log enricher to trace an application. Procedure To trace an application, create a SeccompProfile logging profile: apiVersion: security-profiles-operator.x-k8s.io/v1beta1 kind: SeccompProfile metadata: name: log namespace: default spec: defaultAction: SCMP_ACT_LOG Create a pod object to use the profile: apiVersion: v1 kind: Pod metadata: name: log-pod spec: securityContext: runAsNonRoot: true seccompProfile: type: Localhost localhostProfile: operator/default/log.json containers: - name: log-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] Examine the log enricher output by running the following command: USD oc -n openshift-security-profiles logs -f ds/spod log-enricher Example 7.1. Example output ... I0623 12:59:11.479869 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=3 "syscallName"="close" "timestamp"="1624453150.205:1061" "type"="seccomp" I0623 12:59:11.487323 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=157 "syscallName"="prctl" "timestamp"="1624453150.205:1062" "type"="seccomp" I0623 12:59:11.492157 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=157 "syscallName"="prctl" "timestamp"="1624453150.205:1063" "type"="seccomp" ... I0623 12:59:20.258523 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=12 "syscallName"="brk" "timestamp"="1624453150.235:2873" "type"="seccomp" I0623 12:59:20.263349 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=21 "syscallName"="access" "timestamp"="1624453150.235:2874" "type"="seccomp" I0623 12:59:20.354091 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=257 "syscallName"="openat" "timestamp"="1624453150.235:2875" "type"="seccomp" I0623 12:59:20.358844 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=5 "syscallName"="fstat" "timestamp"="1624453150.235:2876" "type"="seccomp" I0623 12:59:20.363510 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=9 "syscallName"="mmap" "timestamp"="1624453150.235:2877" "type"="seccomp" I0623 12:59:20.454127 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=3 "syscallName"="close" "timestamp"="1624453150.235:2878" "type"="seccomp" I0623 12:59:20.458654 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=257 "syscallName"="openat" "timestamp"="1624453150.235:2879" "type"="seccomp" ... 7.8.8. Configuring webhooks Profile binding and profile recording objects can use webhooks. Profile binding and recording object configurations are MutatingWebhookConfiguration CRs, managed by the Security Profiles Operator. To change the webhook configuration, the spod CR exposes a webhookOptions field that allows modification of the failurePolicy , namespaceSelector , and objectSelector variables. This allows you to set the webhooks to "soft-fail" or restrict them to a subset of a namespaces so that even if the webhooks failed, other namespaces or resources are not affected. Procedure Set the recording.spo.io webhook configuration to record only pods labeled with spo-record=true by creating the following patch file: spec: webhookOptions: - name: recording.spo.io objectSelector: matchExpressions: - key: spo-record operator: In values: - "true" Patch the spod/spod instance by running the following command: USD oc -n openshift-security-profiles patch spod \ spod -p USD(cat /tmp/spod-wh.patch) --type=merge To view the resulting MutatingWebhookConfiguration object, run the following command: USD oc get MutatingWebhookConfiguration \ spo-mutating-webhook-configuration -oyaml 7.9. Troubleshooting the Security Profiles Operator Troubleshoot the Security Profiles Operator to diagnose a problem or provide information in a bug report. 7.9.1. Inspecting seccomp profiles Corrupted seccomp profiles can disrupt your workloads. Ensure that the user cannot abuse the system by not allowing other workloads to map any part of the path /var/lib/kubelet/seccomp/operator . Procedure Confirm that the profile is reconciled by running the following command: USD oc -n openshift-security-profiles logs openshift-security-profiles-<id> Example 7.2. Example output I1019 19:34:14.942464 1 main.go:90] setup "msg"="starting openshift-security-profiles" "buildDate"="2020-10-19T19:31:24Z" "compiler"="gc" "gitCommit"="a3ef0e1ea6405092268c18f240b62015c247dd9d" "gitTreeState"="dirty" "goVersion"="go1.15.1" "platform"="linux/amd64" "version"="0.2.0-dev" I1019 19:34:15.348389 1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen" "addr"=":8080" I1019 19:34:15.349076 1 main.go:126] setup "msg"="starting manager" I1019 19:34:15.349449 1 internal.go:391] controller-runtime/manager "msg"="starting metrics server" "path"="/metrics" I1019 19:34:15.350201 1 controller.go:142] controller "msg"="Starting EventSource" "controller"="profile" "reconcilerGroup"="security-profiles-operator.x-k8s.io" "reconcilerKind"="SeccompProfile" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"defaultAction":""}}} I1019 19:34:15.450674 1 controller.go:149] controller "msg"="Starting Controller" "controller"="profile" "reconcilerGroup"="security-profiles-operator.x-k8s.io" "reconcilerKind"="SeccompProfile" I1019 19:34:15.450757 1 controller.go:176] controller "msg"="Starting workers" "controller"="profile" "reconcilerGroup"="security-profiles-operator.x-k8s.io" "reconcilerKind"="SeccompProfile" "worker count"=1 I1019 19:34:15.453102 1 profile.go:148] profile "msg"="Reconciled profile from SeccompProfile" "namespace"="openshift-security-profiles" "profile"="nginx-1.19.1" "name"="nginx-1.19.1" "resource version"="728" I1019 19:34:15.453618 1 profile.go:148] profile "msg"="Reconciled profile from SeccompProfile" "namespace"="openshift-security-profiles" "profile"="openshift-security-profiles" "name"="openshift-security-profiles" "resource version"="729" Confirm that the seccomp profiles are saved into the correct path by running the following command: USD oc exec -t -n openshift-security-profiles openshift-security-profiles-<id> \ -- ls /var/lib/kubelet/seccomp/operator/my-namespace/my-workload Example output profile-block.json profile-complain.json 7.10. Uninstalling the Security Profiles Operator You can remove the Security Profiles Operator from your cluster by using the OpenShift Container Platform web console. 7.10.1. Uninstall the Security Profiles Operator using the web console To remove the Security Profiles Operator, you must first delete the seccomp and SELinux profiles. After the profiles are removed, you can then remove the Operator and its namespace by deleting the openshift-security-profiles project. Prerequisites Access to an OpenShift Container Platform cluster that uses an account with cluster-admin permissions. The Security Profiles Operator is installed. Procedure To remove the Security Profiles Operator by using the OpenShift Container Platform web console: Navigate to the Operators Installed Operators page. Delete all seccomp profiles, SELinux profiles, and webhook configurations. Switch to the Administration Operators Installed Operators page. Click the Options menu on the Security Profiles Operator entry and select Uninstall Operator . Switch to the Home Projects page. Search for security profiles . Click the Options menu to the openshift-security-profiles project, and select Delete Project . Confirm the deletion by typing openshift-security-profiles in the dialog box, and click Delete . Delete the MutatingWebhookConfiguration object by running the following command: USD oc delete MutatingWebhookConfiguration spo-mutating-webhook-configuration

Commands:
apiVersion: v1 kind: Namespace metadata: name: openshift-security-profiles labels: openshift.io/cluster-monitoring: "true"
oc create -f namespace-object.yaml
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: security-profiles-operator namespace: openshift-security-profiles
oc create -f operator-group-object.yaml
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: security-profiles-operator-sub namespace: openshift-security-profiles spec: channel: release-alpha-rhel-8 installPlanApproval: Automatic name: security-profiles-operator source: redhat-operators sourceNamespace: openshift-marketplace
oc create -f subscription-object.yaml
oc get csv -n openshift-security-profiles
oc get deploy -n openshift-security-profiles
oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"verbosity":1}}'
securityprofilesoperatordaemon.security-profiles-operator.x-k8s.io/spod patched
oc new-project my-namespace
apiVersion: security-profiles-operator.x-k8s.io/v1beta1 kind: SeccompProfile metadata: namespace: my-namespace name: profile1 spec: defaultAction: SCMP_ACT_LOG
apiVersion: v1 kind: Pod metadata: name: test-pod spec: securityContext: runAsNonRoot: true seccompProfile: type: Localhost localhostProfile: operator/my-namespace/profile1.json containers: - name: test-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL]
oc -n my-namespace get seccompprofile profile1 --output wide
NAME STATUS AGE SECCOMPPROFILE.LOCALHOSTPROFILE profile1 Installed 14s operator/my-namespace/profile1.json
oc get sp profile1 --output=jsonpath='{.status.localhostProfile}'
operator/my-namespace/profile1.json
spec: template: spec: securityContext: seccompProfile: type: Localhost localhostProfile: operator/my-namespace/profile1.json
oc -n my-namespace patch deployment myapp --patch-file patch.yaml --type=merge
deployment.apps/myapp patched
oc -n my-namespace get deployment myapp --output=jsonpath='{.spec.template.spec.securityContext}' | jq .
{ "seccompProfile": { "localhostProfile": "operator/my-namespace/profile1.json", "type": "localhost" } }
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileBinding metadata: namespace: my-namespace name: nginx-binding spec: profileRef: kind: SeccompProfile 1 name: profile 2 image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 3
oc label ns my-namespace spo.x-k8s.io/enable-binding=true
apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21
oc create -f test-pod.yaml
oc get pod test-pod -o jsonpath='{.spec.containers[*].securityContext.seccompProfile}'
localhostProfile: operator/my-namespace/profile.json
type: Localhost
oc new-project my-namespace
oc label ns my-namespace spo.x-k8s.io/enable-recording=true
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: namespace: my-namespace name: test-recording spec: kind: SeccompProfile recorder: logs podSelector: matchLabels: app: my-app
apiVersion: v1 kind: Pod metadata: namespace: my-namespace name: my-pod labels: app: my-app spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: nginx image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] - name: redis image: quay.io/security-profiles-operator/redis:6.2.1 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL]
oc -n my-namespace get pods
NAME READY STATUS RESTARTS AGE my-pod 2/2 Running 0 18s
oc -n openshift-security-profiles logs --since=1m --selector name=spod -c log-enricher
I0523 14:19:08.747313 430694 enricher.go:445] log-enricher "msg"="audit" "container"="redis" "executable"="/usr/local/bin/redis-server" "namespace"="my-namespace" "node"="xiyuan-23-5g2q9-worker-eastus2-6rpgf" "pid"=656802 "pod"="my-pod" "syscallID"=0 "syscallName"="read" "timestamp"="1684851548.745:207179" "type"="seccomp"
oc -n my-namepace delete pod my-pod
oc get seccompprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace
NAME STATUS AGE test-recording-nginx Installed 2m48s test-recording-redis Installed 2m48s
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: # The name of the Recording is the same as the resulting SeccompProfile CRD # after reconciliation. name: test-recording namespace: my-namespace spec: kind: SeccompProfile recorder: logs mergeStrategy: containers podSelector: matchLabels: app: sp-record
oc label ns my-namespace security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite=true
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deploy namespace: my-namespace spec: replicas: 3 selector: matchLabels: app: sp-record template: metadata: labels: app: sp-record spec: serviceAccountName: spo-record-sa containers: - name: nginx-record image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080
oc delete deployment nginx-deploy -n my-namespace
oc delete profilerecording test-recording -n my-namespace
oc get seccompprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace
NAME STATUS AGE test-recording-nginx-record Installed 55s
oc get seccompprofiles test-recording-nginx-record -o yaml
oc new-project nginx-deploy
apiVersion: security-profiles-operator.x-k8s.io/v1alpha2 kind: SelinuxProfile metadata: name: nginx-secure namespace: nginx-deploy spec: allow: '@self': tcp_socket: - listen http_cache_port_t: tcp_socket: - name_bind node_t: tcp_socket: - node_bind inherit: - kind: System name: container
oc wait --for=condition=ready -n nginx-deploy selinuxprofile nginx-secure
selinuxprofile.security-profiles-operator.x-k8s.io/nginx-secure condition met
oc -n openshift-security-profiles rsh -c selinuxd ds/spod
cat /etc/selinux.d/nginx-secure_nginx-deploy.cil
(block nginx-secure_nginx-deploy (blockinherit container) (allow process nginx-secure_nginx-deploy.process ( tcp_socket ( listen ))) (allow process http_cache_port_t ( tcp_socket ( name_bind ))) (allow process node_t ( tcp_socket ( node_bind ))) )
semodule -l | grep nginx-secure
nginx-secure_nginx-deploy
oc label ns nginx-deploy security.openshift.io/scc.podSecurityLabelSync=false
oc label ns nginx-deploy --overwrite=true pod-security.kubernetes.io/enforce=privileged
oc get selinuxprofile.security-profiles-operator.x-k8s.io/nginx-secure -n nginx-deploy -ojsonpath='{.status.usage}'
nginx-secure_nginx-deploy.process
apiVersion: v1 kind: Pod metadata: name: nginx-secure namespace: nginx-deploy spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - image: nginxinc/nginx-unprivileged:1.21 name: nginx securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] seLinuxOptions: # NOTE: This uses an appropriate SELinux type type: nginx-secure_nginx-deploy.process
apiVersion: security-profiles-operator.x-k8s.io/v1alpha2 kind: SelinuxProfile metadata: name: nginx-secure namespace: nginx-deploy spec: permissive: true
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileBinding metadata: namespace: my-namespace name: nginx-binding spec: profileRef: kind: SelinuxProfile 1 name: profile 2 image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 3
oc label ns my-namespace spo.x-k8s.io/enable-binding=true
apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21
oc create -f test-pod.yaml
oc get pod test-pod -o jsonpath='{.spec.containers[*].securityContext.seLinuxOptions.type}'
profile_nginx-binding.process
oc new-project nginx-secure
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: spo-nginx namespace: nginx-secure subjects: - kind: ServiceAccount name: spo-deploy-test roleRef: kind: Role name: spo-nginx apiGroup: rbac.authorization.k8s.io
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: spo-nginx namespace: nginx-secure rules: - apiGroups: - security.openshift.io resources: - securitycontextconstraints resourceNames: - privileged verbs: - use
apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: spo-deploy-test namespace: nginx-secure
apiVersion: apps/v1 kind: Deployment metadata: name: selinux-test namespace: nginx-secure metadata: labels: app: selinux-test spec: replicas: 3 selector: matchLabels: app: selinux-test template: metadata: labels: app: selinux-test spec: serviceAccountName: spo-deploy-test securityContext: seLinuxOptions: type: nginx-secure_nginx-secure.process 1 containers: - name: nginx-unpriv image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080
oc new-project my-namespace
oc label ns my-namespace spo.x-k8s.io/enable-recording=true
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: namespace: my-namespace name: test-recording spec: kind: SelinuxProfile recorder: logs podSelector: matchLabels: app: my-app
apiVersion: v1 kind: Pod metadata: namespace: my-namespace name: my-pod labels: app: my-app spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: nginx image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] - name: redis image: quay.io/security-profiles-operator/redis:6.2.1 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL]
oc -n my-namespace get pods
NAME READY STATUS RESTARTS AGE my-pod 2/2 Running 0 18s
oc -n openshift-security-profiles logs --since=1m --selector name=spod -c log-enricher
I0517 13:55:36.383187 348295 enricher.go:376] log-enricher "msg"="audit" "container"="redis" "namespace"="my-namespace" "node"="ip-10-0-189-53.us-east-2.compute.internal" "perm"="name_bind" "pod"="my-pod" "profile"="test-recording_redis_6kmrb_1684331729" "scontext"="system_u:system_r:selinuxrecording.process:s0:c4,c27" "tclass"="tcp_socket" "tcontext"="system_u:object_r:redis_port_t:s0" "timestamp"="1684331735.105:273965" "type"="selinux"
oc -n my-namepace delete pod my-pod
oc get selinuxprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace
NAME USAGE STATE test-recording-nginx test-recording-nginx_my-namespace.process Installed test-recording-redis test-recording-redis_my-namespace.process Installed
apiVersion: security-profiles-operator.x-k8s.io/v1alpha1 kind: ProfileRecording metadata: # The name of the Recording is the same as the resulting SelinuxProfile CRD # after reconciliation. name: test-recording namespace: my-namespace spec: kind: SelinuxProfile recorder: logs mergeStrategy: containers podSelector: matchLabels: app: sp-record
oc label ns my-namespace security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite=true
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deploy namespace: my-namespace spec: replicas: 3 selector: matchLabels: app: sp-record template: metadata: labels: app: sp-record spec: serviceAccountName: spo-record-sa containers: - name: nginx-record image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 ports: - containerPort: 8080
oc delete deployment nginx-deploy -n my-namespace
oc delete profilerecording test-recording -n my-namespace
oc get selinuxprofiles -lspo.x-k8s.io/recording-id=test-recording -n my-namespace
NAME USAGE STATE test-recording-nginx-record test-recording-nginx-record_my-namespace.process Installed
oc get selinuxprofiles test-recording-nginx-record -o yaml
oc -n openshift-security-profiles patch spod spod --type merge -p '{"spec":{"allowedSyscalls": ["exit", "exit_group", "futex", "nanosleep"]}}'
apiVersion: security-profiles-operator.x-k8s.io/v1beta1 kind: SeccompProfile metadata: namespace: my-namespace name: example-name spec: defaultAction: SCMP_ACT_ERRNO baseProfileName: runc-v1.0.0 syscalls: - action: SCMP_ACT_ALLOW names: - exit_group
oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"enableMemoryOptimization":true}}'
apiVersion: v1 kind: Pod metadata: name: my-recording-pod labels: spo.x-k8s.io/enable-recording: "true"
oc -n openshift-security-profiles patch spod spod --type merge -p '{"spec":{"daemonResourceRequirements": { "requests": {"memory": "256Mi", "cpu": "250m"}, "limits": {"memory": "512Mi", "cpu": "500m"}}}}'
oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"priorityClassName":"my-priority-class"}}'
securityprofilesoperatordaemon.openshift-security-profiles.x-k8s.io/spod patched
oc get svc/metrics -n openshift-security-profiles
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE metrics ClusterIP 10.0.0.228 <none> 443/TCP 43s
oc run --rm -i --restart=Never --image=registry.fedoraproject.org/fedora-minimal:latest -n openshift-security-profiles metrics-test -- bash -c 'curl -ks -H "Authorization: Bearer USD(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-security-profiles/metrics-spod'
HELP security_profiles_operator_seccomp_profile_total Counter about seccomp profile operations. TYPE security_profiles_operator_seccomp_profile_total counter security_profiles_operator_seccomp_profile_total{operation="delete"} 1 security_profiles_operator_seccomp_profile_total{operation="update"} 2
oc get clusterrolebinding spo-metrics-client -o wide
NAME ROLE AGE USERS GROUPS SERVICEACCOUNTS spo-metrics-client ClusterRole/spo-metrics-client 35m openshift-security-profiles/default
oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"enableLogEnricher":true}}'
securityprofilesoperatordaemon.security-profiles-operator.x-k8s.io/spod patched
oc -n openshift-security-profiles logs -f ds/spod log-enricher
I0623 12:51:04.257814 1854764 deleg.go:130] setup "msg"="starting component: log-enricher" "buildDate"="1980-01-01T00:00:00Z" "compiler"="gc" "gitCommit"="unknown" "gitTreeState"="clean" "goVersion"="go1.16.2" "platform"="linux/amd64" "version"="0.4.0-dev" I0623 12:51:04.257890 1854764 enricher.go:44] log-enricher "msg"="Starting log-enricher on node: 127.0.0.1" I0623 12:51:04.257898 1854764 enricher.go:46] log-enricher "msg"="Connecting to local GRPC server" I0623 12:51:04.258061 1854764 enricher.go:69] log-enricher "msg"="Reading from file /var/log/audit/audit.log" 2021/06/23 12:51:04 Seeked /var/log/audit/audit.log - &{Offset:0 Whence:2}
apiVersion: security-profiles-operator.x-k8s.io/v1beta1 kind: SeccompProfile metadata: name: log namespace: default spec: defaultAction: SCMP_ACT_LOG
apiVersion: v1 kind: Pod metadata: name: log-pod spec: securityContext: runAsNonRoot: true seccompProfile: type: Localhost localhostProfile: operator/default/log.json containers: - name: log-container image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21 securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL]
oc -n openshift-security-profiles logs -f ds/spod log-enricher
... I0623 12:59:11.479869 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=3 "syscallName"="close" "timestamp"="1624453150.205:1061" "type"="seccomp" I0623 12:59:11.487323 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=157 "syscallName"="prctl" "timestamp"="1624453150.205:1062" "type"="seccomp" I0623 12:59:11.492157 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=157 "syscallName"="prctl" "timestamp"="1624453150.205:1063" "type"="seccomp" ... I0623 12:59:20.258523 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=12 "syscallName"="brk" "timestamp"="1624453150.235:2873" "type"="seccomp" I0623 12:59:20.263349 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=21 "syscallName"="access" "timestamp"="1624453150.235:2874" "type"="seccomp" I0623 12:59:20.354091 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=257 "syscallName"="openat" "timestamp"="1624453150.235:2875" "type"="seccomp" I0623 12:59:20.358844 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=5 "syscallName"="fstat" "timestamp"="1624453150.235:2876" "type"="seccomp" I0623 12:59:20.363510 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=9 "syscallName"="mmap" "timestamp"="1624453150.235:2877" "type"="seccomp" I0623 12:59:20.454127 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=3 "syscallName"="close" "timestamp"="1624453150.235:2878" "type"="seccomp" I0623 12:59:20.458654 1854764 enricher.go:111] log-enricher "msg"="audit" "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=257 "syscallName"="openat" "timestamp"="1624453150.235:2879" "type"="seccomp" ...
spec: webhookOptions: - name: recording.spo.io objectSelector: matchExpressions: - key: spo-record operator: In values: - "true"
oc -n openshift-security-profiles patch spod spod -p USD(cat /tmp/spod-wh.patch) --type=merge
oc get MutatingWebhookConfiguration spo-mutating-webhook-configuration -oyaml
oc -n openshift-security-profiles logs openshift-security-profiles-<id>
I1019 19:34:14.942464 1 main.go:90] setup "msg"="starting openshift-security-profiles" "buildDate"="2020-10-19T19:31:24Z" "compiler"="gc" "gitCommit"="a3ef0e1ea6405092268c18f240b62015c247dd9d" "gitTreeState"="dirty" "goVersion"="go1.15.1" "platform"="linux/amd64" "version"="0.2.0-dev" I1019 19:34:15.348389 1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen" "addr"=":8080" I1019 19:34:15.349076 1 main.go:126] setup "msg"="starting manager" I1019 19:34:15.349449 1 internal.go:391] controller-runtime/manager "msg"="starting metrics server" "path"="/metrics" I1019 19:34:15.350201 1 controller.go:142] controller "msg"="Starting EventSource" "controller"="profile" "reconcilerGroup"="security-profiles-operator.x-k8s.io" "reconcilerKind"="SeccompProfile" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"defaultAction":""}}} I1019 19:34:15.450674 1 controller.go:149] controller "msg"="Starting Controller" "controller"="profile" "reconcilerGroup"="security-profiles-operator.x-k8s.io" "reconcilerKind"="SeccompProfile" I1019 19:34:15.450757 1 controller.go:176] controller "msg"="Starting workers" "controller"="profile" "reconcilerGroup"="security-profiles-operator.x-k8s.io" "reconcilerKind"="SeccompProfile" "worker count"=1 I1019 19:34:15.453102 1 profile.go:148] profile "msg"="Reconciled profile from SeccompProfile" "namespace"="openshift-security-profiles" "profile"="nginx-1.19.1" "name"="nginx-1.19.1" "resource version"="728" I1019 19:34:15.453618 1 profile.go:148] profile "msg"="Reconciled profile from SeccompProfile" "namespace"="openshift-security-profiles" "profile"="openshift-security-profiles" "name"="openshift-security-profiles" "resource version"="729"
oc exec -t -n openshift-security-profiles openshift-security-profiles-<id> -- ls /var/lib/kubelet/seccomp/operator/my-namespace/my-workload
profile-block.json profile-complain.json
oc delete MutatingWebhookConfiguration spo-mutating-webhook-configuration