Repository: kubernetes-sigs/security-profiles-operator
Pull request #479 (2021-06-09 12:01:58)

Fix log enricher read and automatically add `/dev/kmsg` device 
#### What type of PR is this?

/kind feature

#### What this PR does / why we need it:

When using CRI-O, we can utilize the annotation
`io.kubernetes.cri-o.Devices` to correctly mount the device into the
container. This means we do not rely on a symlink any more and can use
the device directly.

The enricher read has been fixed because one read always returns a full
line which makes the offset calculation obsolete.

#### Which issue(s) this PR fixes:

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.

Fixes #

or

None
-->
None
#### Does this PR have test?
None
<!--
If tests aren't applicable just write N/A.
-->

#### Special notes for your reviewer:
None
#### Does this PR introduce a user-facing change?

<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
Automatically mount /dev/kmsg for log enricher usage if running with CRI-O and an allowed `io.kubernetes.cri-o.Devices` annotation.
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/479#" title="Author self-approved">saschagrunert</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)~~ [saschagrunert]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov-commenter: # [Codecov](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/pull/479?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
> Merging [#479](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/pull/479?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (98fc2dd) into [master](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/commit/fd1f6ee464ffa9a17d498ed494804081f220aec7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (fd1f6ee) will **decrease** coverage by `0.13%`.
> The diff coverage is `0.00%`.


```diff
@@            Coverage Diff             @@
##           master     #479      +/-   ##
==========================================
- Coverage   37.92%   37.79%   -0.14%     
==========================================
  Files          23       23              
  Lines        1110     1114       +4     
==========================================
  Hits          421      421              
- Misses        674      678       +4     
  Partials       15       15              
```

k8s-ci-robot: @saschagrunert: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 98fc2dd0f2a19e204e04282ef97fb84cca85d67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/479/pull-security-profiles-operator-test-e2e/1402596834802667520) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=479). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Asaschagrunert). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
saschagrunert: /test pull-security-profiles-operator-test-e2e
k8s-ci-robot: @saschagrunert: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 98fc2dd0f2a19e204e04282ef97fb84cca85d67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/479/pull-security-profiles-operator-test-e2e/1402601569366577152) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=479). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Asaschagrunert). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
saschagrunert: /test pull-security-profiles-operator-test-e2e
k8s-ci-robot: @saschagrunert: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 98fc2dd0f2a19e204e04282ef97fb84cca85d67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/479/pull-security-profiles-operator-test-e2e/1402609425180528640) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=479). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Asaschagrunert). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
saschagrunert: /test pull-security-profiles-operator-test-e2e
JAORMX: /retest
saschagrunert: Rebased, let's see if the tests pass now otherwise it's a bug in the PR.
k8s-ci-robot: @saschagrunert: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | dfa7067e90347179ba2f5e0bfb5a4b0e44a8aabd | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/479/pull-security-profiles-operator-test-e2e/1405058214772346880) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=479). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Asaschagrunert). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
saschagrunert: /retest

Patch:
diff --git a/hack/ci/Vagrantfile b/hack/ci/Vagrantfile
index 7cbcb9353..a3faa19c1 100644
--- a/hack/ci/Vagrantfile
+++ b/hack/ci/Vagrantfile
@@ -31,7 +31,10 @@ Vagrant.configure("2") do |config|
         openssl \
         podman
 
+      # Use a non-localhost DNS to avoid cluster DNS lookup loops
       echo "nameserver 8.8.8.8" > /etc/resolv.conf
+
+      # Configure system to work seamlessly on single node clusters
       modprobe br_netfilter
       ip6tables --list >/dev/null
       iptables -F
diff --git a/hack/ci/env.sh b/hack/ci/env.sh
index 8edd2de0f..35bc80222 100755
--- a/hack/ci/env.sh
+++ b/hack/ci/env.sh
@@ -28,4 +28,5 @@ export KUBE_MASTER_URL=$IP
 export KUBE_MASTER_IP=$IP
 export KUBE_MASTER=$IP
 
+# Added for faster debugging with lower verbosity
 alias k=kubectl
diff --git a/internal/pkg/config/config.go b/internal/pkg/config/config.go
index fe2c8cdbf..ee1bf174c 100644
--- a/internal/pkg/config/config.go
+++ b/internal/pkg/config/config.go
@@ -67,6 +67,9 @@ const (
 
 	// EnricherLogFile is the path to the kernel messages log file used for the enricher.
 	EnricherLogFile = "/var/log/spo.log"
+
+	// DevKmsgPath is the path to the kernel log messages.
+	DevKmsgPath = "/dev/kmsg"
 )
 
 // ProfileRecordingOutputPath is the path where the recorded profiles will be
diff --git a/internal/pkg/daemon/enricher/enricher.go b/internal/pkg/daemon/enricher/enricher.go
index 5238d18e2..5964cd449 100644
--- a/internal/pkg/daemon/enricher/enricher.go
+++ b/internal/pkg/daemon/enricher/enricher.go
@@ -25,6 +25,7 @@ import (
 
 	"github.com/go-logr/logr"
 	"github.com/pkg/errors"
+	"sigs.k8s.io/release-utils/util"
 
 	"sigs.k8s.io/security-profiles-operator/internal/pkg/config"
 )
@@ -39,11 +40,17 @@ func Run(logger logr.Logger) error {
 		return err
 	}
 
-	logger.V(1).Info("Starting log-enricher on node: " + nodeName)
+	logger.Info("Starting log-enricher on node: " + nodeName)
 
 	auditLines := make(chan string)
 	tailErrors := make(chan error)
-	go tailFile(config.EnricherLogFile, auditLines, tailErrors)
+
+	logFile := config.EnricherLogFile
+	if util.Exists(config.DevKmsgPath) {
+		logFile = config.DevKmsgPath
+	}
+	logger.Info("Reading from file " + logFile)
+	go tailFile(logger, logFile, auditLines, tailErrors)
 
 	for {
 		var line string
@@ -84,7 +91,7 @@ func Run(logger logr.Logger) error {
 	}
 }
 
-func tailFile(filePath string, lines chan string, errChan chan error) {
+func tailFile(logger logr.Logger, filePath string, lines chan string, errChan chan error) {
 	file, err := os.Open(filepath.Clean(filePath))
 	if err != nil {
 		errChan <- errors.Wrap(err, "open audit log file")
@@ -97,24 +104,24 @@ func tailFile(filePath string, lines chan string, errChan chan error) {
 		}
 	}()
 
-	offset, err := file.Seek(0, io.SeekEnd)
-	if err != nil {
+	if _, err := file.Seek(0, io.SeekEnd); err != nil {
 		errChan <- errors.Wrap(err, "seek end audit log")
 		return
 	}
 
 	buffer := make([]byte, 1024)
 	for {
-		readBytes, err := file.ReadAt(buffer, offset)
+		readBytes, err := file.Read(buffer)
 		if err != nil {
 			if !errors.Is(err, io.EOF) {
 				errChan <- errors.Wrap(err, "read audit log buffer")
 				return
 			}
 		}
-		offset += int64(readBytes)
 		if readBytes != 0 {
-			lines <- string(buffer[:readBytes])
+			line := string(buffer[:readBytes])
+			logger.Info(line)
+			lines <- line
 		}
 		time.Sleep(time.Second)
 	}
diff --git a/internal/pkg/manager/spod/bindata/spod.go b/internal/pkg/manager/spod/bindata/spod.go
index c0ff29e12..807cd512d 100644
--- a/internal/pkg/manager/spod/bindata/spod.go
+++ b/internal/pkg/manager/spod/bindata/spod.go
@@ -34,7 +34,7 @@ var (
 	truly                           = true
 	userRoot                  int64 = 0
 	userRootless                    = int64(config.UserRootless)
-	hostPathFile                    = v1.HostPathFile
+	hostPathFileOrCreate            = v1.HostPathFileOrCreate
 	hostPathDirectory               = v1.HostPathDirectory
 	hostPathDirectoryOrCreate       = v1.HostPathDirectoryOrCreate
 	metricsPort               int32 = 9443
@@ -65,8 +65,9 @@ var Manifest = &appsv1.DaemonSet{
 		Template: v1.PodTemplateSpec{
 			ObjectMeta: metav1.ObjectMeta{
 				Annotations: map[string]string{
-					"openshift.io/scc":         "privileged",
-					v1.SeccompPodAnnotationKey: v1.SeccompProfileRuntimeDefault,
+					"openshift.io/scc":            "privileged",
+					"io.kubernetes.cri-o.Devices": config.DevKmsgPath,
+					v1.SeccompPodAnnotationKey:    v1.SeccompProfileRuntimeDefault,
 					v1.SeccompContainerAnnotationKeyPrefix + config.OperatorName: "localhost/security-profiles-operator.json",
 				},
 				Labels: map[string]string{
@@ -523,7 +524,7 @@ semodule -i /opt/spo-profiles/selinuxd.cil
 						VolumeSource: v1.VolumeSource{
 							HostPath: &v1.HostPathVolumeSource{
 								Path: config.EnricherLogFile,
-								Type: &hostPathFile,
+								Type: &hostPathFileOrCreate,
 							},
 						},
 					},


Test patch:
diff --git a/test/suite_test.go b/test/suite_test.go
index 1d2c24c3c..37b42b16e 100644
--- a/test/suite_test.go
+++ b/test/suite_test.go
@@ -247,7 +247,6 @@ func (e *kinde2e) TearDownTest() {
 	e.logf("#### Snapshot of security-profiles-operator namespace ####")
 	e.kubectl("--namespace", "security-profiles-operator", "describe", "all")
 	e.logf("########")
-	e.kubectl("-n", "kube-system", "logs", "deploy/coredns")
 
 	e.logf("Destroying cluster")
 	e.run(