Repository: opencontainers/runc
Pull request #3010 (2021-06-08 10:06:34)

libcontainer/cgroups/fs/blkio: support BFQ weight[_device]
- Update the blkio cgroup to support the BFQ I/O Scheduler, that has
  replaced CFQ in the Linux kernel.
- BFQ is controlled through blkio.bfq.weight[_device] instead of
  CFQ's blkio.weight[_device] in cgroups v1.
- BFQ does not support blkio.leaf_weight[_device], so that behavior
  remains untouched.
- Do not change behavior on legacy CFQ systems.
- Enable using blkio weights on BFQ systems.

Signed-off-by: Antti Kervinen <antti.kervinen@intel.com>

askervin: Hi @AkihiroSuda,

There's something essential that I should have brought up earlier:

1. CFQ was removed from Linux kernel (commit f382fb0bcef4, "block: remove legacy IO schedulers") 10/2018.
2. BFQ replaced CFQ in RHEL 8, released 5/2019.
3. BFQ replaced CFQ in Ubuntu 19.10, released 10/2019.

That is, this patch will fix `runc` on Linux running less than 3-years-old kernel. Therefore I'm hoping this patch would be considered to be *a fix* rather than *an enhancement*.

Without this patch adjusting `weight/weight_device` parameters in blockio PRs for [CRI-O](https://github.com/cri-o/cri-o/pull/4873) and [containerd](https://github.com/containerd/containerd/pull/5490) will prevent creating containers on any up-to-date Linux:

```
# kubectl describe pod pod1
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
   ...
  Normal   Created    61s               kubelet            Created container pod1c1
  Normal   Pulled     61s               kubelet            Container image "busybox" already present on machine                                                        
  Warning  Failed     60s               kubelet            Error: container create failed: time="2021-06-09T11:25:57Z" level=error msg="container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/blkio/kubepods.slice/kubepodspodc83ca3da_5552_4fd0_8e17_e62c417b3018.slice/crio-f28b255e00529312c9bd177b3186778b2b8f1369533c9f80a8d48df1455ba7bf.scope/blkio.weight: no such file or directory"
```
kolyshkin: Interesting... we support getting stats from BFQ since PR #2407, but never got to set the parameters for it :)
askervin: > I am looking at a similar code for cgroup v2. In there, we also have a fallback (implemented in commit [8c7ece1](https://github.com/opencontainers/runc/commit/8c7ece1e6d1aba12c85cffa78f814042de262b8a)). There's a weight conversion needed though. Do we need some kind of conversion here as well?

Good question. The situation is:
- CFQ blkio.weight has range [10..1000], the default is 500.
- BFQ blkio.bfq.weight has range [1..1000], the default is 100.
- We do not have information on which I/O scheduler the user/admin thinks is under the hood.

I considered two options:

1. A full-range conversion (linear, or non-linear due to different defaults). I believe that converting values on our code only adds complexity and is likely to result in bug reports like _I tried to set weight "100", but got "500" or "101"_.

2. Convert values < 10 to 10 if running CFQ. After all, even in this case I'd prefer `Numerical result out of range` error. It reveals that the system is running an unexpected I/O scheduler. Either the system needs reconfiguration or weights that the user/admin is using should be recalibrated. I think errors like this should not be hidden.

> It seems that for BFQ we need kernel 5.x, but since this is cgroup v1, our Fedora is not good. Do we need to add a recent Debian or Ubuntu-based test? Or reboot the same Fedora image with cgroup v1 (modify kernel parameters with something like `grubby --args="systemd.unified_cgroup_hierarchy=0" --update-kernel $(grubby --default-kernel)`, reboot, wait, repeat all the tests with cgroup v1).

Maybe adding a new platform to run all tests would be a good topic for another PR?
kolyshkin: @askervin thank you for a detailed reply! I agree that in this case no conversion is probably the best thing to do.

I also agree that testing can be discussed separately -- but without the test we can't be sure if the whole thing works or not (even if there is not too much code).

Patch:
diff --git a/libcontainer/cgroups/fs/blkio.go b/libcontainer/cgroups/fs/blkio.go
index 99407474..e5240a17 100644
--- a/libcontainer/cgroups/fs/blkio.go
+++ b/libcontainer/cgroups/fs/blkio.go
@@ -6,6 +6,7 @@ import (
 	"bufio"
 	"fmt"
 	"os"
+	"path/filepath"
 	"strconv"
 	"strings"
 
@@ -14,7 +15,10 @@ import (
 	"github.com/opencontainers/runc/libcontainer/configs"
 )
 
-type BlkioGroup struct{}
+type BlkioGroup struct {
+	weightFilename       string
+	weightDeviceFilename string
+}
 
 func (s *BlkioGroup) Name() string {
 	return "blkio"
@@ -25,8 +29,9 @@ func (s *BlkioGroup) Apply(path string, d *cgroupData) error {
 }
 
 func (s *BlkioGroup) Set(path string, r *configs.Resources) error {
+	s.detectWeightFilenames(path)
 	if r.BlkioWeight != 0 {
-		if err := fscommon.WriteFile(path, "blkio.weight", strconv.FormatUint(uint64(r.BlkioWeight), 10)); err != nil {
+		if err := fscommon.WriteFile(path, s.weightFilename, strconv.FormatUint(uint64(r.BlkioWeight), 10)); err != nil {
 			return err
 		}
 	}
@@ -37,7 +42,7 @@ func (s *BlkioGroup) Set(path string, r *configs.Resources) error {
 		}
 	}
 	for _, wd := range r.BlkioWeightDevice {
-		if err := fscommon.WriteFile(path, "blkio.weight_device", wd.WeightString()); err != nil {
+		if err := fscommon.WriteFile(path, s.weightDeviceFilename, wd.WeightString()); err != nil {
 			return err
 		}
 		if err := fscommon.WriteFile(path, "blkio.leaf_weight_device", wd.LeafWeightString()); err != nil {
@@ -287,3 +292,17 @@ func (s *BlkioGroup) GetStats(path string, stats *cgroups.Stats) error {
 	}
 	return nil
 }
+
+func (s *BlkioGroup) detectWeightFilenames(path string) {
+	if s.weightFilename != "" {
+		// Already detected.
+		return
+	}
+	if cgroups.PathExists(filepath.Join(path, "blkio.weight")) {
+		s.weightFilename = "blkio.weight"
+		s.weightDeviceFilename = "blkio.weight_device"
+	} else {
+		s.weightFilename = "blkio.bfq.weight"
+		s.weightDeviceFilename = "blkio.bfq.weight_device"
+	}
+}
diff --git a/libcontainer/cgroups/fs/memory.go b/libcontainer/cgroups/fs/memory.go
index 35c77dc5..15820fa4 100644
--- a/libcontainer/cgroups/fs/memory.go
+++ b/libcontainer/cgroups/fs/memory.go
@@ -255,7 +255,6 @@ func getPageUsageByNUMA(cgroupPath string) (cgroups.PageUsageByNUMA, error) {
 	} else if err != nil {
 		return stats, err
 	}
-	defer file.Close()
 
 	// File format is documented in linux/Documentation/cgroup-v1/memory.txt
 	// and it looks like this:
diff --git a/libcontainer/cgroups/systemd/common.go b/libcontainer/cgroups/systemd/common.go
index c9995bfe..67c61278 100644
--- a/libcontainer/cgroups/systemd/common.go
+++ b/libcontainer/cgroups/systemd/common.go
@@ -178,7 +178,7 @@ func allowAllDevices() []systemdDbus.Property {
 // corresponding set of systemd properties to configure the devices correctly.
 func generateDeviceProperties(r *configs.Resources) ([]systemdDbus.Property, error) {
 	if r.SkipDevices {
-		return nil, nil
+		return allowAllDevices(), nil
 	}
 
 	properties := []systemdDbus.Property{
diff --git a/libcontainer/cgroups/systemd/dbus.go b/libcontainer/cgroups/systemd/dbus.go
index a70a9df4..3c30f6bb 100644
--- a/libcontainer/cgroups/systemd/dbus.go
+++ b/libcontainer/cgroups/systemd/dbus.go
@@ -21,12 +21,9 @@ type dbusConnManager struct{}
 
 // newDbusConnManager initializes systemd dbus connection manager.
 func newDbusConnManager(rootless bool) *dbusConnManager {
-	dbusMu.Lock()
-	defer dbusMu.Unlock()
 	if dbusInited && rootless != dbusRootless {
 		panic("can't have both root and rootless dbus")
 	}
-	dbusInited = true
 	dbusRootless = rootless
 	return &dbusConnManager{}
 }
diff --git a/libcontainer/cgroups/systemd/v1.go b/libcontainer/cgroups/systemd/v1.go
index 1f533c81..06526c54 100644
--- a/libcontainer/cgroups/systemd/v1.go
+++ b/libcontainer/cgroups/systemd/v1.go
@@ -339,24 +339,27 @@ func (m *legacyManager) Set(r *configs.Resources) error {
 		return err
 	}
 
-	// Figure out the current freezer state, so we can revert to it after we
-	// temporarily freeze the container.
-	targetFreezerState, err := m.GetFreezerState()
-	if err != nil {
-		return err
-	}
-	if targetFreezerState == configs.Undefined {
-		targetFreezerState = configs.Thawed
-	}
-
 	// We have to freeze the container while systemd sets the cgroup settings.
 	// The reason for this is that systemd's application of DeviceAllow rules
 	// is done disruptively, resulting in spurrious errors to common devices
 	// (unlike our fs driver, they will happily write deny-all rules to running
 	// containers). So we freeze the container to avoid them hitting the cgroup
 	// error. But if the freezer cgroup isn't supported, we just warn about it.
-	if err := m.Freeze(configs.Frozen); err != nil {
-		logrus.Infof("freeze container before SetUnitProperties failed: %v", err)
+	targetFreezerState := configs.Undefined
+	if !m.cgroups.SkipDevices {
+		// Figure out the current freezer state, so we can revert to it after we
+		// temporarily freeze the container.
+		targetFreezerState, err = m.GetFreezerState()
+		if err != nil {
+			return err
+		}
+		if targetFreezerState == configs.Undefined {
+			targetFreezerState = configs.Thawed
+		}
+
+		if err := m.Freeze(configs.Frozen); err != nil {
+			logrus.Infof("freeze container before SetUnitProperties failed: %v", err)
+		}
 	}
 
 	if err := setUnitProperties(m.dbus, getUnitName(m.cgroups), properties...); err != nil {
diff --git a/libcontainer/cgroups/systemd/v2.go b/libcontainer/cgroups/systemd/v2.go
index 262aaca8..cbd3a308 100644
--- a/libcontainer/cgroups/systemd/v2.go
+++ b/libcontainer/cgroups/systemd/v2.go
@@ -418,24 +418,27 @@ func (m *unifiedManager) Set(r *configs.Resources) error {
 		return err
 	}
 
-	// Figure out the current freezer state, so we can revert to it after we
-	// temporarily freeze the container.
-	targetFreezerState, err := m.GetFreezerState()
-	if err != nil {
-		return err
-	}
-	if targetFreezerState == configs.Undefined {
-		targetFreezerState = configs.Thawed
-	}
-
 	// We have to freeze the container while systemd sets the cgroup settings.
 	// The reason for this is that systemd's application of DeviceAllow rules
 	// is done disruptively, resulting in spurrious errors to common devices
 	// (unlike our fs driver, they will happily write deny-all rules to running
 	// containers). So we freeze the container to avoid them hitting the cgroup
 	// error. But if the freezer cgroup isn't supported, we just warn about it.
-	if err := m.Freeze(configs.Frozen); err != nil {
-		logrus.Infof("freeze container before SetUnitProperties failed: %v", err)
+	targetFreezerState := configs.Undefined
+	if !m.cgroups.SkipDevices {
+		// Figure out the current freezer state, so we can revert to it after we
+		// temporarily freeze the container.
+		targetFreezerState, err = m.GetFreezerState()
+		if err != nil {
+			return err
+		}
+		if targetFreezerState == configs.Undefined {
+			targetFreezerState = configs.Thawed
+		}
+
+		if err := m.Freeze(configs.Frozen); err != nil {
+			logrus.Infof("freeze container before SetUnitProperties failed: %v", err)
+		}
 	}
 
 	if err := setUnitProperties(m.dbus, getUnitName(m.cgroups), properties...); err != nil {
diff --git a/libcontainer/configs/validate/validator.go b/libcontainer/configs/validate/validator.go
index b0254600..0b38b8bd 100644
--- a/libcontainer/configs/validate/validator.go
+++ b/libcontainer/configs/validate/validator.go
@@ -12,7 +12,6 @@ import (
 	"github.com/opencontainers/runc/libcontainer/configs"
 	"github.com/opencontainers/runc/libcontainer/intelrdt"
 	selinux "github.com/opencontainers/selinux/go-selinux"
-	"github.com/sirupsen/logrus"
 	"golang.org/x/sys/unix"
 )
 
@@ -30,7 +29,6 @@ type check func(config *configs.Config) error
 
 func (v *ConfigValidator) Validate(config *configs.Config) error {
 	checks := []check{
-		v.cgroups,
 		v.rootfs,
 		v.network,
 		v.hostname,
@@ -40,21 +38,17 @@ func (v *ConfigValidator) Validate(config *configs.Config) error {
 		v.sysctl,
 		v.intelrdt,
 		v.rootlessEUID,
+		v.mounts,
 	}
 	for _, c := range checks {
 		if err := c(config); err != nil {
 			return err
 		}
 	}
-	// Relaxed validation rules for backward compatibility
-	warns := []check{
-		v.mounts, // TODO (runc v1.x.x): make this an error instead of a warning
-	}
-	for _, c := range warns {
-		if err := c(config); err != nil {
-			logrus.WithError(err).Warnf("invalid configuration")
-		}
+	if err := v.cgroups(config); err != nil {
+		return err
 	}
+
 	return nil
 }
 
diff --git a/libcontainer/specconv/spec_linux.go b/libcontainer/specconv/spec_linux.go
index 8474769c..ac83123d 100644
--- a/libcontainer/specconv/spec_linux.go
+++ b/libcontainer/specconv/spec_linux.go
@@ -333,10 +333,7 @@ func CreateLibcontainerConfig(opts *CreateOpts) (*configs.Config, error) {
 
 func createLibcontainerMount(cwd string, m specs.Mount) (*configs.Mount, error) {
 	if !filepath.IsAbs(m.Destination) {
-		// Relax validation for backward compatibility
-		// TODO (runc v1.x.x): change warning to an error
-		// return nil, fmt.Errorf("mount destination %s is not absolute", m.Destination)
-		logrus.Warnf("mount destination %s is not absolute. Support for non-absolute mount destinations will be removed in a future release.", m.Destination)
+		return nil, fmt.Errorf("mount destination %s not absolute", m.Destination)
 	}
 	flags, pgflags, data, ext := parseMountOptions(m.Options)
 	source := m.Source


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 47be5804..cc46afe6 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -19,7 +19,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        # Docker/Moby still builds runc with Go 1.13, so we should still support Go 1.13.
+        # Dockre/Moby still builds runc with Go 1.13, so we should still support Go 1.13.
         go-version: [1.13.x, 1.15.x, 1.16.x]
         rootless: ["rootless", ""]
         race: ["-race", ""]
diff --git a/libcontainer/cgroups/fs/blkio_test.go b/libcontainer/cgroups/fs/blkio_test.go
index 4dbb48ad..793c0432 100644
--- a/libcontainer/cgroups/fs/blkio_test.go
+++ b/libcontainer/cgroups/fs/blkio_test.go
@@ -171,62 +171,80 @@ func appendBlkioStatEntry(blkioStatEntries *[]cgroups.BlkioStatEntry, major, min
 }
 
 func TestBlkioSetWeight(t *testing.T) {
-	helper := NewCgroupTestUtil("blkio", t)
-	defer helper.cleanup()
-
 	const (
 		weightBefore = 100
 		weightAfter  = 200
 	)
 
-	helper.writeFileContents(map[string]string{
-		"blkio.weight": strconv.Itoa(weightBefore),
-	})
-
-	helper.CgroupData.config.Resources.BlkioWeight = weightAfter
-	blkio := &BlkioGroup{}
-	if err := blkio.Set(helper.CgroupPath, helper.CgroupData.config.Resources); err != nil {
-		t.Fatal(err)
-	}
-
-	value, err := fscommon.GetCgroupParamUint(helper.CgroupPath, "blkio.weight")
-	if err != nil {
-		t.Fatalf("Failed to parse blkio.weight - %s", err)
-	}
-
-	if value != weightAfter {
-		t.Fatal("Got the wrong value, set blkio.weight failed.")
+	for _, legacyIOScheduler := range []bool{false, true} {
+		// Populate cgroup
+		helper := NewCgroupTestUtil("blkio", t)
+		defer helper.cleanup()
+		weightFilename := "blkio.bfq.weight"
+		if legacyIOScheduler {
+			weightFilename = "blkio.weight"
+		}
+		helper.writeFileContents(map[string]string{
+			weightFilename: strconv.Itoa(weightBefore),
+		})
+		// Apply new configuration
+		helper.CgroupData.config.Resources.BlkioWeight = weightAfter
+		blkio := &BlkioGroup{}
+		if err := blkio.Set(helper.CgroupPath, helper.CgroupData.config.Resources); err != nil {
+			t.Fatal(err)
+		}
+		// Verify results
+		if weightFilename != blkio.weightFilename {
+			t.Fatalf("weight filename detection failed: expected %q, detected %q", weightFilename, blkio.weightFilename)
+		}
+		value, err := fscommon.GetCgroupParamUint(helper.CgroupPath, weightFilename)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if value != weightAfter {
+			t.Fatalf("Got the wrong value, set %s failed.", weightFilename)
+		}
 	}
 }
 
 func TestBlkioSetWeightDevice(t *testing.T) {
-	helper := NewCgroupTestUtil("blkio", t)
-	defer helper.cleanup()
-
 	const (
 		weightDeviceBefore = "8:0 400"
 	)
 
-	wd := configs.NewWeightDevice(8, 0, 500, 0)
-	weightDeviceAfter := wd.WeightString()
-
-	helper.writeFileContents(map[string]string{
-		"blkio.weight_device": weightDeviceBefore,
-	})
-
-	helper.CgroupData.config.Resources.BlkioWeightDevice = []*configs.WeightDevice{wd}
-	blkio := &BlkioGroup{}
-	if err := blkio.Set(helper.CgroupPath, helper.CgroupData.config.Resources); err != nil {
-		t.Fatal(err)
-	}
-
-	value, err := fscommon.GetCgroupParamString(helper.CgroupPath, "blkio.weight_device")
-	if err != nil {
-		t.Fatalf("Failed to parse blkio.weight_device - %s", err)
-	}
-
-	if value != weightDeviceAfter {
-		t.Fatal("Got the wrong value, set blkio.weight_device failed.")
+	for _, legacyIOScheduler := range []bool{false, true} {
+		// Populate cgroup
+		helper := NewCgroupTestUtil("blkio", t)
+		defer helper.cleanup()
+		weightFilename := "blkio.bfq.weight"
+		weightDeviceFilename := "blkio.bfq.weight_device"
+		if legacyIOScheduler {
+			weightFilename = "blkio.weight"
+			weightDeviceFilename = "blkio.weight_device"
+		}
+		helper.writeFileContents(map[string]string{
+			weightFilename:       "",
+			weightDeviceFilename: weightDeviceBefore,
+		})
+		// Apply new configuration
+		wd := configs.NewWeightDevice(8, 0, 500, 0)
+		weightDeviceAfter := wd.WeightString()
+		helper.CgroupData.config.Resources.BlkioWeightDevice = []*configs.WeightDevice{wd}
+		blkio := &BlkioGroup{}
+		if err := blkio.Set(helper.CgroupPath, helper.CgroupData.config.Resources); err != nil {
+			t.Fatal(err)
+		}
+		// Verify results
+		if weightDeviceFilename != blkio.weightDeviceFilename {
+			t.Fatalf("weight_device filename detection failed: expected %q, detected %q", weightDeviceFilename, blkio.weightDeviceFilename)
+		}
+		value, err := fscommon.GetCgroupParamString(helper.CgroupPath, weightDeviceFilename)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if value != weightDeviceAfter {
+			t.Fatalf("Got the wrong value, set %s failed.", weightDeviceFilename)
+		}
 	}
 }
 
@@ -247,23 +265,27 @@ func TestBlkioSetMultipleWeightDevice(t *testing.T) {
 	// is present will suffice for the test to ensure multiple writes are done.
 	weightDeviceAfter := wd2.WeightString()
 
+	blkio := &BlkioGroup{}
+	blkio.detectWeightFilenames(helper.CgroupPath)
+	if blkio.weightDeviceFilename != "blkio.bfq.weight_device" {
+		t.Fatalf("when blkio controller is unavailable, expected to use \"blkio.bfq.weight_device\", tried to use %q", blkio.weightDeviceFilename)
+	}
 	helper.writeFileContents(map[string]string{
-		"blkio.weight_device": weightDeviceBefore,
+		blkio.weightDeviceFilename: weightDeviceBefore,
 	})
 
 	helper.CgroupData.config.Resources.BlkioWeightDevice = []*configs.WeightDevice{wd1, wd2}
-	blkio := &BlkioGroup{}
 	if err := blkio.Set(helper.CgroupPath, helper.CgroupData.config.Resources); err != nil {
 		t.Fatal(err)
 	}
 
-	value, err := fscommon.GetCgroupParamString(helper.CgroupPath, "blkio.weight_device")
+	value, err := fscommon.GetCgroupParamString(helper.CgroupPath, blkio.weightDeviceFilename)
 	if err != nil {
-		t.Fatalf("Failed to parse blkio.weight_device - %s", err)
+		t.Fatal(err)
 	}
 
 	if value != weightDeviceAfter {
-		t.Fatal("Got the wrong value, set blkio.weight_device failed.")
+		t.Fatalf("Got the wrong value, set %s failed.", blkio.weightDeviceFilename)
 	}
 }
 
diff --git a/libcontainer/configs/validate/validator_test.go b/libcontainer/configs/validate/validator_test.go
index 687ce1b0..7d1ef0f6 100644
--- a/libcontainer/configs/validate/validator_test.go
+++ b/libcontainer/configs/validate/validator_test.go
@@ -323,12 +323,10 @@ func TestValidateMounts(t *testing.T) {
 		isErr bool
 		dest  string
 	}{
-		// TODO (runc v1.x.x): make these relative paths an error. See https://github.com/opencontainers/runc/pull/3004
-		{isErr: false, dest: "not/an/abs/path"},
-		{isErr: false, dest: "./rel/path"},
-		{isErr: false, dest: "./rel/path"},
-		{isErr: false, dest: "../../path"},
-
+		{isErr: true, dest: "not/an/abs/path"},
+		{isErr: true, dest: "./rel/path"},
+		{isErr: true, dest: "./rel/path"},
+		{isErr: true, dest: "../../path"},
 		{isErr: false, dest: "/abs/path"},
 		{isErr: false, dest: "/abs/but/../unclean"},
 	}
diff --git a/tests/integration/dev.bats b/tests/integration/dev.bats
index 35a9ac06..80ceb0a3 100644
--- a/tests/integration/dev.bats
+++ b/tests/integration/dev.bats
@@ -33,7 +33,7 @@ function teardown() {
 	[[ "${lines[0]}" =~ "crw-rw-rw".+"1".+"0".+"0".+"5,".+"2".+"/dev/ptmx" ]]
 }
 
-@test "runc run/update [device cgroup deny]" {
+@test "runc run [device cgroup deny]" {
 	requires root
 
 	update_config ' .linux.resources.devices = [{"allow": false, "access": "rwm"}]
@@ -52,18 +52,6 @@ function teardown() {
 	runc exec test_deny sh -c 'head -n 1 /dev/kmsg'
 	[ "$status" -eq 1 ]
 	[[ "${output}" == *'Operation not permitted'* ]]
-
-	runc update test_deny --pids-limit 42
-
-	# test write
-	runc exec test_deny sh -c 'hostname | tee /dev/kmsg'
-	[ "$status" -eq 1 ]
-	[[ "${output}" == *'Operation not permitted'* ]]
-
-	# test read
-	runc exec test_deny sh -c 'head -n 1 /dev/kmsg'
-	[ "$status" -eq 1 ]
-	[[ "${output}" == *'Operation not permitted'* ]]
 }
 
 @test "runc run [device cgroup allow rw char device]" {