Repository: opencontainers/runc
Pull request #2466 (2020-06-10 02:25:20)

cgroups/systemd: add setting CPUQuotaPeriod prop
_This is currently based on and includes #2458 and will be rebased once that one is merged. Please only review the last commit._

For some reason, runc systemd drivers (both v1 and v2) never set
systemd unit property named `CPUQuotaPeriod` (known as
`CPUQuotaPeriodUSec` on dbus and in `systemctl show` output).

Set it, and add a check to all the integration tests. The check is less
than trivial because, when not set, the value is shown as "infinity" but
when set to the same (default) value, shown as "100ms", so in case we
expect 100ms (period = 100000 us), we have to _also_ check for
"infinity".

Fixes https://github.com/opencontainers/runc/issues/2465

kolyshkin: On Ubuntu 18.04 we have this:

> Cannot set property CPUQuotaPeriodUSec, or unknown property.

Apparently systemd in ubuntu-18.04 did not have support for this parameter.

Looking at the systemd git, this was added in **systemd v242** (https://github.com/systemd/systemd/pull/9594).

I think we can check systemd version during controller initialization and set this conditionally.
AkihiroSuda: https://github.com/opencontainers/runc/pull/2458 is merged, please rebase

kolyshkin: Added systemd version detection and conditional setting. 

I'm not quite sure why this one fails. @AkihiroSuda can you please take a look?

From https://travis-ci.org/github/opencontainers/runc/jobs/698672952#L4689:

> make localrootlessintegration RUNC_USE_SYSTEMD=yes
> ...
> [01] run rootless tests ... ()
> ...
> not ok 79 update cgroup v1/v2 common limits
> (in test file /vagrant/tests/integration/update.bats, line 41)
>  `[ "$status" -eq 0 ]' failed
> ...
>  time="2020-06-15T19:58:00Z" level=error msg="container_linux.go:353: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: error while starting unit \"runc-cgroups-integration-test.scope\" with properties [{Name:Description Value:\"libcontainer container integration-test\"} {Name:Slice Value:\"machine.slice\"} {Name:PIDs Value:@au [56928]} {Name:Delegate Value:true} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:DefaultDependencies Value:false} {Name:DevicePolicy Value:\"strict\"} {Name:DeviceAllow Value:@a(ss) []} {Name:DeviceAllow Value:[\"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\"]} {Name:MemoryMax Value:@t 33554432} {Name:MemoryLow Value:@t 25165824} {Name:CPUWeight Value:@t 4} {Name:CPUQuotaPeriodUSec Value:@t 1000000} {Name:CPUQuotaPerSecUSec Value:@t 500000} {Name:TasksAccounting Value:true} {Name:TasksMax Value:@t 20}]: Permission denied"
kolyshkin: Added a commit to temporarily disable setting cpu quota period to check that the error above is caused by it.
kolyshkin: Still have the same error after commenting out setting `CPUQuotaPeriod`. Hmm, is the master broken or I am not seeing something?

> not ok 79 update cgroup v1/v2 common limits
> (in test file /vagrant/tests/integration/update.bats, line 41)
>  `[ "$status" -eq 0 ]' failed
> time="2020-06-15T21:54:35Z" level=error msg="container_linux.go:353: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: error while starting unit \"runc-cgroups-integration-test.scope\" with properties [{Name:Description Value:\"libcontainer container integration-test\"} {Name:Slice Value:\"machine.slice\"} {Name:PIDs Value:@au [56608]} {Name:Delegate Value:true} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:DefaultDependencies Value:false} {Name:DevicePolicy Value:\"strict\"} {Name:DeviceAllow Value:@a(ss) []} {Name:DeviceAllow Value:[\"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\", \"INVALID\"]} {Name:MemoryMax Value:@t 33554432} {Name:MemoryLow Value:@t 25165824} {Name:CPUWeight Value:@t 4} {Name:CPUQuotaPerSecUSec Value:@t 500000} {Name:TasksAccounting Value:true} {Name:TasksMax Value:@t 20}]: Permission denied"

Patch:
diff --git a/.gitignore b/.gitignore
index 84485cb9..282e34ec 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,5 +4,3 @@ vendor/pkg
 contrib/cmd/recvtty/recvtty
 man/man8
 release
-Vagrantfile
-.vagrant
diff --git a/.travis.yml b/.travis.yml
index 6a4c2dc9..e6351555 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,10 +20,15 @@ matrix:
       script:
         - make all
         - sudo PATH="$PATH" make localintegration RUNC_USE_SYSTEMD=1
-    - name: "fedora32"
+    - go: 1.13.x
+      name: "cgroup-v2"
+      env:
+        - VAGRANT_VERSION=2.2.7
       before_install:
-        - sudo ./script/install-vagrant.sh
-        - ln -sf Vagrantfile.fedora32 Vagrantfile
+        - cat /proc/cpuinfo
+# https://github.com/alvistack/ansible-role-virtualbox/blob/6887b020b0ca5c59ddb6620d73f053ffb84f4126/.travis.yml#L30
+        - sudo apt-get install -q -y bridge-utils dnsmasq-base ebtables libvirt-bin libvirt-dev qemu-kvm qemu-utils ruby-dev && wget https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}_$(uname -m).deb && sudo dpkg -i vagrant_${VAGRANT_VERSION}_$(uname -m).deb && rm -f vagrant_${VAGRANT_VERSION}_$(uname -m).deb
+        - sudo vagrant plugin install vagrant-libvirt
         - sudo vagrant up && sudo mkdir -p /root/.ssh && sudo sh -c "vagrant ssh-config >> /root/.ssh/config"
       script:
         - sudo ssh default -t 'cd /vagrant && sudo make localunittest'
@@ -35,23 +40,6 @@ matrix:
         - sudo ssh default -t 'cd /vagrant && sudo make localrootlessintegration RUNC_USE_SYSTEMD=yes'
         # same setup but with fs2 driver (rootless) instead of systemd
         - sudo ssh default -t 'cd /vagrant && sudo make localrootlessintegration'
-    - name: "centos7"
-      before_install:
-        - sudo ./script/install-vagrant.sh
-        - ln -sf Vagrantfile.centos7 Vagrantfile
-        - sudo vagrant up && sudo mkdir -p /root/.ssh && sudo sh -c "vagrant ssh-config >> /root/.ssh/config"
-      script:
-        # kernel 3.10 (frankenized), systemd 219
-        - sudo ssh default 'rpm -q centos-release kernel systemd'
-        # FIXME: the following unit tests are skipped (TESTFLAGS=-short):
-        #        FAIL: TestPidsSystemd: utils_test.go:55: exec_test.go:630: unexpected error: container_linux.go:353: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: mountpoint for devices not found
-        #        FAIL: TestRunWithKernelMemorySystemd: exec_test.go:713: runContainer failed with kernel memory limit: container_linux.go:353: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: mkdir : no such file or directory
-        - sudo ssh default -t 'sudo -i make -C /vagrant localunittest TESTFLAGS=-short'
-        - sudo ssh default -t 'sudo -i make -C /vagrant localintegration'
-        - sudo ssh default -t 'sudo -i make -C /vagrant localintegration RUNC_USE_SYSTEMD=1'
-        # FIXME: rootless is skipped because of EPERM on writing cgroup.procs
-        # - sudo ssh default -t 'sudo -i make -C /vagrant localrootlessintegration'
-
   allow_failures:
     - go: tip
 
diff --git a/Vagrantfile.fedora32 b/Vagrantfile
similarity index 100%
rename from Vagrantfile.fedora32
rename to Vagrantfile
diff --git a/Vagrantfile.centos7 b/Vagrantfile.centos7
deleted file mode 100644
index 49cb2779..00000000
--- a/Vagrantfile.centos7
+++ /dev/null
@@ -1,54 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-Vagrant.configure("2") do |config|
-  config.vm.box = "centos/7"
-  config.vm.provider :virtualbox do |v|
-    v.memory = 2048
-    v.cpus = 2
-  end
-  config.vm.provider :libvirt do |v|
-    v.memory = 2048
-    v.cpus = 2
-  end
-  config.vm.provision "shell", inline: <<-SHELL
-    # configuration
-    GO_VERSION="1.13.11"
-    BATS_VERSION="v1.2.0"
-
-    # install yum packages
-    yum install -y -q epel-release
-    yum install -y -q gcc git iptables jq libseccomp-devel make
-    yum clean all
-
-    # install Go
-    curl -fsSL "https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz" | tar Cxz /usr/local
-
-    # install bats
-    git clone https://github.com/bats-core/bats-core
-    cd bats-core
-    git checkout $BATS_VERSION
-    ./install.sh /usr/local
-
-    # NOTE: criu is NOT installed. criu tests are skipped.
-
-    # set PATH (NOTE: sudo without -i ignores this PATH)
-    cat >> /etc/profile.d/sh.local <<EOF
-PATH=/usr/local/go/bin:/usr/local/bin:$PATH
-export PATH
-EOF
-    source /etc/profile.d/sh.local
-
-    # sysctl
-    echo "user.max_user_namespaces=15076" > /etc/sysctl.d/userns.conf
-    sysctl --system
-
-    # Add a user for rootless tests
-    useradd -u2000 -m -d/home/rootless -s/bin/bash rootless
-
-    # Add busybox for libcontainer/integration tests
-    . /vagrant/tests/integration/multi-arch.bash \
-        && mkdir /busybox \
-        && curl -fsSL $(get_busybox) | tar xfJC - /busybox
-  SHELL
-end
diff --git a/libcontainer/cgroups/cgroups.go b/libcontainer/cgroups/cgroups.go
index 071f8f2e..2bb2d1e8 100644
--- a/libcontainer/cgroups/cgroups.go
+++ b/libcontainer/cgroups/cgroups.go
@@ -47,9 +47,6 @@ type Manager interface {
 
 	// GetFreezerState retrieves the current FreezerState of the cgroup.
 	GetFreezerState() (configs.FreezerState, error)
-
-	// Whether the cgroup path exists or not
-	Exists() bool
 }
 
 type NotFoundError struct {
diff --git a/libcontainer/cgroups/fs/apply_raw.go b/libcontainer/cgroups/fs/apply_raw.go
index 2093e0f8..b5bace36 100644
--- a/libcontainer/cgroups/fs/apply_raw.go
+++ b/libcontainer/cgroups/fs/apply_raw.go
@@ -264,7 +264,7 @@ func (m *manager) Set(container *configs.Config) error {
 // Freeze toggles the container's freezer cgroup depending on the state
 // provided
 func (m *manager) Freeze(state configs.FreezerState) (Err error) {
-	path := m.Path("freezer")
+	path := m.GetPaths()["freezer"]
 	if m.cgroups == nil || path == "" {
 		return errors.New("cannot toggle freezer: cgroups not configured for container")
 	}
@@ -382,7 +382,8 @@ func (m *manager) GetCgroups() (*configs.Cgroup, error) {
 }
 
 func (m *manager) GetFreezerState() (configs.FreezerState, error) {
-	dir := m.Path("freezer")
+	paths := m.GetPaths()
+	dir := paths["freezer"]
 	freezer, err := m.getSubsystems().Get("freezer")
 
 	// If the container doesn't have the freezer cgroup, say it's undefined.
@@ -391,7 +392,3 @@ func (m *manager) GetFreezerState() (configs.FreezerState, error) {
 	}
 	return freezer.(*FreezerGroup).GetState(dir)
 }
-
-func (m *manager) Exists() bool {
-	return cgroups.PathExists(m.Path("devices"))
-}
diff --git a/libcontainer/cgroups/fs/memory.go b/libcontainer/cgroups/fs/memory.go
index f7b8f999..95087690 100644
--- a/libcontainer/cgroups/fs/memory.go
+++ b/libcontainer/cgroups/fs/memory.go
@@ -291,9 +291,7 @@ func getPageUsageByNUMA(cgroupPath string) (cgroups.PageUsageByNUMA, error) {
 	stats := cgroups.PageUsageByNUMA{}
 
 	file, err := os.Open(path.Join(cgroupPath, cgroupMemoryPagesByNuma))
-	if os.IsNotExist(err) {
-		return stats, nil
-	} else if err != nil {
+	if err != nil {
 		return stats, err
 	}
 
diff --git a/libcontainer/cgroups/fs2/fs2.go b/libcontainer/cgroups/fs2/fs2.go
index 7be26211..69b9948a 100644
--- a/libcontainer/cgroups/fs2/fs2.go
+++ b/libcontainer/cgroups/fs2/fs2.go
@@ -262,7 +262,3 @@ func (m *manager) GetCgroups() (*configs.Cgroup, error) {
 func (m *manager) GetFreezerState() (configs.FreezerState, error) {
 	return getFreezer(m.dirPath)
 }
-
-func (m *manager) Exists() bool {
-	return cgroups.PathExists(m.dirPath)
-}
diff --git a/libcontainer/cgroups/systemd/common.go b/libcontainer/cgroups/systemd/common.go
index 5c00c0bb..cdf656b8 100644
--- a/libcontainer/cgroups/systemd/common.go
+++ b/libcontainer/cgroups/systemd/common.go
@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"math"
 	"os"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -21,6 +22,10 @@ var (
 	connOnce sync.Once
 	connDbus *systemdDbus.Conn
 	connErr  error
+
+	versionOnce sync.Once
+	version     int
+	versionErr  error
 )
 
 // NOTE: This function comes from package github.com/coreos/go-systemd/util
@@ -351,21 +356,56 @@ func stopUnit(dbusConnection *systemdDbus.Conn, unitName string) error {
 	return nil
 }
 
-func addCpuQuota(properties *[]systemdDbus.Property, quota int64, period uint64) {
+func systemdVersion(conn *systemdDbus.Conn) (int, error) {
+	versionOnce.Do(func() {
+		version = -1
+		verStr, err := conn.GetManagerProperty("Version")
+		if err != nil {
+			versionErr = err
+			return
+		}
+
+		// verStr is like "v245.4-1.fc32" (including quotes)
+		if !strings.HasPrefix(verStr, `"v`) {
+			versionErr = fmt.Errorf("can't parse version %s", verStr)
+			return
+		}
+		// remove `"v` prefix and everything after the first dot
+		ver, err := strconv.Atoi(strings.SplitN(verStr[2:], ".", 2)[0])
+		if err != nil {
+			versionErr = err
+		}
+		version = ver
+	})
+
+	return version, versionErr
+}
+
+func addCpuQuota(conn *systemdDbus.Conn, properties *[]systemdDbus.Property, quota int64, period uint64) {
+	if period != 0 {
+		// systemd only supports CPUQuotaPeriodUSec since v242
+		sdVer, err := systemdVersion(conn)
+		if err != nil {
+			logrus.Warnf("systemdVersion: %s", err)
+		} else if sdVer >= 242 {
+			*properties = append(*properties,
+				newProp("CPUQuotaPeriodUSec", period))
+		}
+	}
 	if quota != 0 || period != 0 {
 		// corresponds to USEC_INFINITY in systemd
 		cpuQuotaPerSecUSec := uint64(math.MaxUint64)
 		if quota > 0 {
-			// systemd converts CPUQuotaPerSecUSec (microseconds per CPU second) to CPUQuota
-			// (integer percentage of CPU) internally.  This means that if a fractional percent of
-			// CPU is indicated by Resources.CpuQuota, we need to round up to the nearest
-			// 10ms (1% of a second) such that child cgroups can set the cpu.cfs_quota_us they expect.
 			if period == 0 {
 				// assume the default kernel value of 100000 us (100 ms), same for v1 and v2.
 				// v1: https://www.kernel.org/doc/html/latest/scheduler/sched-bwc.html and
 				// v2: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
 				period = 100000
 			}
+			// systemd converts CPUQuotaPerSecUSec (microseconds per CPU second) to CPUQuota
+			// (integer percentage of CPU) internally.  This means that if a fractional percent of
+			// CPU is indicated by Resources.CpuQuota, we need to round up to the nearest
+			// 10ms (1% of a second) such that child cgroups can set the cpu.cfs_quota_us they expect.
 			cpuQuotaPerSecUSec = uint64(quota*1000000) / period
 			if cpuQuotaPerSecUSec%10000 != 0 {
 				cpuQuotaPerSecUSec = ((cpuQuotaPerSecUSec / 10000) + 1) * 10000
diff --git a/libcontainer/cgroups/systemd/unsupported.go b/libcontainer/cgroups/systemd/unsupported.go
index cb4d00c8..46d3beca 100644
--- a/libcontainer/cgroups/systemd/unsupported.go
+++ b/libcontainer/cgroups/systemd/unsupported.go
@@ -65,7 +65,3 @@ func Freeze(c *configs.Cgroup, state configs.FreezerState) error {
 func (m *Manager) GetCgroups() (*configs.Cgroup, error) {
 	return nil, errors.New("Systemd not supported")
 }
-
-func (m *Manager) Exists() bool {
-	return false
-}
diff --git a/libcontainer/cgroups/systemd/v1.go b/libcontainer/cgroups/systemd/v1.go
index d139a7df..c4d5e526 100644
--- a/libcontainer/cgroups/systemd/v1.go
+++ b/libcontainer/cgroups/systemd/v1.go
@@ -68,7 +68,7 @@ var legacySubsystems = subsystemSet{
 	&fs.NameGroup{GroupName: "name=systemd"},
 }
 
-func genV1ResourcesProperties(c *configs.Cgroup) ([]systemdDbus.Property, error) {
+func genV1ResourcesProperties(c *configs.Cgroup, conn *systemdDbus.Conn) ([]systemdDbus.Property, error) {
 	var properties []systemdDbus.Property
 	r := c.Resources
 
@@ -88,7 +88,7 @@ func genV1ResourcesProperties(c *configs.Cgroup) ([]systemdDbus.Property, error)
 			newProp("CPUShares", r.CpuShares))
 	}
 
-	addCpuQuota(&properties, r.CpuQuota, r.CpuPeriod)
+	addCpuQuota(conn, &properties, r.CpuQuota, r.CpuPeriod)
 
 	if r.BlkioWeight != 0 {
 		properties = append(properties,
@@ -112,8 +112,6 @@ func (m *legacyManager) Apply(pid int) error {
 		properties []systemdDbus.Property
 	)
 
-	m.mu.Lock()
-	defer m.mu.Unlock()
 	if c.Paths != nil {
 		paths := make(map[string]string)
 		for name, path := range c.Paths {
@@ -167,7 +165,11 @@ func (m *legacyManager) Apply(pid int) error {
 	properties = append(properties,
 		newProp("DefaultDependencies", false))
 
-	resourcesProperties, err := genV1ResourcesProperties(c)
+	dbusConnection, err := getDbusConnection(false)
+	if err != nil {
+		return err
+	}
+	resourcesProperties, err := genV1ResourcesProperties(c, dbusConnection)
 	if err != nil {
 		return err
 	}
@@ -182,10 +184,6 @@ func (m *legacyManager) Apply(pid int) error {
 		}
 	}
 
-	dbusConnection, err := getDbusConnection(false)
-	if err != nil {
-		return err
-	}
 	if err := startUnit(dbusConnection, unitName, properties); err != nil {
 		return err
 	}
@@ -370,7 +368,11 @@ func (m *legacyManager) Set(container *configs.Config) error {
 	if m.cgroups.Paths != nil {
 		return nil
 	}
-	properties, err := genV1ResourcesProperties(container.Cgroups)
+	dbusConnection, err := getDbusConnection(false)
+	if err != nil {
+		return err
+	}
+	properties, err := genV1ResourcesProperties(container.Cgroups, dbusConnection)
 	if err != nil {
 		return err
 	}
@@ -395,11 +397,6 @@ func (m *legacyManager) Set(container *configs.Config) error {
 		logrus.Infof("freeze container before SetUnitProperties failed: %v", err)
 	}
 
-	dbusConnection, err := getDbusConnection(false)
-	if err != nil {
-		_ = m.Freeze(targetFreezerState)
-		return err
-	}
 	if err := dbusConnection.SetUnitProperties(getUnitName(container.Cgroups), true, properties...); err != nil {
 		_ = m.Freeze(targetFreezerState)
 		return err
@@ -465,7 +462,3 @@ func (m *legacyManager) GetFreezerState() (configs.FreezerState, error) {
 	}
 	return freezer.(*fs.FreezerGroup).GetState(path)
 }
-
-func (m *legacyManager) Exists() bool {
-	return cgroups.PathExists(m.Path("devices"))
-}
diff --git a/libcontainer/cgroups/systemd/v2.go b/libcontainer/cgroups/systemd/v2.go
index 333747c4..c44d7734 100644
--- a/libcontainer/cgroups/systemd/v2.go
+++ b/libcontainer/cgroups/systemd/v2.go
@@ -34,7 +34,7 @@ func NewUnifiedManager(config *configs.Cgroup, path string, rootless bool) cgrou
 	}
 }
 
-func genV2ResourcesProperties(c *configs.Cgroup) ([]systemdDbus.Property, error) {
+func genV2ResourcesProperties(c *configs.Cgroup, conn *systemdDbus.Conn) ([]systemdDbus.Property, error) {
 	var properties []systemdDbus.Property
 	r := c.Resources
 
@@ -72,7 +72,7 @@ func genV2ResourcesProperties(c *configs.Cgroup) ([]systemdDbus.Property, error)
 			newProp("CPUWeight", r.CpuWeight))
 	}
 
-	addCpuQuota(&properties, r.CpuQuota, r.CpuPeriod)
+	addCpuQuota(conn, &properties, r.CpuQuota, r.CpuPeriod)
 
 	if r.PidsLimit > 0 || r.PidsLimit == -1 {
 		properties = append(properties,
@@ -136,17 +136,17 @@ func (m *unifiedManager) Apply(pid int) error {
 	properties = append(properties,
 		newProp("DefaultDependencies", false))
 
-	resourcesProperties, err := genV2ResourcesProperties(c)
+	dbusConnection, err := getDbusConnection(m.rootless)
 	if err != nil {
 		return err
 	}
-	properties = append(properties, resourcesProperties...)
-	properties = append(properties, c.SystemdProps...)
-
-	dbusConnection, err := getDbusConnection(m.rootless)
+	resourcesProperties, err := genV2ResourcesProperties(c, dbusConnection)
 	if err != nil {
 		return err
 	}
+	properties = append(properties, resourcesProperties...)
+	properties = append(properties, c.SystemdProps...)
+
 	if err := startUnit(dbusConnection, unitName, properties); err != nil {
 		return errors.Wrapf(err, "error while starting unit %q with properties %+v", unitName, properties)
 	}
@@ -289,7 +289,11 @@ func (m *unifiedManager) GetStats() (*cgroups.Stats, error) {
 }
 
 func (m *unifiedManager) Set(container *configs.Config) error {
-	properties, err := genV2ResourcesProperties(m.cgroups)
+	dbusConnection, err := getDbusConnection(m.rootless)
+	if err != nil {
+		return err
+	}
+	properties, err := genV2ResourcesProperties(m.cgroups, dbusConnection)
 	if err != nil {
 		return err
 	}
@@ -314,11 +318,6 @@ func (m *unifiedManager) Set(container *configs.Config) error {
 		logrus.Infof("freeze container before SetUnitProperties failed: %v", err)
 	}
 
-	dbusConnection, err := getDbusConnection(m.rootless)
-	if err != nil {
-		_ = m.Freeze(targetFreezerState)
-		return err
-	}
 	if err := dbusConnection.SetUnitProperties(getUnitName(m.cgroups), true, properties...); err != nil {
 		_ = m.Freeze(targetFreezerState)
 		return errors.Wrap(err, "error while setting unit properties")
@@ -352,7 +351,3 @@ func (m *unifiedManager) GetFreezerState() (configs.FreezerState, error) {
 	}
 	return fsMgr.GetFreezerState()
 }
-
-func (m *unifiedManager) Exists() bool {
-	return cgroups.PathExists(m.path)
-}
diff --git a/libcontainer/container_linux.go b/libcontainer/container_linux.go
index 2c7420ef..92bba2a7 100644
--- a/libcontainer/container_linux.go
+++ b/libcontainer/container_linux.go
@@ -169,17 +169,7 @@ func (c *linuxContainer) OCIState() (*specs.State, error) {
 }
 
 func (c *linuxContainer) Processes() ([]int, error) {
-	var pids []int
-	status, err := c.currentStatus()
-	if err != nil {
-		return pids, err
-	}
-	// for systemd cgroup, the unit's cgroup path will be auto removed if container's all processes exited
-	if status == Stopped && !c.cgroupManager.Exists() {
-		return pids, nil
-	}
-
-	pids, err = c.cgroupManager.GetAllPids()
+	pids, err := c.cgroupManager.GetAllPids()
 	if err != nil {
 		return nil, newSystemErrorWithCause(err, "getting all container pids from cgroups")
 	}
@@ -395,17 +385,13 @@ func (c *linuxContainer) start(process *Process) error {
 func (c *linuxContainer) Signal(s os.Signal, all bool) error {
 	c.m.Lock()
 	defer c.m.Unlock()
+	if all {
+		return signalAllProcesses(c.cgroupManager, s)
+	}
 	status, err := c.currentStatus()
 	if err != nil {
 		return err
 	}
-	if all {
-		// for systemd cgroup, the unit's cgroup path will be auto removed if container's all processes exited
-		if status == Stopped && !c.cgroupManager.Exists() {
-			return nil
-		}
-		return signalAllProcesses(c.cgroupManager, s)
-	}
 	// to avoid a PID reuse attack
 	if status == Running || status == Created || status == Paused {
 		if err := c.initProcess.signal(s); err != nil {
@@ -988,11 +974,7 @@ func (c *linuxContainer) Checkpoint(criuOpts *CriuOpts) error {
 				return fmt.Errorf("invalid --status-fd argument %d: not writable", fd)
 			}
 
-			if c.checkCriuVersion(31500) != nil {
-				// For criu 3.15+, use notifications (see case "status-ready"
-				// in criuNotifications). Otherwise, rely on criu status fd.
-				rpcOpts.StatusFd = proto.Int32(int32(fd))
-			}
+			rpcOpts.StatusFd = proto.Int32(int32(fd))
 		}
 	}
 
@@ -1597,24 +1579,23 @@ func (c *linuxContainer) criuNotifications(resp *criurpc.CriuResp, process *Proc
 	if notify == nil {
 		return fmt.Errorf("invalid response: %s", resp.String())
 	}
-	script := notify.GetScript()
-	logrus.Debugf("notify: %s\n", script)
-	switch script {
-	case "post-dump":
+	logrus.Debugf("notify: %s\n", notify.GetScript())
+	switch {
+	case notify.GetScript() == "post-dump":
 		f, err := os.Create(filepath.Join(c.root, "checkpoint"))
 		if err != nil {
 			return err
 		}
 		f.Close()
-	case "network-unlock":
+	case notify.GetScript() == "network-unlock":
 		if err := unlockNetwork(c.config); err != nil {
 			return err
 		}
-	case "network-lock":
+	case notify.GetScript() == "network-lock":
 		if err := lockNetwork(c.config); err != nil {
 			return err
 		}
-	case "setup-namespaces":
+	case notify.GetScript() == "setup-namespaces":
 		if c.config.Hooks != nil {
 			s, err := c.currentOCIState()
 			if err != nil {
@@ -1627,7 +1608,7 @@ func (c *linuxContainer) criuNotifications(resp *criurpc.CriuResp, process *Proc
 				}
 			}
 		}
-	case "post-restore":
+	case notify.GetScript() == "post-restore":
 		pid := notify.GetPid()
 
 		p, err := os.FindProcess(int(pid))
@@ -1657,7 +1638,7 @@ func (c *linuxContainer) criuNotifications(resp *criurpc.CriuResp, process *Proc
 				logrus.Error(err)
 			}
 		}
-	case "orphan-pts-master":
+	case notify.GetScript() == "orphan-pts-master":
 		scm, err := unix.ParseSocketControlMessage(oob)
 		if err != nil {
 			return err
@@ -1674,16 +1655,6 @@ func (c *linuxContainer) criuNotifications(resp *criurpc.CriuResp, process *Proc
 		if err := utils.SendFd(process.ConsoleSocket, master.Name(), master.Fd()); err != nil {
 			return err
 		}
-	case "status-ready":
-		if opts.StatusFd != -1 {
-			// write \0 to status fd to notify that lazy page server is ready
-			_, err := unix.Write(opts.StatusFd, []byte{0})
-			if err != nil {
-				logrus.Warnf("can't write \\0 to status fd: %v", err)
-			}
-			_ = unix.Close(opts.StatusFd)
-			opts.StatusFd = -1
-		}
 	}
 	return nil
 }
diff --git a/script/install-vagrant.sh b/script/install-vagrant.sh
deleted file mode 100755
index ed3b60ba..00000000
--- a/script/install-vagrant.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/bash
-set -eux -o pipefail
-VAGRANT_VERSION="2.2.7"
-
-# https://github.com/alvistack/ansible-role-virtualbox/blob/6887b020b0ca5c59ddb6620d73f053ffb84f4126/.travis.yml#L30
-apt-get update
-apt-get install -q -y bridge-utils dnsmasq-base ebtables libvirt-bin libvirt-dev qemu-kvm qemu-utils ruby-dev
-wget https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}_$(uname -m).deb
-dpkg -i vagrant_${VAGRANT_VERSION}_$(uname -m).deb
-rm -f vagrant_${VAGRANT_VERSION}_$(uname -m).deb
-vagrant plugin install vagrant-libvirt


Test patch:
diff --git a/libcontainer/cgroups/fs/memory_test.go b/libcontainer/cgroups/fs/memory_test.go
index 52244fc7..c84b4fe3 100644
--- a/libcontainer/cgroups/fs/memory_test.go
+++ b/libcontainer/cgroups/fs/memory_test.go
@@ -504,14 +504,3 @@ func TestNoHierarchicalNumaStat(t *testing.T) {
 	}
 	expectPageUsageByNUMAEquals(t, pageUsageByNUMA, actualStats)
 }
-
-func TestWithoutNumaStat(t *testing.T) {
-	helper := NewCgroupTestUtil("memory", t)
-	defer helper.cleanup()
-
-	actualStats, err := getPageUsageByNUMA(helper.CgroupPath)
-	if err != nil {
-		t.Fatal(err)
-	}
-	expectPageUsageByNUMAEquals(t, cgroups.PageUsageByNUMA{}, actualStats)
-}
diff --git a/libcontainer/container_linux_test.go b/libcontainer/container_linux_test.go
index 72ca1ef7..72f0835c 100644
--- a/libcontainer/container_linux_test.go
+++ b/libcontainer/container_linux_test.go
@@ -50,11 +50,6 @@ func (m *mockCgroupManager) Destroy() error {
 	return nil
 }
 
-func (m *mockCgroupManager) Exists() bool {
-	_, err := os.Lstat(m.Path("devices"))
-	return err == nil
-}
-
 func (m *mockCgroupManager) GetPaths() map[string]string {
 	return m.paths
 }
@@ -139,27 +134,11 @@ func (m *mockProcess) forwardChildLogs() {
 }
 
 func TestGetContainerPids(t *testing.T) {
-	pid := 1
-	stat, err := system.Stat(pid)
-	if err != nil {
-		t.Fatalf("can't stat pid %d, got %v", pid, err)
-	}
 	container := &linuxContainer{
-		id:     "myid",
-		config: &configs.Config{},
-		cgroupManager: &mockCgroupManager{
-			allPids: []int{1, 2, 3},
-			paths: map[string]string{
-				"device": "/proc/self/cgroups",
-			},
-		},
-		initProcess: &mockProcess{
-			_pid:    1,
-			started: 10,
-		},
-		initProcessStartTime: stat.StartTime,
+		id:            "myid",
+		config:        &configs.Config{},
+		cgroupManager: &mockCgroupManager{allPids: []int{1, 2, 3}},
 	}
-	container.state = &runningState{c: container}
 	pids, err := container.Processes()
 	if err != nil {
 		t.Fatal(err)
diff --git a/libcontainer/integration/checkpoint_test.go b/libcontainer/integration/checkpoint_test.go
index 28fdbdb0..ad34fb36 100644
--- a/libcontainer/integration/checkpoint_test.go
+++ b/libcontainer/integration/checkpoint_test.go
@@ -60,10 +60,6 @@ func testCheckpoint(t *testing.T, userns bool) {
 		return
 	}
 
-	if _, err := exec.LookPath("criu"); err != nil {
-		t.Skipf("criu binary not found: %v", err)
-	}
-
 	root, err := newTestRoot()
 	if err != nil {
 		t.Fatal(err)
diff --git a/tests/integration/helpers.bash b/tests/integration/helpers.bash
index 5db99320..7cf3580a 100644
--- a/tests/integration/helpers.bash
+++ b/tests/integration/helpers.bash
@@ -105,6 +105,16 @@ function runc_rootless_cgroup() {
 	update_config '.linux.resources += {"memory":{},"cpu":{},"blockio":{},"pids":{}}' $bundle
 }
 
+# Returns systemd version as a number (-1 if systemd is not enabled/supported).
+function systemd_version() {
+	if [ -n "${RUNC_USE_SYSTEMD}" ]; then
+		systemctl --version | awk '/^systemd / {print $2; exit}'
+		return
+	fi
+
+	echo "-1"
+}
+
 function init_cgroup_paths() {
 	# init once
 	test -n "$CGROUP_UNIFIED" && return
@@ -180,15 +190,16 @@ function check_cgroup_value() {
 # Helper to check a value in systemd.
 function check_systemd_value() {
 	[ -z "${RUNC_USE_SYSTEMD}" ] && return
-	source=$1
+	local source=$1
 	[ "$source" = "unsupported" ] && return
-	expected="$2"
-	user=""
+	local expected="$2"
+	local expected2="$3"
+	local user=""
 	[ $(id -u) != "0" ] && user="--user"
 
 	current=$(systemctl show $user --property $source $SD_UNIT_NAME | awk -F= '{print $2}')
-	echo "systemd $source: current $current !? $expected"
-	[ "$current" = "$expected" ]
+	echo "systemd $source: current $current !? $expected $expected2"
+	[ "$current" = "$expected" ] || [ -n "$expected2" -a "$current" = "$expected2" ]
 }
 
 # Helper function to set a resources limit
diff --git a/tests/integration/kill.bats b/tests/integration/kill.bats
index ef8de3b2..d9afe927 100644
--- a/tests/integration/kill.bats
+++ b/tests/integration/kill.bats
@@ -25,10 +25,6 @@ function teardown() {
 
   retry 10 1 eval "__runc state test_busybox | grep -q 'stopped'"
 
-  # we should ensure kill work after the container stopped
-  runc kill -a test_busybox 0
-  [ "$status" -eq 0 ]
-
   runc delete test_busybox
   [ "$status" -eq 0 ]
 }
diff --git a/tests/integration/ps.bats b/tests/integration/ps.bats
index e8ea5532..646b5ab7 100644
--- a/tests/integration/ps.bats
+++ b/tests/integration/ps.bats
@@ -60,27 +60,3 @@ function teardown() {
   [[ ${lines[0]} =~ \ +PID\ +TTY\ +STAT\ +TIME\ +COMMAND+ ]]
   [[ "${lines[1]}" =~ [0-9]+ ]]
 }
-
-@test "ps after the container stopped" {
-  # ps requires cgroups
-  [[ "$ROOTLESS" -ne 0 ]] && requires rootless_cgroup
-  set_cgroups_path "$BUSYBOX_BUNDLE"
-
-  # start busybox detached
-  runc run -d --console-socket $CONSOLE_SOCKET test_busybox
-  [ "$status" -eq 0 ]
-
-  # check state
-  testcontainer test_busybox running
-
-  runc ps test_busybox
-  [ "$status" -eq 0 ]
-
-  runc kill test_busybox KILL
-  [ "$status" -eq 0 ]
-
-  retry 10 1 eval "__runc state test_busybox | grep -q 'stopped'"
-
-  runc ps test_busybox
-  [ "$status" -eq 0 ]
-}
diff --git a/tests/integration/spec.bats b/tests/integration/spec.bats
index 1aa38815..d924007d 100644
--- a/tests/integration/spec.bats
+++ b/tests/integration/spec.bats
@@ -79,7 +79,7 @@ function teardown() {
 
   SPEC_REF=$([[ -z "$SPEC_COMMIT" ]] && echo $SPEC_VERSION || echo $SPEC_COMMIT)
 
-  run bash -c "cd src/runtime-spec && git reset --hard ${SPEC_REF}"
+  run git -C src/runtime-spec reset --hard "${SPEC_REF}"
 
   [ "$status" -eq 0 ]
   [ -e src/runtime-spec/schema/config-schema.json ]
@@ -87,7 +87,7 @@ function teardown() {
   run bash -c "GOPATH='$GOPATH' go get github.com/xeipuuv/gojsonschema"
   [ "$status" -eq 0 ]
 
-  run bash -c "cd ${GOPATH}/src/github.com/xeipuuv/gojsonschema && git reset --hard 6637feb73ee44cd4640bb3def285c29774234c7f"
+  run git -C "${GOPATH}/src/github.com/xeipuuv/gojsonschema" reset --hard 6637feb73ee44cd4640bb3def285c29774234c7f
   [ "$status" -eq 0 ]
 
   GOPATH="$GOPATH" go build src/runtime-spec/schema/validate.go
diff --git a/tests/integration/update.bats b/tests/integration/update.bats
index ed80f51b..72902f4a 100644
--- a/tests/integration/update.bats
+++ b/tests/integration/update.bats
@@ -67,10 +67,6 @@ function setup() {
         ;;
     esac
     SD_UNLIMITED="infinity"
-    SD_VERSION=$(systemctl --version | awk '{print $2; exit}')
-    if [ $SD_VERSION -lt 227 ]; then
-        SD_UNLIMITED="18446744073709551615"
-    fi
 
     # check that initial values were properly set
     check_cgroup_value "cpuset.cpus" 0
@@ -254,8 +250,18 @@ function check_cpu_quota() {
         check_cgroup_value "cpu.cfs_quota_us" $quota
         check_cgroup_value "cpu.cfs_period_us" $period
     fi
-    # systemd value is the same for v1 and v2
+    # systemd values are the same for v1 and v2
     check_systemd_value "CPUQuotaPerSecUSec" $sd_quota
+
+    # CPUQuotaPeriodUSec requires systemd >= v242
+    [ "$(systemd_version)" -lt 242 ] && return
+
+    local sd_period=$(( period/1000 ))ms
+    [ "$sd_period" = "1000ms" ] && sd_period="1s"
+    local sd_infinity=""
+    # 100ms is the default value, and if not set, shown as infinity
+    [ "$sd_period" = "100ms" ] && sd_infinity="infinity"
+    check_systemd_value "CPUQuotaPeriodUSec" $sd_period $sd_infinity
 }
 
 function check_cpu_shares() {
diff --git a/tests/rootless.sh b/tests/rootless.sh
index c9c79383..19ec43e1 100755
--- a/tests/rootless.sh
+++ b/tests/rootless.sh
@@ -146,7 +146,7 @@ do
 		# https://bugzilla.redhat.com/show_bug.cgi?id=1788616
 	        ssh -t -t -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i $HOME/rootless.key rootless@localhost -- PATH="$PATH" RUNC_USE_SYSTEMD="$RUNC_USE_SYSTEMD" bats -t "$ROOT/tests/integration$ROOTLESS_TESTPATH"
 	else
-		sudo -HE -u rootless PATH="$PATH" $(which bats) -t "$ROOT/tests/integration$ROOTLESS_TESTPATH"
+		sudo -HE -u rootless PATH="$PATH" bats -t "$ROOT/tests/integration$ROOTLESS_TESTPATH"
 	fi
 	set +e
 done