Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Setting a Resource requests and limits on a Pod other Pods running on the node where it was scheduled are moved to the cgroups created to serve the QoS request. What you expected to happen: Pods to stay in their assigned cgroups. How to reproduce it (as minimally and precisely as possible): kubelet running with the following flags: Create a pod without resource reservations and then create another one with resource reservations. For example, using the following template: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig node
This is the memory cgroup tree layout before starting an pod with resource request and limits. This is after the pod is created. Notice that is moved to the cgroup where nginx is running in
Interesting, since is running as a static pod
/milestone v1.12
this looks off... can you provide the output for the kube-proxy yaml file? what container runtime is being used? can you share the kubelet configuration?
We need to figure out if this is a regression in 1.12. Can you take a look?
I can reproduce it on GCE:
Node with same flags, except , instead of `--cgroup-root="/dashpole/" So the problem only occurs when is specified
I can reproduce this on 1.11.3 as well. I don't think this is a regression with the 1.12 release, although it is still a serious bug.
Correct, was being specified in the case above. As mentioned over Slack, we were able to track it back to 1.11.0-beta (can't recall the beta version, though) but not 1.11.0-alpha.2. The only PR we've identified which could have an impact is .
I will try to take a look today. I am curious the motivation for changing the cgroup-root, in retrospect, would have preferred we didnt expose it.
I am unable to reproduce this using systemd cgroup driver when using Given this exists currently in 1.11, we should fix this, but it should not block 1.12 milestone.
Thanks /priority important-soon
can you help to reproduce? is OOO until Monday.
i just ran out of time to validate this with alternate driver (had assumed it would have happened with both), but it appears localized to one.
Nevertheless it's a good datapoint, it means it's only affecting cgroupfs driver. Regarding your question, the cgroup-root flag was the cleanest way we could find to have the kubelet respect the Mesos cgroup setup. Lmk if I can help you with anything.
ping , is this on track for 1.13? any related PRs?
I haven't looked deeper into this yet. Ill keep this on my list for 1.13
any porgress on this ? I don't see any associated PRs linked here. We are nearing code slush for 1.13 next Friday 11/9 and Code freeze in 2 weeks. If you think this issue isn't critical for 1.13 and work might extend beyond code slush please work with Niko Pen to move this out. thanks
Ill pick up this work over the next few days
I have narrowed it down to this strange behavior in golang: Basically, appending to a list constructed with has different behavior than appending to a list constructed literally, but only with lists where the size is larger than 1... This strange behavior with golang slices results in the "best-effort" qos cgroup in the qosContainerManager being changed to a pod's cgroup.
A simpler reproduction case:

Discussion:
panic(fmt.Errorf("invalid character in component [%q] of CgroupName", component)) } } <del> return CgroupName(append(base, components...)) </del> <ins> // copy data from the base cgroup to eliminate cases where CgroupNames share underlying slices.  See #68416 baseCopy := make([]string, len(base)) copy(baseCopy, base) return CgroupName(append(baseCopy, components...)) </ins> } func escapeSystemdCgroupName(part string) string {

import ( "path" <ins> "reflect" </ins> "testing" ) <ins> // TestNewCgroupName tests confirms that #68416 is fixed func TestNewCgroupName(t *testing.T) { a := ParseCgroupfsToCgroupName("/a/") ab := NewCgroupName(a, "b") expectedAB := CgroupName([]string{"a", "", "b"}) if !reflect.DeepEqual(ab, expectedAB) { t.Errorf("Expected %d%+v; got %d%+v", len(expectedAB), expectedAB, len(ab), ab) } abc := NewCgroupName(ab, "c") expectedABC := CgroupName([]string{"a", "", "b", "c"}) if !reflect.DeepEqual(abc, expectedABC) { t.Errorf("Expected %d%+v; got %d%+v", len(expectedABC), expectedABC, len(abc), abc) } _ = NewCgroupName(ab, "d") if !reflect.DeepEqual(abc, expectedABC) { t.Errorf("Expected %d%+v; got %d%+v", len(expectedABC), expectedABC, len(abc), abc) } } </ins> func TestCgroupNameToSystemdBasename(t *testing.T) { testCases := []struct { input    CgroupName