Code change:
Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly. If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the . I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information? The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation

Related discussion:
text: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package windows import ( "crypto/tls" "encoding/json" "fmt" "io/ioutil" "net/http" "regexp" "strconv" "time" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/client-go/kubernetes/scheme" kubeletconfigv1beta1 "k8s.io/kubelet/config/v1beta1" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" "github.com/onsi/gomega" ) var _ = SIGDescribe("[Feature:Windows] Memory Limits [Serial] [Slow]", func() { f := framework.NewDefaultFramework("memory-limit-test-windows") ginkgo.BeforeEach(func() { // NOTE(vyta): these tests are Windows specific framework.SkipUnlessNodeOSDistroIs("windows") }) ginkgo.Context("Allocatable node memory", func() { ginkgo.It("should be equal to a calculated allocatable memory value", func() { checkNodeAllocatableTest(f) }) }) ginkgo.Context("attempt to deploy past allocatable memory limits", func() { ginkgo.It("should fail deployments of pods once there isn't enough memory", func() { overrideAllocatableMemoryTest(f, 4) }) }) }) type nodeMemory struct { // capacity capacity resource.Quantity // allocatable memory allocatable resource.Quantity // memory reserved for OS level processes systemReserve resource.Quantity // memory reserved for kubelet (not implemented) kubeReserve resource.Quantity // grace period memory limit (not implemented) softEviction resource.Quantity // no grace period memory limit hardEviction resource.Quantity } // runDensityBatchTest runs the density batch pod creation test // checks that a calculated value for NodeAllocatable is equal to the reported value func checkNodeAllocatableTest(f *framework.Framework) { nodeMem := getNodeMemory(f) framework.Logf("nodeMem says: %+v", nodeMem) // calculate the allocatable mem based on capacity - reserved amounts calculatedNodeAlloc := nodeMem.capacity.Copy() calculatedNodeAlloc.Sub(nodeMem.systemReserve) calculatedNodeAlloc.Sub(nodeMem.kubeReserve) calculatedNodeAlloc.Sub(nodeMem.softEviction) calculatedNodeAlloc.Sub(nodeMem.hardEviction) ginkgo.By(fmt.Sprintf("Checking stated allocatable memory %v against calculated allocatable memory %v", &nodeMem.allocatable, calculatedNodeAlloc)) // sanity check against stated allocatable gomega.Expect(calculatedNodeAlloc.Cmp(nodeMem.allocatable)).To(gomega.Equal(0)) } // Deploys `allocatablePods + 1` pods, each with a memory limit of `1/allocatablePods` of the total allocatable // memory, then confirms that the last pod failed because of failedScheduling func overrideAllocatableMemoryTest(f *framework.Framework, allocatablePods int) { const ( podType = "memory_limit_test_pod" ) totalAllocatable := getTotalAllocatableMemory(f) memValue := totalAllocatable.Value() memPerPod := memValue / int64(allocatablePods) ginkgo.By(fmt.Sprintf("Deploying %d pods with mem limit %v, then one additional pod", allocatablePods, memPerPod)) // these should all work pods := newMemLimitTestPods(allocatablePods, imageutils.GetPauseImageName(), podType, strconv.FormatInt(memPerPod, 10)) f.PodClient().CreateBatch(pods) failurePods := newMemLimitTestPods(1, imageutils.GetPauseImageName(), podType, strconv.FormatInt(memPerPod, 10)) f.PodClient().Create(failurePods[0]) gomega.Eventually(func() bool { eventList, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).List(metav1.ListOptions{}) framework.ExpectNoError(err) for _, e := range eventList.Items { // Look for an event that shows FailedScheduling if e.Type == "Warning" && e.Reason == "FailedScheduling" && e.InvolvedObject.Name == failurePods[0].ObjectMeta.Name { framework.Logf("Found %+v event with message %+v", e.Reason, e.Message) return true } } return false }, 3*time.Minute, 10*time.Second).Should(gomega.Equal(true)) } // newMemLimitTestPods creates a list of pods (specification) for test. func newMemLimitTestPods(numPods int, imageName, podType string, memoryLimit string) []*v1.Pod { var pods []*v1.Pod memLimitQuantity, err := resource.ParseQuantity(memoryLimit) framework.ExpectNoError(err) for i := 0; i < numPods; i++ { podName := "test-" + string(uuid.NewUUID()) pod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, Labels: map[string]string{ "type": podType, "name": podName, }, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceMemory: memLimitQuantity, }, }, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, }, } pods = append(pods, &pod) } return pods } // getNodeMemory populates a nodeMemory struct with information from the first func getNodeMemory(f *framework.Framework) nodeMemory { selector := labels.Set{"beta.kubernetes.io/os": "windows"}.AsSelector() nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{ LabelSelector: selector.String(), }) framework.ExpectNoError(err) // Assuming that agent nodes have the same config // Make sure there is >0 agent nodes, then use the first one for info gomega.Expect(nodeList.Size()).NotTo(gomega.Equal(0)) ginkgo.By("Getting memory details from node status and kubelet config") status := nodeList.Items[0].Status nodeName := nodeList.Items[0].ObjectMeta.Name kubeletConfig, err := getCurrentKubeletConfig(nodeName) framework.ExpectNoError(err) systemReserve, err := resource.ParseQuantity(kubeletConfig.SystemReserved["memory"]) if err != nil { systemReserve = *resource.NewQuantity(0, resource.BinarySI) } kubeReserve, err := resource.ParseQuantity(kubeletConfig.KubeReserved["memory"]) if err != nil { kubeReserve = *resource.NewQuantity(0, resource.BinarySI) } hardEviction, err := resource.ParseQuantity(kubeletConfig.EvictionHard["memory.available"]) if err != nil { hardEviction = *resource.NewQuantity(0, resource.BinarySI) } softEviction, err := resource.ParseQuantity(kubeletConfig.EvictionSoft["memory.available"]) if err != nil { softEviction = *resource.NewQuantity(0, resource.BinarySI) } nodeMem := nodeMemory{ capacity:      status.Capacity[v1.ResourceMemory], allocatable:   status.Allocatable[v1.ResourceMemory], systemReserve: systemReserve, hardEviction:  hardEviction, // these are not implemented and are here for future use - will always be 0 at the moment kubeReserve:  kubeReserve, softEviction: softEviction, } return nodeMem } // getTotalAllocatableMemory gets the sum of all agent node's allocatable memory func getTotalAllocatableMemory(f *framework.Framework) *resource.Quantity { selector := labels.Set{"beta.kubernetes.io/os": "windows"}.AsSelector() nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{ LabelSelector: selector.String(), }) framework.ExpectNoError(err) ginkgo.By("Summing allocatable memory across all agent nodes") totalAllocatable := resource.NewQuantity(0, resource.BinarySI) for _, node := range nodeList.Items { status := node.Status totalAllocatable.Add(status.Allocatable[v1.ResourceMemory]) } return totalAllocatable } // getCurrentKubeletConfig modified from test/e2e_node/util.go func getCurrentKubeletConfig(nodeName string) (*kubeletconfig.KubeletConfiguration, error) { resp := pollConfigz(5*time.Minute, 5*time.Second, nodeName) kubeCfg, err := decodeConfigz(resp) if err != nil { return nil, err } return kubeCfg, nil } // Causes the test to fail, or returns a status 200 response from the /configz endpoint func pollConfigz(timeout time.Duration, pollInterval time.Duration, nodeName string) *http.Response { // start local proxy, so we can send graceful deletion over query string, rather than body parameter ginkgo.By("Opening proxy to cluster") cmd := framework.KubectlCmd("proxy", "-p", "0") stdout, stderr, err := framework.StartCmdAndStreamOutput(cmd) framework.ExpectNoError(err) defer stdout.Close() defer stderr.Close() defer framework.TryKill(cmd) buf := make([]byte, 128) var n int n, err = stdout.Read(buf) framework.ExpectNoError(err) output := string(buf[:n]) proxyRegexp := regexp.MustCompile("Starting to serve on 127.0.0.1:([0-9]+)") match := proxyRegexp.FindStringSubmatch(output) gomega.Expect(len(match)).To(gomega.Equal(2)) port, err := strconv.Atoi(match[1]) framework.ExpectNoError(err) ginkgo.By("http requesting node kubelet /configz") endpoint := fmt.Sprintf("http://127.0.0.1:%d/api/v1/nodes/%s/proxy/configz", port, nodeName) tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, } client := &http.Client{Transport: tr} req, err := http.NewRequest("GET", endpoint, nil) framework.ExpectNoError(err) req.Header.Add("Accept", "application/json") var resp *http.Response gomega.Eventually(func() bool { resp, err = client.Do(req) if err != nil { framework.Logf("Failed to get /configz, retrying. Error: %v", err) return false } if resp.StatusCode != 200 { framework.Logf("/configz response status not 200, retrying. Response was: %+v", resp) return false } return true }, timeout, pollInterval).Should(gomega.Equal(true)) return resp } // Decodes the http response from /configz and returns a kubeletconfig.KubeletConfiguration (internal type). func decodeConfigz(resp *http.Response) (*kubeletconfig.KubeletConfiguration, error) { // This hack because /configz reports the following structure: // {"kubeletconfig": {the JSON representation of kubeletconfigv1beta1.KubeletConfiguration}} type configzWrapper struct { ComponentConfig kubeletconfigv1beta1.KubeletConfiguration `json:"kubeletconfig"` } configz := configzWrapper{} kubeCfg := kubeletconfig.KubeletConfiguration{} contentsBytes, err := ioutil.ReadAll(resp.Body) if err != nil { return nil, err } err = json.Unmarshal(contentsBytes, &configz) if err != nil { return nil, err } err = scheme.Scheme.Convert(&configz.ComponentConfig, &kubeCfg, nil) if err != nil { return nil, err } return &kubeCfg, nil } </ins>