Change:
Testgrid: https://k8s- https://k8s- Example: https://k8s- This seems to be related to /sig scheduling /kind failing-test /priority important-soon /cc
/assign RaviSantosh Gudimetla
GitHub didn't allow me to assign the following users: RaviSantosh, Gudimetla. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign RaviSantosh Gudimetla Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looked at the kube-scheduler logs for GCE, seems the feature_gate was enabled. I don't see enabled, will setup the cluster locally to test if this is the issue.
have you been able to look into this? These tests are broken right now for unrelated reasons, but once they get fixed I expect to keep seeing the scheduler failure.
/milestone v1.13
is the one open PR expected to fix the remaining failures we are seeing the release dashboards? If not fixed by this week, these failures will block Beta cut targeted for 11/6.
- Apologies for the delay. Yes, this PR should fix the failure, if not, I will disable this test for now and revert the feature promotion PR associated with this test. Also, I need to rethink, if e2e is an appropriate test scenario to handle this feature or should unit tests/integration suffice. I could not find a test which does something like resizing a node in the existing e2e test suite.
I'm looking at this, and I'm not sure this is going to fix the failures. I'm looking at the tests, and it seems to be setting the node status capacity/allocatable to pretend to be a 12 core machine, and then it tries to allocate a pod on every node such that every node is 50% utilized. It thus tries to allocate a pod of ~ 6 cores on the resized node. But that node still only has 2 cores, so it fails to schedule: I think that's why the test has been failing, at least for the past few runs which I checked. What I have yet to figure out is how this passed here...
is it safe to rephrase this as a pure test design issue then?
the test still failed on the last . Even though it was a couple of hours after PR merged, I dont see it in the delta changes . I will watch for the next run result. Assuming it still doesnt pass (as pointed out about) we need to figure out what this e2e test failure means and should it block 1.13 Beta0 for next Tuesday. Considering the test did pass very few times since its introduction, this seems like an issue with the test design itself and not the feature. Do you have other tests (unit or integration or different approach to e2e) that can infirm us on the health of thew feature? If those exist and are green we can unblock Beta on those results while you determine the right way to author the e2e. Barring those being present right idea, I think we will have to err on the side of caution and roll back the feature itself before Beta. Either way we need to take a call and submit relevant PRs by tomorrow so we get green CI over the weekend, in time for Beta cut on Tuesday.
This is not a blocker, as I told earlier as well, e2e may not be good avenue for this test. I actually thought more about this problem last day. The problem is I updated the node capacity and I was checking if the node could handle the pod based on the scheduler cache which we get from kubelet. But kubelet on the other side is updating it's node information(like CPU, memory etc) in the API server there by causing a race condition. So, it could pass with a very little chance. So, I will remove this test from e2e. Yes, we have unit tests, that help in validating the feature. I will try to get an integration test but this is not a blocker for feature.
Thanks for the explanation and analysis do you plan on adding more e2e tests later or stick to unit and integration for validating this feature? and to weigh in on the path forward i.e., remove e2e tests for this feature and rely on unit and integration.
- We discussed internally in the sig meeting yesterday and it was decided to have an integration test instead of e2e test. I plan to add an integration test(which we don't have today) later.
since this PR is actually tracking an enhancement work graduating to Beta in 1.13, it should have been tracked via a corresponding issue in k/enhancements repo in 1.13 timeframe. We are now past Feature freeze for 1.13 and its late to open an issue against it. Also we are missing integration tests for this feature, which we are . Given the above two points and from offline discussions with SIG leads regarding the criticality of this feature going to Beta now vs in 1.14 timeframe, I am asking for it to be punted to 1.14. As AIs Please prepare a PR to remove it from Beta in 1.13 Please open a tracking issue in for 1.14 milestone.

Discussion:
"//pkg/apis/core:go_default_library", "//pkg/apis/extensions:go_default_library", "//pkg/apis/scheduling:go_default_library", <del> "//pkg/features:go_default_library", </del> "//pkg/kubelet/apis:go_default_library", "//pkg/quota/v1/evaluator/core:go_default_library", "//pkg/scheduler/algorithm/priorities/util:go_default_library",

"//staging/src/k8s.io/apimachinery/pkg/util/version:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", <del> "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", </del> "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/tools/cache:go_default_library", "//test/e2e/common:go_default_library",

"k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" <del> utilfeature "k8s.io/apiserver/pkg/util/feature" </del> clientset "k8s.io/client-go/kubernetes" <del> "k8s.io/kubernetes/pkg/features" </del> priorityutil "k8s.io/kubernetes/pkg/scheduler/algorithm/priorities/util" "k8s.io/kubernetes/test/e2e/common" "k8s.io/kubernetes/test/e2e/framework"

Expect(err).NotTo(HaveOccurred()) Expect(tolePod.Spec.NodeName).To(Equal(nodeName)) }) <del> It("Pod should be preferably scheduled to nodes which satisfy its limits", func() { if !utilfeature.DefaultFeatureGate.Enabled(features.ResourceLimitsPriorityFunction) { framework.Skipf("ResourceLimits Priority function is not enabled, so skipping this test") } var podwithLargeRequestedResource *v1.ResourceRequirements = &v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("100Mi"), v1.ResourceCPU:    resource.MustParse("100m"), }, Limits: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("3000Mi"), v1.ResourceCPU:    resource.MustParse("5000m"), }, } // Update one node to have large allocatable. lastNode := nodeList.Items[len(nodeList.Items)-1] nodeName := lastNode.Name nodeOriginalMemory, found := lastNode.Status.Allocatable[v1.ResourceMemory] nodeOriginalCPU, found := lastNode.Status.Allocatable[v1.ResourceCPU] nodeOriginalCPUCapacity, found := lastNode.Status.Capacity[v1.ResourceCPU] nodeOriginalMemoryCapacity, found := lastNode.Status.Capacity[v1.ResourceMemory] Expect(found).To(Equal(true)) nodeOriginalMemoryVal := nodeOriginalMemory.Value() nodeOriginalCPUVal := nodeOriginalCPU.MilliValue() nodeOriginalMemoryCapacityVal := nodeOriginalMemoryCapacity.Value() nodeOriginalCPUCapacityVal := nodeOriginalCPUCapacity.MilliValue() err := updateNodeResources(cs, nodeName, int64(10737418240), int64(12000), int64(10737418240), int64(12000)) Expect(err).NotTo(HaveOccurred()) defer func() { // Resize the node back to its original allocatable values. if err := updateNodeResources(cs, nodeName, nodeOriginalMemoryVal, nodeOriginalCPUVal, nodeOriginalMemoryCapacityVal, nodeOriginalCPUCapacityVal); err != nil { framework.Logf("Failed to revert node resources with %v", err) } // Reset the node list with its old entry. nodeList.Items[len(nodeList.Items)-1] = lastNode }() // update nodeList with newNode. newNode, err := cs.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) if err != nil { framework.Logf("Failed to get node", err) } nodeList.Items[len(nodeList.Items)-1] = *newNode err = createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) framework.ExpectNoError(err) // After the above we should see 50% of node to be available which is 5000MiB memory, 6000m cpu for large node. By("Create a pod with unusual large limits") podWithLargeLimits := "with-large-limits" pod := createPausePod(f, pausePodConfig{ Name:      podWithLargeLimits, Resources: podwithLargeRequestedResource, }) framework.ExpectNoError(f.WaitForPodRunning(pod.Name)) By("Pod should preferably scheduled to nodes which satisfy its limits") // The pod should land onto large node(which has 5000MiB free) which satisfies the pod limits which is 3000MiB. podHighLimits, err := cs.CoreV1().Pods(ns).Get(podWithLargeLimits, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(podHighLimits.Spec.NodeName).To(Equal(nodeName)) }) </del> }) // createBalancedPodForNodes creates a pod per node that asks for enough resources to make all nodes have the same mem/cpu usage ratio.

framework.ExpectNodeHasTaint(cs, nodeName, &testTaint) return &testTaint } <del> // updateNodeResources updates the resource of given node with the given values. func updateNodeResources(c clientset.Interface, nodeName string, memoryAllocatable, cpuAllocatable, memoryCapacity, cpuCapacity int64) error { node, err := c.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) framework.ExpectNoError(err) if err != nil { return err } node.Status.Capacity[v1.ResourceMemory] = *resource.NewQuantity(memoryCapacity, resource.BinarySI) node.Status.Capacity[v1.ResourceCPU] = *resource.NewMilliQuantity(cpuCapacity, resource.DecimalSI) node.Status.Allocatable[v1.ResourceMemory] = *resource.NewQuantity(memoryAllocatable, resource.BinarySI) node.Status.Allocatable[v1.ResourceCPU] = *resource.NewMilliQuantity(cpuAllocatable, resource.DecimalSI) _, err = c.CoreV1().Nodes().UpdateStatus(node) return err } </del>