Change:
For example: I'm disabling the test in But we need to re-enable it, or roll-back the test.
fyi
Shouldn't this is part slow test, and the fix should be either: 1) Increase Kubernetes-e2-gke-ci timeout value if decide to include slow to that project or 2) disable all slow tests on that project? cc/ Which one is required here?
The test runs 50 pods per node, which is over the default --max-pod limit of kubelet. The test relies on the flag --max-pods=100 being set in cluster/gce/config- I will disable it for all GKE tests in On the other hand, it's not clear to me where the timeout is set for a specific build. /cc
Timeouts for each Jenkins build are in the Jenkins project configurations themselves, which are not currently saved in version control anywhere. :
Discussed with about the test offline. There are several issues: Do we want to run any kind of kubelet resource monitoring test on GKE? Is it useful? We only override max-pods in GCE. Should the test itself figure out what platform it can run based on the #pods in the test (see ) Per-test configuration allows greater flexibility (e.g., we may run 100 pods on GCE to test kubelet's scalability), but it'd make the tests more complicated as they need to be aware of the test environment (which many e2e tests already do); there is also the burden of updating the tests once the max-pod value changes. Does this test belong to SLOW tests or should it belong to a new category? Not all slow tests have the same attributes. Arguably this test could be enabled only on selective builds. The downside would be more flags -- less readable/manageable e2e configs. Not all builds which run SLOW tests have the proper timeout value to run this test. We need to manually change them every time a slow test is . Re-assigning this back to since she has more concerns about the PR.
Had offline discussion with We are going to split current Kubelet resource usage tracking test into two, one is targeted for GCE, another one is for GKE. In this case, we won't mess up the test code itself by introducing knobs. Re-assign this back to

Discussion:
"Shell" "Daemonsset" "Deployment" <ins> "experimentalsresourcesusagestracking" # Expect --max-pods=100 </ins> ) <ins> # Tests which cannot be run on AWS. AWS_REQUIRED_SKIP_TESTS=( "experimentalsresourcesusagestracking" # Expect --max-pods=100 ) </ins> # The following tests are known to be flaky, and are thus run only in their own # -flaky- build variants. GCE_FLAKY_TESTS=(

${GCE_PARALLEL_SKIP_TESTS[@]:+${GCE_PARALLEL_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  ${GCE_PARALLEL_FLAKY_TESTS[@]:+${GCE_PARALLEL_FLAKY_TESTS[@]}}  <ins> ${AWS_REQUIRED_SKIP_TESTS[@]:+${AWS_REQUIRED_SKIP_TESTS[@]}}  </ins> )"} : ${ENABLE_DEPLOYMENTS:=true} # Override AWS defaults.

${REBOOT_SKIP_TESTS[@]:+${REBOOT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  ${GCE_PARALLEL_SKIP_TESTS[@]:+${GCE_PARALLEL_SKIP_TESTS[@]}}  <ins> ${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}}  </ins> )"} ;;

${GKE_REQUIRED_SKIP_TESTS[@]:+${GKE_REQUIRED_SKIP_TESTS[@]}}  ${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  <ins> ${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}}  </ins> )"} ;;

} } <ins> func runResourceTrackingTest(framework *Framework, podsPerNode int, nodeNames sets.String, resourceMonitor *resourceMonitor) { numNodes := nodeNames.Len() totalPods := podsPerNode * numNodes By(fmt.Sprintf("Creating a RC of %d pods and wait until all pods of this RC are running", totalPods)) rcName := fmt.Sprintf("resource%d-%s", totalPods, string(util.NewUUID())) // TODO: Use a more realistic workload Expect(RunRC(RCConfig{ Client:    framework.Client, Name:      rcName, Namespace: framework.Namespace.Name, Image:     "gcr.io/google_containers/pause:go", Replicas:  totalPods, })).NotTo(HaveOccurred()) // Log once and flush the stats. resourceMonitor.LogLatest() resourceMonitor.Reset() By("Start monitoring resource usage") // Periodically dump the cpu summary until the deadline is met. // Note that without calling resourceMonitor.Reset(), the stats // would occupy increasingly more memory. This should be fine // for the current test duration, but we should reclaim the // entries if we plan to monitor longer (e.g., 8 hours). deadline := time.Now().Add(monitoringTime) for time.Now().Before(deadline) { Logf("Still running...%v left", deadline.Sub(time.Now())) time.Sleep(reportingPeriod) timeLeft := deadline.Sub(time.Now()) Logf("Still running...%v left", timeLeft) if timeLeft < reportingPeriod { time.Sleep(timeLeft) } else { time.Sleep(reportingPeriod) } logPodsOnNodes(framework.Client, nodeNames.List()) } By("Reporting overall resource usage") logPodsOnNodes(framework.Client, nodeNames.List()) resourceMonitor.LogCPUSummary() resourceMonitor.LogLatest() By("Deleting the RC") DeleteRC(framework.Client, framework.Namespace.Name, rcName) } </ins> var _ = Describe("Kubelet", func() { <del> var numNodes int </del> var nodeNames sets.String framework := NewFramework("kubelet-perf") var resourceMonitor *resourceMonitor

BeforeEach(func() { nodes, err := framework.Client.Nodes().List(labels.Everything(), fields.Everything()) expectNoError(err) <del> numNodes = len(nodes.Items) </del> nodeNames = sets.NewString() for _, node := range nodes.Items { nodeNames.Insert(node.Name)

resourceMonitor.Stop() }) <del> Describe("resource usage tracking", func() { density := []int{0, 50} </del> <ins> Describe("regular resource usage tracking", func() { density := []int{0, 35} for i := range density { podsPerNode := density[i] name := fmt.Sprintf( "over %v with %d pods per node.", monitoringTime, podsPerNode) It(name, func() { runResourceTrackingTest(framework, podsPerNode, nodeNames, resourceMonitor) }) } }) Describe("experimental resource usage tracking", func() { density := []int{50} </ins> for i := range density { podsPerNode := density[i] name := fmt.Sprintf( "over %v with %d pods per node.", monitoringTime, podsPerNode) It(name, func() { <del> // Skip this test for GKE. // TODO: Re-activate this for GKE SkipIfProviderIs("gke") totalPods := podsPerNode * numNodes By(fmt.Sprintf("Creating a RC of %d pods and wait until all pods of this RC are running", totalPods)) rcName := fmt.Sprintf("resource%d-%s", totalPods, string(util.NewUUID())) // TODO: Use a more realistic workload Expect(RunRC(RCConfig{ Client:    framework.Client, Name:      rcName, Namespace: framework.Namespace.Name, Image:     "gcr.io/google_containers/pause:go", Replicas:  totalPods, })).NotTo(HaveOccurred()) // Log once and flush the stats. resourceMonitor.LogLatest() resourceMonitor.Reset() By("Start monitoring resource usage") // Periodically dump the cpu summary until the deadline is met. // Note that without calling resourceMonitor.Reset(), the stats // would occupy increasingly more memory. This should be fine // for the current test duration, but we should reclaim the // entries if we plan to monitor longer (e.g., 8 hours). deadline := time.Now().Add(monitoringTime) for time.Now().Before(deadline) { Logf("Still running...%v left", deadline.Sub(time.Now())) time.Sleep(reportingPeriod) logPodsOnNodes(framework.Client, nodeNames.List()) } By("Reporting overall resource usage") logPodsOnNodes(framework.Client, nodeNames.List()) resourceMonitor.LogCPUSummary() resourceMonitor.LogLatest() By("Deleting the RC") DeleteRC(framework.Client, framework.Namespace.Name, rcName) </del> <ins> runResourceTrackingTest(framework, podsPerNode, nodeNames, resourceMonitor) </ins> }) } })