Change:
The following E2E test is failing in the GCE 1.5-1.6 upgrade jobs. Please take a look and let me know ASAP if this is a 1.6 blocker or not (I think it may be a flake, but need you to confirm). https://k8s- Expected error: <*errors.errorString | 0xc421d5ef10: { s: "expected to find one replica for RC or deployment with label heapster but got 0", } expected to find one replica for RC or deployment with label heapster but got 0 not to have occurred
/assign
can you help get this looked at and triaged ASAP? Thanks.
I pinged by email. He's really the one who should look at this.
The problem seems to be not with the test itself but with Kubernetes or test framework, since the test failed on starting Heapster. I'll investigate more later today.
I haven't got time to investigate. This seems to be unrelated to monitoring itself, so any help during MTV day would be great. If not I'll try to fix this tomorrow
thanks
The problem is in , in function . The function is called immediately from , w/o waiting for system pods (heapster) to start. We should give some time for heapster to become ready, as it takes few minutes after cluster creation.
CC
thanks a lot for investigation! is going to fix it.

Discussion:
influxdb "github.com/influxdata/influxdb/client" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/client/clientset_generated/clientset" "k8s.io/kubernetes/test/e2e/framework"

}) const ( <del> influxdbService      = "monitoring-influxdb" influxdbDatabaseName = "k8s" podlistQuery         = "show tag values from "cpu/usage" with key = pod_id" nodelistQuery        = "show tag values from "cpu/usage" with key = hostname" sleepBetweenAttempts = 5 * time.Second testTimeout          = 5 * time.Minute </del> <ins> influxdbService       = "monitoring-influxdb" influxdbDatabaseName  = "k8s" podlistQuery          = "show tag values from "cpu/usage" with key = pod_id" nodelistQuery         = "show tag values from "cpu/usage" with key = hostname" sleepBetweenAttempts  = 5 * time.Second testTimeout           = 5 * time.Minute initializationTimeout = 5 * time.Minute </ins> ) var (

func testMonitoringUsingHeapsterInfluxdb(c clientset.Interface) { // Check if heapster pods and services are up. <del> expectedPods, err := verifyExpectedRcsExistAndGetExpectedPods(c) framework.ExpectNoError(err) framework.ExpectNoError(expectedServicesExist(c)) // TODO: Wait for all pods and services to be running. </del> <ins> var expectedPods []string rcErr := fmt.Errorf("failed to verify expected RCs within timeout") serviceErr := fmt.Errorf("failed to verify expected services within timeout") err := wait.PollImmediate(sleepBetweenAttempts, initializationTimeout, func() (bool, error) { expectedPods, rcErr = verifyExpectedRcsExistAndGetExpectedPods(c) if rcErr != nil { framework.Logf("Waiting for expected RCs (got error: %v)", rcErr) return false, nil } serviceErr = expectedServicesExist(c) if serviceErr != nil { framework.Logf("Waiting for expected services (got error: %v)", serviceErr) return false, nil } return true, nil }) if err != nil { framework.ExpectNoError(rcErr) framework.ExpectNoError(serviceErr) framework.Failf("Failed to verify RCs and services within timeout: %v", err) } </ins> expectedNodes, err := getAllNodesInCluster(c) framework.ExpectNoError(err)