Code change:
We found yesterday that cAdvisor wasn't being started properly in 0.9.0 and 0.9.1 due to an issue with manifest file format changes (, fixed in ). If any of our e2e tests even checked that cAdvisor was running or exposing metrics on all nodes, it would have been caught before release. We should add at least some coverage for it to catch bugs like this.
If any of you are up for adding an e2e test for cadvisor, that'd be much appreciated!
I will work on adding one for cadvisor. , Alex Mohr wrote:
We already have a e2e test, which validate monitoring of Clusters using cAdvisor. We should figure out why the test passed without detecting such failure, and improve that test.
The existing test will succeed even if some of the nodes do not have cadvisor running. We can enhance the monitoring test to specifically check for cadvisor's health as well. , Dawn Chen wrote:
This test seems to be failing deterministically.
This was after fix to retry, but we're not exactly sure why.
Once I have emerged from the e2e test for cluster level logging with Elasticsearch quagmire (soon I hope -- perhaps by Wednesday) then I might also be willing to lend a hand with this since some of the experience might be relevant.
Actually latest error looks like So it's still not coming up after 5 minutes. The question is how long should it be taking to come up, given that the cadvisor containers get scheduled basically on node creation. The test finished in < 1s on my e2e cluster that has been up for awhile.
Or there's an actual bug here. can you look into it?
(I looked at the Jenkins log, and this test ran a good 10 minutes after startup, so it probably had 15 minutes to finish coming up. That seems wrong, given that cAdvisor is actually the first container we ever schedule, and it gets scheduled raw.)
Sorry, I hadn't seen -- not sure if that fixes this?
No, there are failures after as well. On Feb 20, 2015 5:39 PM, "David Oppenheimer" wrote:
I don't see any issue with the test itself. The test attempts to access a REST endpoint ('/stats') via the proxy on the api-server. Is it possible that the api-server doesn't get to a stable state for a while?
This test executes after several others that hit the apiserver? On Feb 22, 2015 4:15 PM, "Vish Kannan" wrote:
Noob question: How can we tell if it runs this test after the other tests?
You can't tell deterministically (i.e. it's not hardcoded). I was merely commenting that in several runs where this fails, it has run after many other tests. We've been randomizing the spec order for a while. You can look at the Jenkins console logs to get the random seed for a given run: And the console logs have the order just by manually scanning the horizontal bars, but it's kind of pain. (That seed was off of go/k8s-test/job/kubernetes-e2e-gce/2638/consoleFull)
The reason for this failure is . I am closing this issue since the test itself is fine.

Related discussion:
text: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" . "github.com/onsi/ginkgo" ) var _ = Describe("Cadvisor", func() { var c *client.Client BeforeEach(func() { var err error c, err = loadClient() expectNoError(err) }) It("cadvisor should be healthy on every node.", func() { CheckCadvisorHealthOnAllNodes(c) }) }) func CheckCadvisorHealthOnAllNodes(c *client.Client) { By("getting list of nodes") nodeList, err := c.Nodes().List() expectNoError(err) for _, node := range nodeList.Items { // cadvisor is not accessible directly unless its port (4194 by default) is exposed. // Here, we access '/stats/' REST endpoint on the kubelet which polls cadvisor internally. statsResource := fmt.Sprintf("api/v1beta1/proxy/minions/%s/stats/", node.Name) By(fmt.Sprintf("Querying stats from node %s using url %s", node.Name, statsResource)) _, err = c.Get().AbsPath(statsResource).Timeout(1 * time.Second).Do().Raw() expectNoError(err) } } </ins>