Change:
We for uploading to GCS on , but not on successful e2e runs or failed test -up. We should definitely be dumping those logs for failed cluster-ups, and it would also be nice to throw in the successes too (good to have recent green for comparison).
Getting logs uploaded on a failed test -up seems really important for debugging.
+1 There were just a bunch of failures in scalability test suite (run 1623-1628) that failed because not all system pods were running. Since there are no logs, there is no easy way to debug it. It would be super useful to have all the logs for failed cluster-ups. cc
Unless I'm confused, it looks as if these logs are still not being uploaded. FYI.
For example: gs://kubernetes-jenkins/logs/kubernetes-e2e-gce/7500
That particular build was a timeout. When Jenkins kills the job, the log-collecting code never runs.
Another case we won't collect logs is if the cluster fails to initialize. (Which is a really important place to collect logs, but it's not a trivial fix.)
Moving this into post, since it seems unlikely to get fixed in the near term. feel free to correct me if I'm wrong.
We also don't collect logs on test job timeouts, which is frustrating.
+1
will collect the logs when the cluster fails to initialize. That's not all the cases listed here, but it's a good one.
I'd like to bump the priority for this further - we don't have any logs to help with debugging , which started to happen more often recently - it seems that didn't help. cc
I have too many P0s assigned to me already. This is at least 3rd in line right now.
This one is half year old and is P0 for a month...
From discussion with on hangouts, a vague plan: move log-dumping logic from into a standalone command in something like (feel free to bikeshed the name) update Jenkins to call this command as a post-build step probably remove and the debug stuff recently to . Right now the stuff is pretty GCE/GKE specific and uses ssh. We should probably change to a more provider-agnostic method, but this would be a good start.
Hm, actually, post-build step actually won't work, since we tear down the cluster first. But making it a separate command can still let us call it in more places.
FYI - we also need logs from timed-out runs (not only the failed ones).
moving out into its own binary should help accomplish this.
I'm worried that the best way to do this is to write a bash script that grabs IPs from 's and functions and just does an ssh and cats the logs. Then we could point at that. It's sorta difficult to make work on its own.
I think at this point we're happy with anything that usually works. We can work on making in nice and shiny later.
We now get logs when kube-up fails, which recently happened :) Still some work to do here, I'll leave this issue open until timeouts are handled properly and points here.
- I can't thank you enough!
Ok, is in so we should get logs for kubernetes-e2e-* jobs and PR Jenkins jobs on timeout.
I'm gonna call this done.

Discussion:
import ( "flag" "fmt" <ins> "os" </ins> "path" "strings" "testing"

"github.com/onsi/ginkgo" "github.com/onsi/ginkgo/config" "github.com/onsi/ginkgo/reporters" <del> "github.com/onsi/ginkgo/types" </del> "github.com/onsi/gomega" )

reportDir = flag.String("report-dir", "", "Path to the directory where the JUnit XML reports should be saved. Default is empty, which doesn't generate these reports.") ) <del> type failReporter struct { failed bool } func (f *failReporter) SpecSuiteWillBegin(config config.GinkgoConfigType, summary *types.SuiteSummary) { } func (f *failReporter) BeforeSuiteDidRun(setupSummary *types.SetupSummary) {} func (f *failReporter) SpecWillRun(specSummary *types.SpecSummary)         {} func (f *failReporter) SpecDidComplete(specSummary *types.SpecSummary) { if specSummary.Failed() { f.failed = true } } func (f *failReporter) AfterSuiteDidRun(setupSummary *types.SetupSummary) {} func (f *failReporter) SpecSuiteDidEnd(summary *types.SuiteSummary)       {} </del> func init() { // Turn on verbose by default to get spec names config.DefaultReporterConfig.Verbose = true

util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() <ins> if *reportDir != "" { if err := os.MkdirAll(*reportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } defer coreDump(*reportDir) } </ins> if testContext.Provider == "" { glog.Info("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.")

// test pods from running, and tests that ensure all pods are running and // ready will fail). if err := waitForPodsRunningReady(api.NamespaceDefault, testContext.MinStartupPods, podStartupTimeout); err != nil { <del> glog.Fatalf("Error waiting for all pods to be running and ready: %v", err) </del> <ins> t.Errorf("Error waiting for all pods to be running and ready: %v", err) return </ins> } // Run tests through the Ginkgo runner with output to console + JUnit for Jenkins var r []ginkgo.Reporter if *reportDir != "" { r = append(r, reporters.NewJUnitReporter(path.Join(*reportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) <del> failReport := &failReporter{} r = append(r, failReport) defer func() { if failReport.failed { coreDump(*reportDir) } }() </del> } ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }