Change:
var r []ginkgo.Reporter if *reportDir != "" { r = append(r, reporters.NewJUnitReporter(path.Join(*reportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) <ins> failReport := &failReporter{} r = append(r, failReport) defer func() { if failReport.failed { coreDump(*reportDir) } }() </ins> } ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }

Discussion:
We need to be able to diagnose e2e test failures after the relevant test cluster has been shut down. Right now this almost works, but not quite. Fluentd can currently only be configured to send log files to one place, for example: Cloud Logging, or (running as a pod in the e2e test cluster, and saved to local disk on the node). With the current configuration in , logs typically go to ElasticSearch (and there is an e2e test to check that that works correctly), and tears down the e2e test cluster after each test run, so the logs are deleted. The proposed solution is to have fluentd configurable to send to multiple log collector backends, and configure it as such to both Google Cloud Logging and ElasticSearch for CI e2e test runs to facilitate easier debugging. Satnam6502@ has offered to sort this out.
Originally it was possible to send logs to both Elasticsearch and Cloud Logging but at some point this got made into a binary choice of one or the other (by or ?). I propose to undo this unless there are strong objections. : what do you think?
I didn't turn it into a binary choice. :) , Satnam Singh wrote:
Yeah, pretty sure it was Joe. I am sure it was a wide decision at the time.
Would be nice to have these at maximum verbosity because when we need them we need them all, IIRC we currently only get V(2) on a kube-up cluster
I see this is marked as a post v1.0 milestone? If we don't want this urgently then I will un-assign myself for now.
There is a valid argument that we would not block the 1.0 release for lack of these persisted test logs, and hence the v1.0 milestone was tentatively removed. There is a counter argument that we stand little chance of being able to debug the existing and still-to-be-found flaky test failures in the absence of log files, and that one or more of those bugs, if not fixed, will block the launch anyway. I'd say that if there is nothing on the v1.0 milestone that you can sensibly work on, this one is fair game.

I would find it useful, and I suppose this should be on par with writing tests (of which we have a few 1.0: )
I'm not sure we'll be able to convince folks to move this to 1.0, but feel free to work on this, as it seems very valuable.
I thought about changing our setup to allow both Elasticsearch and Cloud Logging to be targeted. I've been a little hesitant because this could lead to a breakage (esp. for GKE) and we are getting close to 1.0 ...
/cc who might have memories about a GKE breakage caused by changes in env vars used for logging.
I'd love to see this fixed soon to help debug flakes, but yes, it'd be nice to at least be cc'ed on env var changes.
Try again to have it to the high priority list, as this is making debugging the system very difficult. cc Who expressed interest.
How do we access those logs? I tried to find them for gce-scalability jenkins suite, but didn't find it.