Change:
<ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "io/ioutil" "net" "strings" "sync" ) type command struct { cmd       string component string } func coreDump(dir string) { c, err := loadClient() if err != nil { fmt.Printf("Error creating client: %v", err) return } provider := testContext.Provider if !providerIs("gce", "gke") { fmt.Printf("Skipping SSH core dump, which is not implemented for %s", provider) return } // Get all nodes' external IPs. hosts, err := NodeSSHHosts(c) if err != nil { fmt.Printf("Error getting node hostnames: %v", err) return } cmds := []command{ {"cat /var/log/kubelet.log", "kubelet"}, {"cat /var/log/kube-proxy.log", "kube-proxy"}, } logCore(cmds, hosts, dir, provider) // I wish there was a better way to get the master IP... config, err := loadConfig() if err != nil { fmt.Printf("Error loading config: %v") } ix := strings.LastIndex(config.Host, "/") master := net.JoinHostPort(config.Host[ix+1:], "22") cmds = []command{ {"cat /var/log/kube-apiserver.log", "kube-apiserver"}, {"cat /var/log/kube-scheduler.log", "kube-scheduler"}, {"cat /var/log/kube-controller-manager.log", "kube-controller-manager"}, } logCore(cmds, []string{master}, dir, provider) } func logCore(cmds []command, hosts []string, dir, provider string) { wg := &sync.WaitGroup{} // Run commands on all nodes via SSH. for _, cmd := range cmds { fmt.Printf("SSH'ing to all nodes and running %sn", cmd.cmd) for _, host := range hosts { go func() { defer wg.Done() wg.Add(1) logfile := fmt.Sprintf("%s/%s-%s.log", dir, host, cmd.component) fmt.Printf("Writing to %s.n", logfile) stdout, stderr, _, err := SSH(cmd.cmd, host, provider) if err != nil { fmt.Printf("Error running command: %vn", err) } if err := ioutil.WriteFile(logfile, []byte(stdout+stderr), 0777); err != nil { fmt.Printf("Error writing logfile: %vn", err) } }() } } wg.Wait() } </ins>

Discussion:
We need to be able to diagnose e2e test failures after the relevant test cluster has been shut down. Right now this almost works, but not quite. Fluentd can currently only be configured to send log files to one place, for example: Cloud Logging, or (running as a pod in the e2e test cluster, and saved to local disk on the node). With the current configuration in , logs typically go to ElasticSearch (and there is an e2e test to check that that works correctly), and tears down the e2e test cluster after each test run, so the logs are deleted. The proposed solution is to have fluentd configurable to send to multiple log collector backends, and configure it as such to both Google Cloud Logging and ElasticSearch for CI e2e test runs to facilitate easier debugging. Satnam6502@ has offered to sort this out.
Originally it was possible to send logs to both Elasticsearch and Cloud Logging but at some point this got made into a binary choice of one or the other (by or ?). I propose to undo this unless there are strong objections. : what do you think?
I didn't turn it into a binary choice. :) , Satnam Singh wrote:
Yeah, pretty sure it was Joe. I am sure it was a wide decision at the time.
Would be nice to have these at maximum verbosity because when we need them we need them all, IIRC we currently only get V(2) on a kube-up cluster
I see this is marked as a post v1.0 milestone? If we don't want this urgently then I will un-assign myself for now.
There is a valid argument that we would not block the 1.0 release for lack of these persisted test logs, and hence the v1.0 milestone was tentatively removed. There is a counter argument that we stand little chance of being able to debug the existing and still-to-be-found flaky test failures in the absence of log files, and that one or more of those bugs, if not fixed, will block the launch anyway. I'd say that if there is nothing on the v1.0 milestone that you can sensibly work on, this one is fair game.

I would find it useful, and I suppose this should be on par with writing tests (of which we have a few 1.0: )
I'm not sure we'll be able to convince folks to move this to 1.0, but feel free to work on this, as it seems very valuable.
I thought about changing our setup to allow both Elasticsearch and Cloud Logging to be targeted. I've been a little hesitant because this could lead to a breakage (esp. for GKE) and we are getting close to 1.0 ...
/cc who might have memories about a GKE breakage caused by changes in env vars used for logging.
I'd love to see this fixed soon to help debug flakes, but yes, it'd be nice to at least be cc'ed on env var changes.
Try again to have it to the high priority list, as this is making debugging the system very difficult. cc Who expressed interest.
How do we access those logs? I tried to find them for gce-scalability jenkins suite, but didn't find it.