Change:
<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package reporters import ( "bytes" "encoding/json" "fmt" "io/ioutil" "net/http" "strings" "time" "k8s.io/klog" "github.com/onsi/ginkgo/config" "github.com/onsi/ginkgo/types" ) // ProgressReporter is a ginkgo reporter which tracks the total number of tests to be run/passed/failed/skipped. // As new tests are completed it updates the values and prints them to stdout and optionally, sends the updates // to the configured URL. type ProgressReporter struct { LastMsg string `json:"msg"` TestsTotal     int `json:"total"` TestsCompleted int `json:"completed"` TestsSkipped   int `json:"skipped"` TestsFailed    int `json:"failed"` Failures []string `json:"failures,omitempty"` progressURL string client      *http.Client } // NewProgressReporter returns a progress reporter which posts updates to the given URL. func NewProgressReporter(progressReportURL string) *ProgressReporter { rep := &ProgressReporter{ Failures:    []string{}, progressURL: progressReportURL, } if len(progressReportURL) > 0 { rep.client = &http.Client{ Timeout: time.Second * 10, } } return rep } // SpecSuiteWillBegin is invoked by ginkgo when the suite is about to start and is the first point in which we can // antipate the number of tests which will be run. func (reporter *ProgressReporter) SpecSuiteWillBegin(cfg config.GinkgoConfigType, summary *types.SuiteSummary) { reporter.TestsTotal = summary.NumberOfSpecsThatWillBeRun reporter.LastMsg = "Test Suite starting" reporter.sendUpdates() } // SpecSuiteDidEnd is the last method invoked by Ginkgo after all the specs are run. func (reporter *ProgressReporter) SpecSuiteDidEnd(summary *types.SuiteSummary) { reporter.LastMsg = "Test Suite completed" reporter.sendUpdates() } // SpecDidComplete is invoked by Ginkgo each time a spec is completed (including skipped specs). func (reporter *ProgressReporter) SpecDidComplete(specSummary *types.SpecSummary) { testname := strings.Join(specSummary.ComponentTexts[1:], " ") switch specSummary.State { case types.SpecStateFailed: if len(specSummary.ComponentTexts) > 0 { reporter.Failures = append(reporter.Failures, testname) } else { reporter.Failures = append(reporter.Failures, "Unknown test name") } reporter.TestsFailed++ reporter.LastMsg = fmt.Sprintf("FAILED %v", testname) case types.SpecStatePassed: reporter.TestsCompleted++ reporter.LastMsg = fmt.Sprintf("PASSED %v", testname) case types.SpecStateSkipped: reporter.TestsSkipped++ return default: return } reporter.sendUpdates() } // sendUpdates serializes the current progress and prints it to stdout and also posts it to the configured endpoint if set. func (reporter *ProgressReporter) sendUpdates() { b := reporter.serialize() fmt.Println(string(b)) go reporter.postProgressToURL(b) } func (reporter *ProgressReporter) postProgressToURL(b []byte) { // If a progressURL and client is set/available then POST to it. Noop otherwise. if reporter.client == nil || len(reporter.progressURL) == 0 { return } resp, err := reporter.client.Post(reporter.progressURL, "application/json", bytes.NewReader(b)) if err != nil { klog.Errorf("Failed to post progress update to %v: %v", reporter.progressURL, err) return } if resp.StatusCode >= 400 { klog.Errorf("Unexpected response when posting progress update to %v: %v", reporter.progressURL, resp.StatusCode) if resp.Body != nil { defer resp.Body.Close() respBody, err := ioutil.ReadAll(resp.Body) if err != nil { klog.Errorf("Failed to read response body from posting progress: %v", err) return } klog.Errorf("Response body from posting progress update: %v", respBody) } return } } func (reporter *ProgressReporter) serialize() []byte { b, err := json.Marshal(reporter) if err != nil { return []byte(fmt.Sprintf(`{"msg":"%v", "error":"%v"}`, reporter.LastMsg, err)) } return b } // SpecWillRun is implemented as a noop to satisfy the reporter interface for ginkgo. func (reporter *ProgressReporter) SpecWillRun(specSummary *types.SpecSummary) {} // BeforeSuiteDidRun is implemented as a noop to satisfy the reporter interface for ginkgo. func (reporter *ProgressReporter) BeforeSuiteDidRun(setupSummary *types.SetupSummary) {} // AfterSuiteDidRun is implemented as a noop to satisfy the reporter interface for ginkgo. func (reporter *ProgressReporter) AfterSuiteDidRun(setupSummary *types.SetupSummary) {} </ins>

Discussion:
What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale