Change:
I'm getting tired of seeing flake issues like where every single failure is of the form This is about as unhelpful an error message as you can get. The problem is that we have peppered throughout the code everywhere, and in this form, you can only figure out what failed if you read the Ginkgo error, remember which line has the line number you actually want, and then try to read surrounding code in that file. Possible improvements given in Use Annotate the assertions Basically, do anything except just Expect(err).NotTo(HaveOccurred()) cc
:cry: Annotate your assertions! From gubernator you can click on the error line and it will try to take you to the appropriate place within the code, but it's not always right.
The github flake issues don't have that link. Maybe they should?
Also, it's even worse:
Think it's worth adding a presubmit verify check ensuring that the first number doesn't go up?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig testing
Is this believed to still be a problem?
*not%20to%20have%20occurred)
Are the additional messages in the errors more helpful? I am not sure if removing the error message is a good approach. What do you think?
If the underlying err was more descriptive than "timed out waiting for the condition" this anti-pattern wouldn't be nearly as annoying.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale this continues to be a great place for new contributors to help out
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I've raised a PR to improve the error messages in the DNS related e2e tests: Just started learning Go/Kubernetes - all feedback welcome :-)
thanks for that PR! I'm going to label this a good-first-issue because this involves fixing error messages. If you are intersted to take this up, pick a few (related) e2e tests and follow the same methods as that of for improving the error messages. Please feel free to ask questions on slack if you get stuck. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : thanks for that PR! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup
I've also raised a PR () to improve error messages in some e2e auth related tests. I tried to follow . I am new to Go/Kubernetes so all feedback are welcome.
Curious where we're seeing ?
Is there a spot we could add some sort of simple grep check that would run on git commits to prevent more of this sneaking in? I'd love to open a PR on that, I'm just not sure where it'd go.
it might be some sort of check, perhaps.
fixed some of the tests, but a bunch of them are still remaining. /reopen
Reopened this issue. <detailsIn response to : fixed some of the tests, but a bunch of them are still remaining. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area deflake More meaningful error text would help us track down flakes
Hi :wave: , I will be working on file. Will send a PR soon. :smile:
/reopen PR fixed some issues, but not all. PR is still open.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We can use framework.ExpectNoError() instead of Expect(err).NotTo(HaveOccurred()) like That will lead we can avoid which is invalid against golint in long term.
But then still needs an additional message string, doesn't it? Otherwise it's not better than .
This may be useful for sig-testing-commons xref: /area e2e-test-framework
I have raised a PR() trying to improve the error messages in bootstrap related file and I am also trying to follow . I am new to GO and kubernetes and this is my first PR, so I am happy to get any feedback.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Some updates on this: the testing-commons project is working on refactoring a lot of the tests and the framework, which will deal with some of the unannotated tests, and we are also working on writing some e2e testing best practices to prevent more unannotated tests in the future.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Leaving a question for the community here... A lot of the work in testing commons has ended up being enforcing that tests spew out meaningful messages. At this point in time, plain that output generic error messages on failure are not happening a whole lot and in many instances the error itself has plenty of information about the test. Thus the issue raised here is largely resolved but there may be places where things could be improved. Places where things could be improved (where an e2e tests requires additional annotation on error checking) are unkown and there is not a simple way of tracking them down (some instances of do not have an annotation on purpose because the possible error has already plenty of information). Does anyone have any additional suggestions of work that should be done? If not, then we can follow the lazy consensus route and just let fejta bot close this issue.
+1
I'd rather err on the side of caution and always add an explanation, because how can the test author be sure that all errors will be informative enough? Can we document somewhere how to use and properly, i.e. "add an explanation unless it is certain that the error itself is informative"? In the past, I've used this issue as reference in code reviews. It would be nice to have something a bit more concise and permanent.
About 50% of the Expect(No)Error calls still lack an explanation However, the unexplained errs are happening far less frequently: Can we document somewhere how to use Expect(err).NotTo(HaveOccurred()) and framework.ExpectNoError() properly, i.e. "add an explanation unless it is certain that the error itself is informative"? (which could probably use a rewrite, but I think this specific link addresses your concern)
/close Since my comment just bumped the recency of this, I'm going to play the role of for today. Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .
Closing this issue. <detailsIn response to : /close Since my comment just bumped the recency of this, I'm going to play the role of for today. Reopen the issue with . Mark the issue as fresh with . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
By("Ensuring at least two jobs and at least one finished job exists by listing jobs explicitly") err = waitForJobsAtLeast(f.ClientSet, f.Namespace.Name, 2) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure at least two job exists in namespace %s", f.Namespace.Name) </ins> err = waitForAnyFinishedJob(f.ClientSet, f.Namespace.Name) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure at least on finished job exists in namespace %s", f.Namespace.Name) </ins> By("Ensuring no unexpected event has happened") err = waitForEventWithReason(f.ClientSet, f.Namespace.Name, cronJob.Name, []string{"MissingJob", "UnexpectedJob"})

By("Ensuring a job is scheduled") err = waitForActiveJobs(f.ClientSet, f.Namespace.Name, cronJob.Name, 1) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure a %s cronjob is scheduled in namespace %s", cronJob.Name, f.Namespace.Name) </ins> By("Ensuring exactly one is scheduled") cronJob, err = getCronJob(f.ClientSet, f.Namespace.Name, cronJob.Name) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure exactly one %s cronjob is scheduled in namespace %s", cronJob.Name, f.Namespace.Name) </ins> Expect(cronJob.Status.Active).Should(HaveLen(1)) By("Deleting the job")

By("Ensuring the job is not in the cronjob active list") err = waitForJobNotActive(f.ClientSet, f.Namespace.Name, cronJob.Name, job.Name) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure the %s cronjob is not in active list in namespace %s", cronJob.Name, f.Namespace.Name) </ins> By("Ensuring MissingJob event has occurred") err = waitForEventWithReason(f.ClientSet, f.Namespace.Name, cronJob.Name, []string{"MissingJob"}) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure missing job event has occurred for %s cronjob in namespace %s", cronJob.Name, f.Namespace.Name) </ins> By("Removing cronjob") err = deleteCronJob(f.ClientSet, f.Namespace.Name, cronJob.Name) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to remove %s cronjob in namespace %s", cronJob.Name, f.Namespace.Name) </ins> }) // cleanup of successful finished jobs, with limit of one successful job

cronJob := newTestCronJob("concurrent-limit", "*/1 * * * ?", batchv1beta1.AllowConcurrent, successCommand, &successLimit) cronJob, err := createCronJob(f.ClientSet, f.Namespace.Name, cronJob) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to create allowconcurrent cronjob with custom history limits in namespace %s", f.Namespace.Name) </ins> // Job is going to complete instantly: do not check for an active job // as we are most likely to miss it By("Ensuring a finished job exists") err = waitForAnyFinishedJob(f.ClientSet, f.Namespace.Name) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure a finished cronjob exists in namespace %s", f.Namespace.Name) </ins> By("Ensuring a finished job exists by listing jobs explicitly") jobs, err := f.ClientSet.BatchV1().Jobs(f.Namespace.Name).List(metav1.ListOptions{}) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure a finished cronjob exists by listing jobs explicitly in namespace %s", f.Namespace.Name) </ins> _, finishedJobs := filterActiveJobs(jobs) Expect(len(finishedJobs) == 1).To(BeTrue()) // Job should get deleted when the next job finishes the next minute By("Ensuring this job and its pods does not exist anymore") err = waitForJobToDisappear(f.ClientSet, f.Namespace.Name, finishedJobs[0]) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure that job does not exists anymore in namespace %s", f.Namespace.Name) </ins> err = waitForJobsPodToDisappear(f.ClientSet, f.Namespace.Name, finishedJobs[0]) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure that pods for job does not exists anymore in namespace %s", f.Namespace.Name) </ins> By("Ensuring there is 1 finished job by listing jobs explicitly") jobs, err = f.ClientSet.BatchV1().Jobs(f.Namespace.Name).List(metav1.ListOptions{}) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to ensure there is one finished job by listing job explicitly in namespace %s", f.Namespace.Name) </ins> _, finishedJobs = filterActiveJobs(jobs) Expect(len(finishedJobs) == 1).To(BeTrue()) By("Removing cronjob") err = deleteCronJob(f.ClientSet, f.Namespace.Name, cronJob.Name) <del> Expect(err).NotTo(HaveOccurred()) </del> <ins> Expect(err).NotTo(HaveOccurred(), "Failed to remove the %s cronjob in namespace %s", cronJob.Name, f.Namespace.Name) </ins> }) })