symfony/symfony — pull request #11031 by cleentfaar (2014-06-01 15:07:44+00:00)
Title: PR for #10454: Random failures in test suite

| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | no
| BC breaks? | no
| Deprecations? | no
| Tests pass? | yes
| Fixed tickets | https://github.com/symfony/symfony/issues/10454
| License | MIT
| Doc PR | -

I've added a ``@group benchmark`` annotation to specific Stopwatch tests that rely on timing, so they can be treated differently if the user demands it. For symfony, it fixes the issue we have with Travis failing on these timing tests because this ``benchmark`` group is excluded from Travis' phpunit coverage.

Thus it improves testing time (saving on retry attempts done by Travis after each random failure), and prevents false-positives reported under a contributor's PR.

In my opinion we should be able to trust Travis tests and any reported failures. Having to ignore certain test results because they can't be fixed suggests to me we should disable that test until we can trust it again.

Alternatively; I could make a new group like ``timing`` so it's more distinct from the other ``benchmark`` tests. Let me know.