jclouds/jclouds — pull request #577 by demobox (2014-10-18 23:50:21+00:00)
Title: Expecting 2 of 3 or the first flaky test to succeed

Really only opening a PR for this to gather some thoughts on this approach to dealing with flaky tests. It's easy to configure and makes use of out-of-the-box functionality of TestNG, but _does_ introduce the overhead of running this test three times on each run.

An approach an I also considered (similar to what's discussed in [this post](http://stackoverflow.com/questions/7803691/how-to-optimize-testng-and-seleniums-tests)) was to create a custom retry analyzer that would only run the test three times _if the first invocation failed_.

Given the relatively small number of consistently flaky tests we have, though, my thought was first to try the out-of-the-box approach, and to go for the custom code if we feel the total overhead imposed by running the flaky tests multiple times on each run gets too big.