Repository: pantsbuild/pants

Issue:
`TestJUnitHtmlReport.test_merged_conflict_first` is flaky under py3 CI shard
```
                     
                   Invalidated 1 target.
22:41:09 04:58       [run]
                     ============== test session starts ===============
                     platform linux -- Python 3.4.3, pytest-3.6.4, py-1.5.4, pluggy-0.7.1 -- /usr/bin/python3.4
                     cachedir: ../../../.pytest_cache
                     rootdir: /home/travis/build/pantsbuild/pants/.pants.d, inifile: /home/travis/build/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.4.3/35c8a8a888bcdafcaa244e97c736198c49a3f69d/pytest.ini
                     plugins: cov-2.4.0, timeout-1.2.1
                     collecting ... collected 13 items
                     
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_all <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [  7%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_empty <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 15%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_errored <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 23%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_failed <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 30%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_merged_conflict_error <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 38%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_merged_conflict_first <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py FAILED [ 46%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_merged_no_conflict <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 53%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_no_open_report <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 61%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_open_report <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 69%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_passing <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 76%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_skipped <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 84%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_time <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [ 92%]
                     tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_unicode <- pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py PASSED [100%]
                     
                     ==================== FAILURES ====================
                     _ TestJUnitHtmlReport.test_merged_conflict_first _
                     
                     self = <pants_test.backend.jvm.tasks.reports.test_junit_html_report.TestJUnitHtmlReport testMethod=test_merged_conflict_first>
                     
                         def test_merged_conflict_first(self):
                           with self.merge_conflict() as xml_dir:
                             report = JUnitHtmlReport.create(xml_dir, error_on_conflict=False)
                         
                             testsuites = report._parse_xml_files()
                             self.assertEqual(1, len(testsuites))
                         
                             suite_a = testsuites[0]
                             self.assertEqual(1, suite_a.errors)
                             self.assertEqual(0, suite_a.failures)
                             self.assertEqual(0, suite_a.skipped)
                             self.assertEqual(0.04, suite_a.time)
                             self.assertEqual(2, suite_a.tests)
                             self.assertEqual([ReportTestCase(name='test-a', time=0.02),
                                               ReportTestCase(name='test-b', time=0.02,
                                                              error='java.lang.RuntimeException!')],
                     >                        suite_a.testcases)
                     E       AssertionError: Lists differ: [Repo[17 chars]test-a', time=0.02, failure=None, error=None, [118 chars]lse)] != [Repo[17 chars]test-b', time=0.02, failure=None, error='java.[118 chars]lse)]
                     E       
                     E       First differing element 0:
                     E       ReportTestCase(name=test-a, time=0.02, failure=None, error=None, skipped=False)
                     E       ReportTestCase(name=test-b, time=0.02, failure=None, error=java.lang.RuntimeException!, skipped=False)
                     E       
                     E       - [ReportTestCase(name='test-a', time=0.02, failure=None, error=None, skipped=False),
                     E       -  ReportTestCase(name='test-b', time=0.02, failure=None, error='java.lang.RuntimeException!', skipped=False)]
                     E       ? ^                                                                                                          ^
                     E       
                     E       + [ReportTestCase(name='test-b', time=0.02, failure=None, error='java.lang.RuntimeException!', skipped=False),
                     E       ? ^                                                                                                          ^
                     E       
                     E       +  ReportTestCase(name='test-a', time=0.02, failure=None, error=None, skipped=False)]
                     
                     pants_test/backend/jvm/tasks/reports/test_junit_html_report.py:211: AssertionError
                     --------------- Captured log call ----------------
                     junit_html_report.py        77 WARNING  Found duplicate test case results in suite 'suite-a' from files: /tmp/tmpw2s5vziy/TEST-a-1.xml, /tmp/tmpw2s5vziy/TEST-a-2.xml, using first result:
                      -> ReportTestCase(name=test-a, time=0.02, failure=None, error=None, skipped=False)
                         ReportTestCase(name=test-a, time=0.01, failure=None, error=None, skipped=False)
                      generated xml file: /home/travis/build/pantsbuild/pants/.pants.d/test/pytest/tests.python.pants_test.backend.jvm.tasks.reports.reports/junitxml/TEST-tests.python.pants_test.backend.jvm.tasks.reports.reports.xml 
                     ============ slowest 3 test durations ============
                     0.01s call     pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_all
                     0.00s call     pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_no_open_report
                     0.00s call     pyprep/sources/af01dcc6bcb5f54a1c75d7e3db35c8053f549edc/tests/python/pants_test/backend/jvm/tasks/reports/test_junit_html_report.py::TestJUnitHtmlReport::test_open_report
                     ====== 1 failed, 12 passed in 0.91 seconds =======
```


Discussion:
And this looks like a random sort issue. I'll take a look to see what set or dict is making this happen.

Resolving patch:
diff --git a/src/python/pants/backend/jvm/tasks/reports/junit_html_report.py b/src/python/pants/backend/jvm/tasks/reports/junit_html_report.py
--- a/src/python/pants/backend/jvm/tasks/reports/junit_html_report.py
+++ b/src/python/pants/backend/jvm/tasks/reports/junit_html_report.py
@@ -101,7 +101,7 @@ def __init__(self, name, tests, errors, failures, skipped, time, testcases, file
     self.failures = int(failures)
     self.skipped = int(skipped)
     self.time = float(time)
-    self.testcases = testcases
+    self.testcases = sorted(testcases)  # Ensures a consistent reporting order.
     self.file = file
 
   def __lt__(self, other):
@@ -236,7 +236,7 @@ def _parse_xml_file(xml_file):
     root = ET.parse(xml_file).getroot()
 
     testcases = []
-    for testcase in sorted(root.iter('testcase'), key=lambda tc: tc.attrib['name']):
+    for testcase in root.iter('testcase'):
       failure = None
       for f in testcase.iter('failure'):
         failure = f.text


Test patch: