Repository: pantsbuild/pants

Issue:
test.pytest shows more tests failed than actually failed
seen here: https://travis-ci.org/pantsbuild/pants/jobs/252109723

the pytest summary indicates a single test failure:

```
 1 failed, 1036 passed, 13 skipped, 1 xfailed, 2 xpassed in 342.35 seconds 
```

but according to the log output, all tests in the shard report as `FAILURE`:

```
tests.python.pants_test.build_graph.target                                      .....   FAILURE
tests.python.pants_test.build_graph.build_file_parser                           .....   FAILURE
tests.python.pants_test.build_graph.build_file_aliases                          .....   FAILURE
tests.python.pants_test.build_graph.address                                     .....   FAILURE
tests.python.pants_test.build_graph.build_file_address_mapper                   .....   FAILURE
tests.python.pants_test.build_graph.build_configuration                         .....   FAILURE
tests.python.pants_test.build_graph.source_mapper                               .....   FAILURE
tests.python.pants_test.build_graph.build_graph                                 .....   FAILURE
tests.python.pants_test.build_graph.scopes                                      .....   FAILURE
tests.python.pants_test.backend.codegen.antlr.java.java                         .....   FAILURE
tests.python.pants_test.backend.jvm.subsystems.jar_dependency_management        .....   FAILURE
tests.python.pants_test.backend.jvm.subsystems.shader                           .....   FAILURE
tests.python.pants_test.backend.jvm.subsystems.custom_scala                     .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.depmap                       .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.dependencies                 .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.filedeps                     .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.ide_gen                      .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.export                       .....   FAILURE
...
```

cc @jsirois 


Discussion:
A known compromise in the invalidation work. I'll think about how to make this better without stepping into results merging, which I had a strong preference for avoiding from the get go.
@jsirois : If it makes things any easier: I don't think there is any need to cache anything for failing tests... I wouldn't expect a run containing a failing test to have hit the cache at all.
No cache here. This is just  a run on N targets in fast mode that, w/o calculating target->test ownership, cannot say more than the whole run failed. The solution will just be making the code more complex to deal with fast vs no fast differently for display purposes.
Addressing this before finishing off caching for pytest in #4734.
OK - I think I have a decent compromise. The summary is now only output for `--no-fast` as was the case in the past. So:
1. `--fast`, 1 partition with all targets: no summary
2. `--no-fast`, 1 partition per-target: summary
3. multiple partitions with multiple targets each: no summary

The 3rd case is not good, but we also don't actually have this partitioning case today, so TODO added. 
Erm, except that means no output at all for `--fast` on "cached" success, ie:
```
...
08:50:11 00:01   [test]
08:50:11 00:01     [test-jvm-prep-command]
08:50:11 00:01       [jvm_prep_command]
08:50:11 00:01     [test-prep-command]
08:50:11 00:01     [test]
08:50:11 00:01     [pytest-prep]
08:50:11 00:01     [pytest]
08:50:12 00:02     [junit]
08:50:12 00:02     [go]
08:50:12 00:02     [node]
08:50:12 00:02   [complete]
               SUCCESS
```

Which looks alot like a faulty noop. May re-introduce the summary for `--fast`, but only on success.

Resolving patch:
diff --git a/src/python/pants/backend/python/tasks2/pytest_run.py b/src/python/pants/backend/python/tasks2/pytest_run.py
--- a/src/python/pants/backend/python/tasks2/pytest_run.py
+++ b/src/python/pants/backend/python/tasks2/pytest_run.py
@@ -526,9 +526,20 @@ def _run_tests(self, targets):
 
     for partition in sorted(results):
       rv = results[partition]
-      for target in partition:
+      if len(partition) == 1 or rv.success:
         log = self.context.log.info if rv.success else self.context.log.error
-        log('{0:80}.....{1:>10}'.format(target.id, rv))
+        for target in partition:
+          log('{0:80}.....{1:>10}'.format(target.id, rv))
+      else:
+        # There is not much useful we can display in summary for a multi-target partition with
+        # failures without parsing those failures to link them to individual targets; ie: targets
+        # 2 and 8 failed in this partition of 10 targets.
+        # TODO(John Sirois): Punting here works since we have in practice just 2 partitionings:
+        # 1. All targets in singleton partitions
+        # 2. All targets in 1 partition
+        # If we get to the point where we have multiple partitions with multiple targets, some sort
+        # of summary for the multi-target partitions will probably be needed.
+        pass
 
     failed_targets = [target
                       for _rv in results.values() if not _rv.success


Test patch: