Repository: python-tap/tappy

Issue:
Sub tests are not included
If the unittests include [subtests](https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests) the results of these subtests are not picked up.

Also, if the test class contains only failing subtests, the tap file may not be generated at all.


Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=h1) Report
> Merging [#92](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=desc) into [master](https://codecov.io/gh/python-tap/tappy/commit/3d73aad083d56a888495f7056f4aff2b94dc4af2?src=pr&el=desc) will **decrease** coverage by `0.86%`.
> The diff coverage is `16.66%`.

[![Impacted file tree graph](https://codecov.io/gh/python-tap/tappy/pull/92/graphs/tree.svg?width=650&token=ULVfoGuR2p&height=150&src=pr)](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #92      +/-   ##
==========================================
- Coverage     100%   99.13%   -0.87%     
==========================================
  Files          12       12              
  Lines         569      575       +6     
==========================================
+ Hits          569      570       +1     
- Misses          0        5       +5
```


| [Impacted Files](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tap/runner.py](https://codecov.io/gh/python-tap/tappy/pull/92/diff?src=pr&el=tree#diff-dGFwL3J1bm5lci5weQ==) | `92.75% <16.66%> (-7.25%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=footer). Last update [3d73aad...70ab2cd](https://codecov.io/gh/python-tap/tappy/pull/92?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ecederstrand: I think coverage may be run on Python 2 where subtests aren't supported. I'm definitely hitting the new `addSubTest()` method in my local tests. Should I make the exposure of that method conditional? E.g.:

```python
class TAPTestResult(TextTestResult):
    if sys.version_info[0] >= 3:
        def addSubTest(self, test, subtest, err):
            ...
```
ecederstrand: Aslo, I believe I'm innocent of this test failure: https://travis-ci.org/python-tap/tappy/jobs/622716225?utm_medium=notification&utm_source=github_status
mblayman: Hi, @ecederstrand, thanks for the contribution! tappy is generally pretty stable so it's been a while since I kicked CI's tires.

Since Python 2 is EOL at the end of this month, this PR prompted me to start up another PR that will drop Python 2 support before the next release. I should get that merged tonight and get the macOS build working so you can merge master back into your PR and get a green build.

I'll give the code a review after Travis CI is happy. Thanks!
mblayman: Oh! I forgot to add that I pushed a couple of commits to your branch to save you the trouble of merging master. You'll need to pull locally (my apologies if you're a Git pro and know all this already. It's hard with first time contributor PRs to know ability level).
ecederstrand: No problem, and thanks for the help with the Python 2 cleanup :-)

Subtests are a way to specify that parts of a test case should be a test case in its own. It's usually done for a single assert in a larger test case:

```python
# test.py
import unittest

from tap import TAPTestRunner


class TestCase(unittest.TestCase):
    def test_subs(self):
        for i in range(4):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)


if __name__ == '__main__':
    runner = TAPTestRunner()
    runner.set_stream(True)
    unittest.main(testRunner=runner)
```

And here's an example of the output with this PR:

```bash
$ python test.py 
# TAP results for TestCase
ok 1 test_subs (__main__.TestCase) (i=0)
not ok 2 test_subs (__main__.TestCase) (i=1)
# Traceback (most recent call last):
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 59, in testPartExecutor
#     yield
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 523, in subTest
#     yield
#   File "test.py", line 10, in test_subs
#     self.assertEqual(i % 2, 0)
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 829, in assertEqual
#     assertion_func(first, second, msg=msg)
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 822, in _baseAssertEqual
#     raise self.failureException(msg)
# AssertionError: 1 != 0
ok 3 test_subs (__main__.TestCase) (i=2)
not ok 4 test_subs (__main__.TestCase) (i=3)
# Traceback (most recent call last):
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 59, in testPartExecutor
#     yield
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 523, in subTest
#     yield
#   File "test.py", line 10, in test_subs
#     self.assertEqual(i % 2, 0)
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 829, in assertEqual
#     assertion_func(first, second, msg=msg)
#   File "/opt/python35/lib/python3.5/unittest/case.py", line 822, in _baseAssertEqual
#     raise self.failureException(msg)
# AssertionError: 1 != 0
1..4
```
ecederstrand: I believe all requested changes are done now.

I noticed one more usage of `sys.version_info` in tap/tests/test_parser.py but left it untouched to not pollute this PR.
ecederstrand: Oh, and Python 3.4 is also EOL: https://devguide.python.org/devcycle/#end-of-life-branches

You may want to remove it from travis.yml

Patch:
diff --git a/AUTHORS b/AUTHORS
index 566404e..06d46c5 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -5,6 +5,7 @@ Contributors
 
 * Andrew McNamara
 * Chris Clarke
+* Erik Cederstrand
 * Marc Abramowitz
 * Mark E. Hamilton
 * Matt Layman
diff --git a/docs/releases.rst b/docs/releases.rst
index b04ab19..ee84c7e 100644
--- a/docs/releases.rst
+++ b/docs/releases.rst
@@ -5,6 +5,7 @@ Version 3.0, To Be Released
 ---------------------------
 
 * Drop support for Python 2 (it is end-of-life).
+* Add support for subtests.
 
 Version 2.6.2, Released October 20, 2019
 ----------------------------------------
diff --git a/tap/runner.py b/tap/runner.py
index 94134e1..10e8510 100644
--- a/tap/runner.py
+++ b/tap/runner.py
@@ -17,6 +17,18 @@ class TAPTestResult(TextTestResult):
     def __init__(self, stream, descriptions, verbosity):
         super(TAPTestResult, self).__init__(stream, descriptions, verbosity)
 
+    def addSubTest(self, test, subtest, err):
+        super(TAPTestResult, self).addSubTest(test, subtest, err)
+        if err is not None:
+            diagnostics = formatter.format_exception(err)
+            self.tracker.add_not_ok(
+                self._cls_name(test),
+                self._description(subtest),
+                diagnostics=diagnostics,
+            )
+        else:
+            self.tracker.add_ok(self._cls_name(test), self._description(subtest))
+
     def stopTestRun(self):
         """Once the test run is complete, generate each of the TAP files."""
         super(TAPTestResult, self).stopTestRun()


Test patch:
diff --git a/tap/tests/test_result.py b/tap/tests/test_result.py
index 06fb7e1..b5c1c9a 100644
--- a/tap/tests/test_result.py
+++ b/tap/tests/test_result.py
@@ -1,7 +1,9 @@
 # Copyright (c) 2019, Matt Layman and contributors
 
+import contextlib
 import os
 import unittest
+import unittest.case
 
 from tap.i18n import _
 from tap.runner import TAPTestResult
@@ -13,6 +15,14 @@ class FakeTestCase(unittest.TestCase):
     def runTest(self):
         pass
 
+    @contextlib.contextmanager
+    def subTest(self, *args, **kwargs):
+        try:
+            self._subtest = unittest.case._SubTest(self, object(), {})
+            yield
+        finally:
+            self._subtest = None
+
     def __call__(self, result):
         pass
 
@@ -70,3 +80,25 @@ class TestTAPTestResult(TestCase):
         self.assertEqual(
             line.directive.text, "TODO {}".format(_("(unexpected success)"))
         )
+
+    def test_adds_subtest_success(self):
+        """Test that the runner handles subtest success results."""
+        result = self._make_one()
+        test = FakeTestCase()
+        with test.subTest():
+            result.addSubTest(test, test._subtest, None)
+        line = result.tracker._test_cases["FakeTestCase"][0]
+        self.assertTrue(line.ok)
+
+    def test_adds_subtest_failure(self):
+        """Test that the runner handles subtest failure results."""
+        result = self._make_one()
+        # Python 3 does some extra testing in unittest on exceptions so fake
+        # the cause as if it were raised.
+        ex = Exception()
+        ex.__cause__ = None
+        test = FakeTestCase()
+        with test.subTest():
+            result.addSubTest(test, test._subtest, (ex.__class__, ex, None))
+        line = result.tracker._test_cases["FakeTestCase"][0]
+        self.assertFalse(line.ok)