Repository: HypothesisWorks/hypothesis

Issue:
Refactor pytest plugin statistics handling for strict JUnit xml validation
pytest-dev/pytest#5202 indicates that using the `record_property` fixture leads Pytest to generate an XML file which is not compliant with some strict schemas.  We don't actually use this fixture, but we *do* hook our statistics data into the underlying data structure so it gets passed between xdist workers correctly.

pytest-dev/pytest#5205 proposes to add a fixture with a better way, but we don't need to use the fixture when we can hook into the underlying method directly (which has the advantage of working on earlier versions of Pytest too).

So, [in our pytest plugin](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/extra/pytestplugin.py), I think we could replace the use of `user_properties` with `getattr(request.config, "_xml", None).add_global_property(name, value)` *if* the `_xml` attribute exists.  Needs some manual exploration and testing to confirm the idea is sound first, but why *not* add yet another path for statistics data?  

(it will also need to be tested, for example, both with and without xdist and checking the resulting xml for content and strict schema compliance)


Discussion:
Proof of concept: https://github.com/HypothesisWorks/hypothesis/compare/master...Zac-HD:interoperable-stats

Valid against the `xunit2` xml schema, but to compensate this introduces a bunch of string escaping issues which... there doesn't seem to be a nice way around :crying_cat_face: 

Not planning to keep working on this, but PRs or sponsored features are always an option.
We've just stumbled upon this issue, because we wanted to remove a custom `xsltproc` script, that we use for post-processing pytest.xml (xunit1) files for making the Jenkins CI happy ;)

Are there any updates on this issue besides a POC implementation, that AFAIC doesn't use the `record_testsuite_property` fixture introduced by `pytest-4.5`.  @Zac-HD you mentioned that your POC implementation/fix has some string escaping issues. I guess that we can't magically solve these issues by using the new pytest fixture, right? (I haven't looked at the implementation of `record_testsuite_property`) 
The POC already uses the (underlying implementation of) the new fixture; the problem is that XML-escaping `"` as `&quot;` etc. causes problems when certain characters or escape sequences are part of the actual statistics to report.  Which, come to think of it, we could get around just by base64-encoding the string before adding it to the xml... there would be opaque blobs in the report, but that's better than the status quo I guess!

So all we need is for someone to write a whole bunch of tests for this, which I am unmotivated to do at the moment while I'm busy getting [HypoFuzz](https://hypofuzz.com/) off the ground.  Happy to help with and review a PR if someone on your team could write it (based on [these tests](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/tests/pytest/test_statistics.py) and whatever xunit xml validation you think appropriate), or as above I could make time if sponsored.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/extra/pytestplugin.py b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
--- a/hypothesis-python/src/hypothesis/extra/pytestplugin.py
+++ b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
@@ -13,6 +13,7 @@
 #
 # END HEADER
 
+import base64
 from inspect import signature
 
 import pytest
@@ -196,7 +197,9 @@ def pytest_runtest_call(item):
 
             def note_statistics(stats):
                 stats["nodeid"] = item.nodeid
-                item.hypothesis_statistics = describe_statistics(stats)
+                item.hypothesis_statistics = base64.b64encode(
+                    describe_statistics(stats).encode()
+                ).decode()
 
             with collector.with_value(note_statistics):
                 with with_reporter(store):
@@ -212,20 +215,39 @@ def pytest_runtest_makereport(item, call):
                 ("Hypothesis", "\n".join(item.hypothesis_report_information))
             )
         if hasattr(item, "hypothesis_statistics") and report.when == "teardown":
-            val = ("hypothesis-stats", item.hypothesis_statistics)
-            report.user_properties.append(val)
+            name = "hypothesis-statistics-" + item.nodeid
+            try:
+                item.config._xml.add_global_property(name, item.hypothesis_statistics)
+            except AttributeError:
+                # --junitxml not passed, or Pytest 4.5 (before add_global_property)
+                # We'll fail xunit2 xml schema checks, upgrade pytest if you care.
+                report.user_properties.append((name, item.hypothesis_statistics))
 
     def pytest_terminal_summary(terminalreporter):
         if not terminalreporter.config.getoption(PRINT_STATISTICS_OPTION):
             return
         terminalreporter.section("Hypothesis Statistics")
-        # terminalreporter.stats is a dict, where the empty string appears to
-        # always be the key for a list of _pytest.reports.TestReport objects
-        # (where we stored the statistics data in pytest_runtest_makereport above)
-        for test_report in terminalreporter.stats.get("", []):
-            for name, value in test_report.user_properties:
-                if name == "hypothesis-stats" and test_report.when == "teardown":
-                    terminalreporter.write_line(value + "\n\n")
+
+        def report(properties):
+            for name, value in properties:
+                if name.startswith("hypothesis-statistics-"):
+                    if hasattr(value, "uniobj"):
+                        # Under old versions of pytest, `value` was a `py.xml.raw`
+                        # rather than a string, so we get the (unicode) string off it.
+                        value = value.uniobj
+                    line = base64.b64decode(value.encode()).decode() + "\n\n"
+                    terminalreporter.write_line(line)
+
+        try:
+            global_properties = terminalreporter.config._xml.global_properties
+        except AttributeError:
+            # terminalreporter.stats is a dict, where the empty string appears to
+            # always be the key for a list of _pytest.reports.TestReport objects
+            for test_report in terminalreporter.stats.get("", []):
+                if test_report.when == "teardown":
+                    report(test_report.user_properties)
+        else:
+            report(global_properties)
 
     def pytest_collection_modifyitems(items):
         for item in items:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/dfa/__init__.py b/hypothesis-python/src/hypothesis/internal/conjecture/dfa/__init__.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/dfa/__init__.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/dfa/__init__.py
@@ -590,7 +590,7 @@ def __init__(self, transitions, accepting, start=0):
           (in which case they map characters to other states) or lists. If they
           are a list they may contain tuples of length 2 or 3. A tuple ``(c, j)``
           indicates that this state transitions to state ``j`` given ``c``. A
-          tuple ``(u, v, j)`` indicates this state transitiosn to state ``j``
+          tuple ``(u, v, j)`` indicates this state transitions to state ``j``
           given any ``c`` with ``u <= c <= v``.
         * ``accepting`` is a set containing the integer labels of accepting
           states.


Test patch:
diff --git a/hypothesis-python/tests/conjecture/test_dfa.py b/hypothesis-python/tests/conjecture/test_dfa.py
--- a/hypothesis-python/tests/conjecture/test_dfa.py
+++ b/hypothesis-python/tests/conjecture/test_dfa.py
@@ -89,6 +89,13 @@ def dfas(draw):
 
 @settings(max_examples=20)
 @given(dfas(), st.booleans())
+@example(
+    ConcreteDFA(
+        transitions=[[(0, 2), (1, 255, 1)], [(0, 2), (1, 255, 0)], []],
+        accepting={2},
+    ),
+    False,
+)
 def test_canonicalised_matches_same_strings(dfa, via_repr):
     canon = dfa.canonicalise()
     note(canon)
diff --git a/hypothesis-python/tests/pytest/test_statistics.py b/hypothesis-python/tests/pytest/test_statistics.py
--- a/hypothesis-python/tests/pytest/test_statistics.py
+++ b/hypothesis-python/tests/pytest/test_statistics.py
@@ -22,6 +22,12 @@
 pytest_plugins = "pytester"
 
 
+def get_output(testdir, suite, *args):
+    script = testdir.makepyfile(suite)
+    result = testdir.runpytest(script, *args)
+    return "\n".join(result.stdout.lines)
+
+
 TESTSUITE = """
 from hypothesis import HealthCheck, given, settings, assume
 from hypothesis.strategies import integers
@@ -45,26 +51,36 @@ def test_iterations(x):
 
 
 def test_does_not_run_statistics_by_default(testdir):
-    script = testdir.makepyfile(TESTSUITE)
-    result = testdir.runpytest(script)
-    out = "\n".join(result.stdout.lines)
+    out = get_output(testdir, TESTSUITE)
     assert "Hypothesis Statistics" not in out
 
 
 def test_prints_statistics_given_option(testdir):
-    script = testdir.makepyfile(TESTSUITE)
-    result = testdir.runpytest(script, PRINT_STATISTICS_OPTION)
-    out = "\n".join(result.stdout.lines)
+    out = get_output(testdir, TESTSUITE, PRINT_STATISTICS_OPTION)
     assert "Hypothesis Statistics" in out
     assert "max_examples=100" in out
     assert "< 10% of examples satisfied assumptions" in out
 
 
-@pytest.mark.skipif(LooseVersion(pytest.__version__) < "3.5", reason="too old")
 def test_prints_statistics_given_option_under_xdist(testdir):
-    script = testdir.makepyfile(TESTSUITE)
-    result = testdir.runpytest(script, PRINT_STATISTICS_OPTION, "-n", "2")
-    out = "\n".join(result.stdout.lines)
+    out = get_output(testdir, TESTSUITE, PRINT_STATISTICS_OPTION, "-n", "2")
+    assert "Hypothesis Statistics" in out
+    assert "max_examples=100" in out
+    assert "< 10% of examples satisfied assumptions" in out
+
+
+def test_prints_statistics_given_option_with_junitxml(testdir):
+    out = get_output(testdir, TESTSUITE, PRINT_STATISTICS_OPTION, "--junit-xml=out.xml")
+    assert "Hypothesis Statistics" in out
+    assert "max_examples=100" in out
+    assert "< 10% of examples satisfied assumptions" in out
+
+
+@pytest.mark.skipif(LooseVersion(pytest.__version__) < "4.6", reason="too old")
+def test_prints_statistics_given_option_under_xdist_with_junitxml(testdir):
+    out = get_output(
+        testdir, TESTSUITE, PRINT_STATISTICS_OPTION, "-n", "2", "--junit-xml=out.xml"
+    )
     assert "Hypothesis Statistics" in out
     assert "max_examples=100" in out
     assert "< 10% of examples satisfied assumptions" in out