Repository: HypothesisWorks/hypothesis

Issue:
Significant performance degradation for v3.30.4
I am noticing a roughly 3x performance penalty between v3.30.3 and v3.30.4 (despite the number of examples being halved). Since I'm also taking a 4x hit from #879 this is keeping me at v3.28.3 for now.

An example is a test that uses this hypothesis statement:

```
@given(ids=st.lists(st.text(alphabet=string.ascii_lowercase)))
```

Running under v3.30.3 these are the run statistics:

```
=================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.6.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /Users/dan/.pyenv/versions/3.6.3/envs/pstat/bin/python
cachedir: .cache
rootdir: /Users/dan/w/pstat, inifile:
plugins: xdist-1.20.1, mock-1.6.3, forked-0.2, cov-2.5.1, hypothesis-3.30.3
collected 28 items

test/test_board.py::test_get_node_devices_calls_device PASSED
================================================================================== Hypothesis Statistics ==================================================================================

test/test_board.py::test_get_node_devices_calls_device:

  - 200 passing examples, 0 failing examples, 0 invalid examples
  - Typical runtimes: 4-17 ms
  - Stopped because settings.max_examples=200

=================================================================================== 27 tests deselected ===================================================================================
========================================================================= 1 passed, 27 deselected in 3.33 seconds =========================================================================
```

And when only changing to v3.30.4:

```=================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.6.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /Users/dan/.pyenv/versions/3.6.3/envs/pstat/bin/python
cachedir: .cache
rootdir: /Users/dan/w/pstat, inifile:
plugins: xdist-1.20.1, mock-1.6.3, forked-0.2, cov-2.5.1, hypothesis-3.30.4
collected 28 items

test/test_board.py::test_get_node_devices_calls_device PASSED
================================================================================== Hypothesis Statistics ==================================================================================

test/test_board.py::test_get_node_devices_calls_device:

  - 100 passing examples, 0 failing examples, 0 invalid examples
  - Typical runtimes: 4-14 ms
  - Stopped because settings.max_examples=100

=================================================================================== 27 tests deselected ===================================================================================
======================================================================== 1 passed, 27 deselected in 10.20 seconds =========================================================================
```

Notice that even though the typical run time is roughly the same, the execution time triples. Here is the profiling for v3.30.3


```
Profiling (from /Users/dan/w/pstat/prof/combined.prof):
Fri Oct  6 18:03:32 2017    /Users/dan/w/pstat/prof/combined.prof

         10990129 function calls (10962229 primitive calls) in 5.074 seconds

   Ordered by: cumulative time
   List reduced from 741 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      4/1    0.000    0.000    5.077    5.077 pluggy.py:263(__init__)
      5/1    0.000    0.000    5.077    5.077 pluggy.py:598(execute)
      3/1    0.000    0.000    5.077    5.077 pluggy.py:238(_wrapped_call)
        1    0.000    0.000    5.077    5.077 runner.py:109(pytest_runtest_call)
        1    0.000    0.000    5.077    5.077 python.py:1167(runtest)
        1    0.000    0.000    5.077    5.077 pluggy.py:743(__call__)
        1    0.000    0.000    5.077    5.077 pluggy.py:336(_hookexec)
        1    0.000    0.000    5.077    5.077 pluggy.py:333(<lambda>)
        1    0.000    0.000    5.077    5.077 python.py:133(pytest_pyfunc_call)
        1    0.017    0.017    5.077    5.077 core.py:931(wrapped_test)
        1    0.000    0.000    5.020    5.020 core.py:747(run)
        1    0.000    0.000    5.020    5.020 engine.py:389(run)
        1    0.004    0.004    5.020    5.020 engine.py:654(_run)
      200    0.191    0.001    5.004    0.025 engine.py:134(test_function)
      200    0.159    0.001    2.605    0.013 core.py:680(evaluate_test_data)
   212115    0.203    0.000    1.921    0.000 {method 'get' of 'dict' objects}
   800081    0.936    0.000    1.834    0.000 _make.py:415(_attrs_to_tuple)
   401321    0.202    0.000    1.259    0.000 _make.py:431(hash_)
   199380    0.153    0.000    1.113    0.000 _make.py:454(eq)
   398760    0.127    0.000    0.961    0.000 _make.py:448(attrs_to_tuple)
```
And v3.30.4:

```Profiling (from /Users/dan/w/pstat/prof/combined.prof):
Fri Oct  6 18:03:59 2017    /Users/dan/w/pstat/prof/combined.prof

         49196866 function calls (48286483 primitive calls) in 18.292 seconds

   Ordered by: cumulative time
   List reduced from 803 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      4/1    0.000    0.000   18.294   18.294 pluggy.py:263(__init__)
      5/1    0.000    0.000   18.294   18.294 pluggy.py:598(execute)
      3/1    0.000    0.000   18.294   18.294 pluggy.py:238(_wrapped_call)
        1    0.000    0.000   18.294   18.294 runner.py:109(pytest_runtest_call)
        1    0.000    0.000   18.294   18.294 python.py:1167(runtest)
        1    0.000    0.000   18.294   18.294 pluggy.py:743(__call__)
        1    0.000    0.000   18.294   18.294 pluggy.py:336(_hookexec)
        1    0.000    0.000   18.294   18.294 pluggy.py:333(<lambda>)
        1    0.000    0.000   18.294   18.294 python.py:133(pytest_pyfunc_call)
        1    0.004    0.004   18.294   18.294 core.py:931(wrapped_test)
        1    0.000    0.000   18.221   18.221 core.py:747(run)
        1    0.000    0.000   18.221   18.221 engine.py:379(run)
        1    0.000    0.000   18.221   18.221 engine.py:751(_run)
        1    0.003    0.003   17.494   17.494 engine.py:649(generate_new_examples)
      100    0.155    0.002   12.739    0.127 engine.py:121(test_function)
   204737    1.026    0.000   10.680    0.000 engine.py:1259(rescore)
  5260988    4.936    0.000   10.255    0.000 _make.py:415(_attrs_to_tuple)
      100    0.276    0.003    9.566    0.096 engine.py:1211(add)
2803366/2458931    1.392    0.000    7.510    0.000 _make.py:431(hash_)
1228811/1228810    0.933    0.000    6.352    0.000 _make.py:454(eq)
```

Do you think this is a hypothesis issue or could it be something I'm doing to cause the degradation?


Discussion:
Sorry, I should have thought to try this earlier, but I discovered just now that setting `use_coverage=False` gets me back to v3.28.3 performance levels:

```
=================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.6.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /Users/dan/.pyenv/versions/3.6.3/envs/pstat/bin/python
cachedir: .cache
rootdir: /Users/dan/w/pstat, inifile:
plugins: xdist-1.20.1, profiling-1.2.11, mock-1.6.3, forked-0.2, hypothesis-3.30.4
collected 28 items

test/test_board.py::test_get_node_devices_calls_device PASSED
================================================================================== Hypothesis Statistics ==================================================================================

test/test_board.py::test_get_node_devices_calls_device:

  - 100 passing examples, 0 failing examples, 0 invalid examples
  - Typical runtimes: 0-5 ms
  - Stopped because settings.max_examples=100

=================================================================================== 27 tests deselected ===================================================================================
========================================================================= 1 passed, 27 deselected in 0.29 seconds =========================================================================
```
> Do you think this is a hypothesis issue or could it be something I'm doing to cause the degradation?

It seems very unlikely this is something you're doing based on the profiling output you provided (thanks for that BTW! It's extremely helpful).

Do you have any sort of reproducible test case you could share? I'll try to take a look into why this might be happening, but knowing what the test it's running is would be quite helpful, as I suspect this is at least partly a function of how much/what gets covered.

Resolving patch:
diff --git a/src/hypothesis/core.py b/src/hypothesis/core.py
--- a/src/hypothesis/core.py
+++ b/src/hypothesis/core.py
@@ -551,11 +551,26 @@ def escalate_warning(msg, slug=None):  # pragma: no cover
     )
 
 
-@attr.s(slots=True, frozen=True)
 class Arc(object):
-    filename = attr.ib()
-    source = attr.ib()
-    target = attr.ib()
+    __slots__ = ('filename', 'source', 'target')
+
+    def __init__(self, filename, source, target):
+        self.filename = filename
+        self.source = source
+        self.target = target
+
+
+ARC_CACHE = {}
+
+
+def arc(filename, source, target):
+    try:
+        return ARC_CACHE[filename][source][target]
+    except KeyError:
+        result = Arc(filename, source, target)
+        ARC_CACHE.setdefault(
+            filename, {}).setdefault(source, {})[target] = result
+        return result
 
 
 in_given = False
@@ -720,7 +735,7 @@ def evaluate_test_data(self, data):
                         if is_hypothesis_file(filename):
                             continue
                         data.tags.update(
-                            Arc(filename, source, target)
+                            arc(filename, source, target)
                             for source, target in covdata.arcs(filename)
                         )
             if result is not None and self.settings.perform_health_check:
diff --git a/src/hypothesis/internal/conjecture/engine.py b/src/hypothesis/internal/conjecture/engine.py
--- a/src/hypothesis/internal/conjecture/engine.py
+++ b/src/hypothesis/internal/conjecture/engine.py
@@ -24,8 +24,6 @@
 from weakref import WeakKeyDictionary
 from collections import defaultdict
 
-import attr
-
 from hypothesis import settings as Settings
 from hypothesis import Phase
 from hypothesis.reporting import debug_report
@@ -138,10 +136,7 @@ def test_function(self, data):
 
         self.debug_data(data)
 
-        tags = frozenset(
-            self.tag_intern_table.setdefault(t, t)
-            for t in data.tags
-        )
+        tags = frozenset(data.tags)
         data.tags = self.tag_intern_table.setdefault(tags, tags)
 
         if data.status == Status.VALID:
@@ -1135,9 +1130,23 @@ def choice(self, random):
         return random.choice(self.__values)
 
 
-@attr.s(slots=True, hash=True, cmp=True)
 class Negated(object):
-    tag = attr.ib()
+    __slots__ = ('tag',)
+
+    def __init__(self, tag):
+        self.tag = tag
+
+
+NEGATED_CACHE = {}
+
+
+def negated(tag):
+    try:
+        return NEGATED_CACHE[tag]
+    except KeyError:
+        result = Negated(tag)
+        NEGATED_CACHE[tag] = result
+        return result
 
 
 universal = UniqueIdentifier('universal')
@@ -1232,7 +1241,7 @@ def add(self, data):
 
         for t in new_tags:
             self.non_universal_tags.add(t)
-            self.examples_by_tags[Negated(t)] = list(
+            self.examples_by_tags[negated(t)] = list(
                 self.examples_by_tags[universal]
             )
 
@@ -1254,7 +1263,7 @@ def tags_for(self, data):
             yield t
         for t in self.non_universal_tags:
             if t not in data.tags:
-                yield Negated(t)
+                yield negated(t)
 
     def rescore(self, tag):
         new_score = (


Test patch:
diff --git a/tests/cover/test_core.py b/tests/cover/test_core.py
--- a/tests/cover/test_core.py
+++ b/tests/cover/test_core.py
@@ -24,6 +24,7 @@
 
 import hypothesis.strategies as s
 from hypothesis import find, given, reject, settings
+from hypothesis.core import arc
 from hypothesis.errors import NoSuchExample, Unsatisfiable
 from tests.common.utils import checks_deprecated_behaviour
 
@@ -98,3 +99,7 @@ def test_settings_are_default_in_find():
     find(
         s.booleans(), lambda x: settings.default is some_normal_settings,
         settings=some_normal_settings)
+
+
+def test_arc_is_memoized():
+    assert arc('foo', 1, 2) is arc('foo', 1, 2)