Repository: HypothesisWorks/hypothesis

Issue:
Long-running fuzzing mode to discover bugs
Currently Hypothesis is intended for being run as part of a small test suite of reasonably fast tests. This is great and all, but particularly given the coverage based discovery features what you really want is to be able to part Hypothesis on a server somewhere and just have it sitting there 24/7 looking for bugs in your code, then run only the corpus of examples that it built up there every time you do a CI run.

Breadth first mode
I am using the fantastic Hypothesis framework (thank you very much) extensively. Locally I have `max_examples=10` and in CI I have `max_examples=200` to control the runtime of the test suite.

From time to time I would like to run extensive tests that would run very long, say, over night. Currently I estimate an appropriate value of `max_examples` based  the current run time and on the time I am willing to spend. However, that does not work out all the time. Sometimes I misestimate, sometimes I do not know how much time I could spend, e.g. over lunch break.

I would like to have a breadth first mode. In this mode, hypothesis would have no upper limit for `max_examples` but instead run all tests in the same proportion, forever. Thus, if I have 10 tests and an hour of runtime, Hypothesis would have executed each test 1,000 times instead of 5 tests 20,000 times and the others not at all.

This is very similar to #171, but there are subtle differences. In particular, in this scenario it is okay to stop at the first error.


Discussion:
This is almost exactly my use case. Is there a "next step" that I can contribute, please?
This isn't really a feature I'm up for someone else taking the lead on I'm afraid. It's big, touches on a lot of issues, and is pretty central to my long-term plans for the project.

The only real next step that someone who isn't me can take on for this is finding a way to pay me to work on it it so I can move it up my priority queue, and I'm overcommitted enough right now that even that will have a fairly long lead time.
:broken_heart:
> 💔

The suggestion of funding it *is* a serious offer. If this is really something that you need, Google isn't exactly short on cash...
I'm attempting a hypothesis project that would justify such a thing, but I can't before it's successful. Given that I really want such a feature, what should I do?
#1567 makes Hypothesis suitable for long (x hours per test) running processes, and the standard database workflow will ensure that discovered bugs can be re-run quickly.  There's plenty more to do though!
There was a short [student project](https://docs.google.com/document/d/1DjizLW_Pb_h9zoHWIqWAvaq1kAAi1dLoMNBMhjP7Dhk/edit) at Imperial College to implement a fuzzing mode for Hypothesis.  This did not result in production-ready code, but did suggest the following approach:

1. Always keep running until every known bug has been fully minimised,
   *(simplifying the standard life-cycle for consistency across modes)*
2. Have an internal flag to restart the cycle after bugs are fully minimised, 
   *(i.e. the "keep fuzzing" mode)*
3. Provide some way for test cases from multiple tests to run interleaved or in parallel.
   *(so we can fuzz better than 'one test per process')*

It's still not the proverbial "simple matter of code", but we at least have some idea what to do.
Before writing our own fuzzing engine, we should start by exposing a way to treat Hypothesis tests as fuzz targets - i.e. a wrapper which takes `bytes` and invokes the test function once - which can be executed by *any* engine.  This is much easier to support, a better match for the scope of our project, and allows both us and users to experiement with a variety of different approaches and tools.  

[`python-afl`](https://github.com/jwilk/python-afl) would only need a small shim, and [`pythonfuzz`](https://github.com/fuzzitdev/pythonfuzz) or [services like `fuzzbuzz`](https://docs.fuzzbuzz.io/getting-started/find-your-first-bug-in-python) wouldn't even need that, so IMO we should keep such an interface even if we do end up developing our own engine.

Post-MVP we'll want to think about e.g whether we need a new `Phase` (or similar) for this, how it integrates with `settings` such as timeouts and healthchecks, and especially how the example database should work.  Definitely good to get an MVP first though...


Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -19,6 +19,7 @@
 import contextlib
 import datetime
 import inspect
+import io
 import random as rnd_module
 import sys
 import traceback
@@ -27,7 +28,7 @@
 from inspect import getfullargspec
 from io import StringIO
 from random import Random
-from typing import Any, Callable, Hashable, List, TypeVar, Union
+from typing import Any, BinaryIO, Callable, Hashable, List, Optional, TypeVar, Union
 from unittest import TestCase
 
 import attr
@@ -376,7 +377,7 @@ def get_random_for_wrapped_test(test, wrapped_test):
 
 
 def process_arguments_to_given(
-    wrapped_test, arguments, kwargs, given_kwargs, argspec, test, settings,
+    wrapped_test, arguments, kwargs, given_kwargs, argspec, settings,
 ):
     selfy = None
     arguments, kwargs = convert_positional_arguments(wrapped_test, arguments, kwargs)
@@ -874,6 +875,7 @@ class HypothesisHandle:
     """
 
     inner_test = attr.ib()
+    fuzz_one_input = attr.ib()
 
 
 def given(
@@ -918,6 +920,27 @@ def run_test_as_given(test):
 
         argspec = new_given_argspec(original_argspec, given_kwargs)
 
+        # Use type information to convert "infer" arguments into appropriate strategies.
+        if infer in given_kwargs.values():
+            hints = get_type_hints(test)
+        for name in [name for name, value in given_kwargs.items() if value is infer]:
+            if name not in hints:
+                # As usual, we want to emit this error when the test is executed,
+                # not when it's decorated.
+
+                @impersonate(test)
+                @define_function_signature(test.__name__, test.__doc__, argspec)
+                def wrapped_test(*arguments, **kwargs):
+                    __tracebackhide__ = True
+                    raise InvalidArgument(
+                        "passed %s=infer for %s, but %s has no type annotation"
+                        % (name, test.__name__, name)
+                    )
+
+                return wrapped_test
+
+            given_kwargs[name] = st.from_type(hints[name])
+
         @impersonate(test)
         @define_function_signature(test.__name__, test.__doc__, argspec)
         def wrapped_test(*arguments, **kwargs):
@@ -943,22 +966,8 @@ def wrapped_test(*arguments, **kwargs):
 
             random = get_random_for_wrapped_test(test, wrapped_test)
 
-            # Use type information to convert "infer" arguments into appropriate
-            # strategies.
-            if infer in given_kwargs.values():
-                hints = get_type_hints(test)
-            for name in [
-                name for name, value in given_kwargs.items() if value is infer
-            ]:
-                if name not in hints:
-                    raise InvalidArgument(
-                        "passed %s=infer for %s, but %s has no type annotation"
-                        % (name, test.__name__, name)
-                    )
-                given_kwargs[name] = st.from_type(hints[name])
-
             processed_args = process_arguments_to_given(
-                wrapped_test, arguments, kwargs, given_kwargs, argspec, test, settings,
+                wrapped_test, arguments, kwargs, given_kwargs, argspec, settings,
             )
             arguments, kwargs, test_runner, search_strategy = processed_args
 
@@ -1080,6 +1089,45 @@ def wrapped_test(*arguments, **kwargs):
                     )
                     raise the_error_hypothesis_found
 
+        def fuzz_one_input(
+            buffer: Union[bytes, bytearray, memoryview, BinaryIO]
+        ) -> Optional[bytes]:
+            """Run the test as a fuzz target, driven with the `buffer` of bytes.
+
+            Returns None if buffer invalid for the strategy, canonical pruned
+            bytes if the buffer was valid, and leaves raised exceptions alone.
+
+            Note: this feature is experimental and may change or be removed.
+            """
+            if isinstance(buffer, io.IOBase):
+                buffer = buffer.read()
+            if isinstance(buffer, (bytearray, memoryview)):
+                buffer = bytes(buffer)
+            assert isinstance(buffer, bytes)
+
+            test = wrapped_test.hypothesis.inner_test
+            settings = wrapped_test._hypothesis_internal_use_settings
+            random = get_random_for_wrapped_test(test, wrapped_test)
+            _args, _kwargs, test_runner, search_strategy = process_arguments_to_given(
+                wrapped_test, (), {}, given_kwargs, argspec, settings,
+            )
+            assert not _args
+            assert not _kwargs
+            state = StateForActualGivenExecution(
+                test_runner, search_strategy, test, settings, random, wrapped_test,
+            )
+
+            data = ConjectureData.for_buffer(buffer)
+            try:
+                state.execute_once(data)
+            except (StopTest, UnsatisfiedAssumption):
+                return None
+            except BaseException:
+                if settings.database is not None:
+                    settings.database.save(function_digest(test), bytes(data.buffer))
+                raise
+            return bytes(data.buffer)
+
         # After having created the decorated test function, we need to copy
         # over some attributes to make the switch as seamless as possible.
 
@@ -1099,7 +1147,7 @@ def wrapped_test(*arguments, **kwargs):
         wrapped_test._hypothesis_internal_use_reproduce_failure = getattr(
             test, "_hypothesis_internal_use_reproduce_failure", None
         )
-        wrapped_test.hypothesis = HypothesisHandle(test)
+        wrapped_test.hypothesis = HypothesisHandle(test, fuzz_one_input)
         return wrapped_test
 
     return run_test_as_given


Test patch:
diff --git a/hypothesis-python/tests/cover/test_fuzz_one_input.py b/hypothesis-python/tests/cover/test_fuzz_one_input.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/cover/test_fuzz_one_input.py
@@ -0,0 +1,110 @@
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2020 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+import io
+import random
+from operator import attrgetter
+
+import pytest
+
+from hypothesis import Phase, given, settings, strategies as st
+from hypothesis.database import InMemoryExampleDatabase
+from hypothesis.internal.conjecture.shrinker import sort_key
+
+
+@pytest.mark.parametrize(
+    "buffer_type",
+    [bytes, bytearray, memoryview, io.BytesIO],
+    ids=attrgetter("__name__"),
+)
+def test_fuzz_one_input(buffer_type):
+    db = InMemoryExampleDatabase()
+    seen = []
+    seeds = []
+
+    # This is a standard `@given` test, which we can also use as a fuzz target.
+    # Note that we specify the DB so we can make more precise assertions,
+    # and tighten the phases so we can be sure the failing examples come from fuzzing.
+    @given(st.text())
+    @settings(database=db, phases=[Phase.reuse, Phase.shrink])
+    def test(s):
+        seen.append(s)
+        assert "\0" not in s, repr(s)
+
+    # Before running fuzz_one_input, there's nothing in `db`, and so the test passes
+    # (because example generation is disabled by the custom settings)
+    test()
+    assert len(seen) == 0
+
+    # If we run a lot of random bytestrings through fuzz_one_input, we'll eventually
+    # find a failing example.
+    with pytest.raises(AssertionError):
+        for _ in range(1000):
+            buf = bytes(random.getrandbits(8) for _ in range(1000))
+            seeds.append(buf)
+            test.hypothesis.fuzz_one_input(buffer_type(buf))
+
+    # fuzz_one_input returns False for invalid bytestrings, due to e.g. assume(False)
+    assert len(seen) <= len(seeds)
+
+    # `db` contains exactly one failing example, which is either the most
+    # recent seed that we tried or the pruned-and-canonicalised form of it.
+    (saved_examples,) = db.data.values()
+    assert len(saved_examples) == 1
+    assert sort_key(seeds[-1]) >= sort_key(list(saved_examples)[0])
+
+    # Now that we have a failure in `db`, re-running our test is sufficient to
+    # reproduce it, *and shrink to a minimal example*.
+    with pytest.raises(AssertionError):
+        test()
+    assert seen[-1] == "\0"
+
+
+def test_can_fuzz_with_database_eq_None():
+    # This test exists to cover the can't-record-failure branch.
+
+    @given(st.none())
+    @settings(database=None)
+    def test(s):
+        assert False
+
+    with pytest.raises(AssertionError):
+        test.hypothesis.fuzz_one_input(b"\x00\x00")
+
+
+def test_fuzzing_unsatisfiable_test_always_returns_None():
+    # There are no examples of `st.none().filter(bool)`, but while the Hypothesis
+    # engine would give up, fuzz_one_input will just return None each time.
+
+    @given(st.none().filter(bool))
+    @settings(database=None)
+    def test(s):
+        assert False
+
+    for _ in range(100):
+        buf = bytes(random.getrandbits(8) for _ in range(3))
+        ret = test.hypothesis.fuzz_one_input(buf)
+        assert ret is None
+
+
+def test_autopruning_of_returned_buffer():
+    @given(st.binary(min_size=4, max_size=4))
+    @settings(database=None)
+    def test(s):
+        pass
+
+    # Unused portions of the input buffer are discarded from output.
+    # (and canonicalised, but that's a no-op for fixed-length `binary()`)
+    assert test.hypothesis.fuzz_one_input(b"deadbeef") == b"dead"