Package: icontract-hypothesis
icontract-hypothesisIcontract-hypothesis combines design-by-contract with automatic testing.It is an integration betweenicontractlibrary for design-by-contract andHypothesislibrary for
property-based testing.The result is a powerful combination that allows you to automatically test
your code. Instead of writing manually the Hypothesis search strategies for
a function, icontract-hypothesis infers them based on
the function’s precondition. This makes automatic testing as effortless as it
goes.You can use icontract-hypothesis:As a library, to write succinct unit tests,As a command-line tool or as a tool integrated in your IDE
(e.g.,icontract-hypothesis-vim,icontract-hypothesis-pycharmandicontract-hypothesis-vscode).This allows you to automatically test functions during the development and
use it in your continuous integration,As a ghostwriter utility giving you a starting point for your more elaborate
Hypothesis strategies.Since the contracts live close to the code, evolving the code also automatically
evolves the tests.UsageLibraryThere are two ways to integrate icontract-hypothesis in your tests as a library.Only assume.First, you can use it for defining the assumptions of the test based on the
precondition:>>>fromhypothesisimportgiven>>>importhypothesis.strategiesasst>>>importicontract>>>importicontract_hypothesis>>>@icontract.require(lambdax:x>0)...@icontract.ensure(lambdaresult:result>0)...defsome_func(x:int)->int:...returnx-1000>>>assume_preconditions=icontract_hypothesis.make_assume_preconditions(...some_func)>>>@given(x=st.integers())...deftest_some_func(x:int)->None:...assume_preconditions(x)...some_func(x)>>>test_some_func()Traceback(mostrecentcalllast):...icontract.errors.ViolationError:File<doctestREADME.rst[4]>,line2in<module>:result>0:resultwas-999xwas1The functionassume_preconditionscreated byicontract_hypothesis.make_assume_preconditionswill reject all the input
values which do not satisfy the pre-conditions ofsome_func.Infer strategy. Second, you can automatically infer the strategy and test the function:>>>importicontract>>>importicontract_hypothesis>>>@icontract.require(lambdax:x>0)...@icontract.ensure(lambdaresult:result>0)...defsome_func(x:int)->int:...returnx-1000>>>icontract_hypothesis.test_with_inferred_strategy(some_func)Traceback(mostrecentcalllast):...icontract.errors.ViolationError:File<doctestREADME.rst[10]>,line2in<module>:result>0:resultwas-999xwas1Which approach to use depends on how you want to write your tests.
The first approach, usingassume_preconditions, is practical if you already
defined your search strategy and you only want to exclude a few edge cases.
The second approach, automatically inferring test strategies, is useful if you
just want to test your function without specifying any particular search strategy.Icontract-hypothesis guarantees that the inferred strategy must satisfy the preconditions.
If it does not, you should consider it a bug in which case
pleasecreate an issueso that we can fix it.If you want to inspect the strategy or further refine it programmatically, useicontract_hypothesis.infer_strategy:>>>importmath>>>importicontract>>>importicontract_hypothesis>>>@icontract.require(lambdax:x>0)...@icontract.require(lambdax:x>math.sqrt(x))...defsome_func(x:float)->int:...pass>>>icontract_hypothesis.infer_strategy(some_func)fixed_dictionaries({'x':floats(min_value=0,exclude_min=True).filter(lambdax:x>math.sqrt(x))})Testing ToolWe providepyicontract-hypothesistestcommand-line tool which you can use
to automatically test a module.usage: pyicontract-hypothesis test [-h] -p PATH
                                   [--settings [SETTINGS [SETTINGS ...]]]
                                   [--inspect] [-i [INCLUDE [INCLUDE ...]]]
                                   [-e [EXCLUDE [EXCLUDE ...]]]

optional arguments:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  Path to the Python file to test
  --settings [SETTINGS [SETTINGS ...]]
                        Specify settings for Hypothesis

                        The settings are assigned by '='.
                        The value of the setting needs to be encoded as JSON.

                        Example: max_examples=500
  --inspect             Only show the strategy and the settings

                        No tests are executed.
  -i [INCLUDE [INCLUDE ...]], --include [INCLUDE [INCLUDE ...]]
                        Regular expressions, lines or line ranges of the functions to process

                        If a line or line range overlaps the body of a function,
                        the function is considered included.

                        Example 1: ^do_something.*$
                        Example 2: 3
                        Example 3: 34-65
  -e [EXCLUDE [EXCLUDE ...]], --exclude [EXCLUDE [EXCLUDE ...]]
                        Regular expressions, lines or line ranges of the functions to exclude

                        If a line or line range overlaps the body of a function,
                        the function is considered excluded.

                        Example 1: ^do_something.*$
                        Example 2: 3
                        Example 3: 34-65Note thatpyicontract-hypothesistestcan be trivially integrated with
your IDE if you can pass in the current cursor position and the
current file name.Ghostwriting ToolWriting property-based tests by hand is tedious and can be partially automated.
To that end, we implemented a ghostwriter utilitypyicontract-hypothesisghostwritethat generates a first draft based on pre-conditions that you manually refine further:usage: pyicontract-hypothesis ghostwrite [-h] (-m MODULE | -p PATH)
                                         [-o OUTPUT] [--explicit] [--bare]
                                         [-i [INCLUDE [INCLUDE ...]]]
                                         [-e [EXCLUDE [EXCLUDE ...]]]

optional arguments:
  -h, --help            show this help message and exit
  -m MODULE, --module MODULE
                        Module to ghostwrite the unit tests for
  -p PATH, --path PATH  Path to the module to ghostwrite the unit tests for.

                        If the file represents a module reachable through
                        sys.path, use the qualified module name in
                        the unit test.

                        Otherwise, the module is represented as the stem
                        of the path with all non-identifier characters
                        replaced with an underscore ("_").
  -o OUTPUT, --output OUTPUT
                        Path to the file where the output should be written.

                        If '-', writes to STDOUT.
  --explicit            Write the inferred strategies explicitly

                        This is practical if you want to tune and
                        refine the strategies and just want to use
                        ghostwriting as a starting point.

                        Mind that pyicontract-hypothesis does not
                        automatically fix imports as this is
                        usually project-specific. You have to fix imports
                        manually after the ghostwriting.
  --bare                Print only the body of the tests and omit header/footer
                        (such as TestCase class or import statements).

                        This is useful when you only want to inspect a single test or
                        include a single test function in a custom test suite.
  -i [INCLUDE [INCLUDE ...]], --include [INCLUDE [INCLUDE ...]]
                        Regular expressions, lines or line ranges of the functions to process

                        If a line or line range overlaps the body of a function,
                        the function is considered included.

                        Example 1: ^do_something.*$
                        Example 2: 3
                        Example 3: 34-65
  -e [EXCLUDE [EXCLUDE ...]], --exclude [EXCLUDE [EXCLUDE ...]]
                        Regular expressions, lines or line ranges of the functions to exclude

                        If a line or line range overlaps the body of a function,
                        the function is considered excluded.

                        Example 1: ^do_something.*$
                        Example 2: 3
                        Example 3: 34-65The examples of ghostwritten tests are available at:tests/pyicontract_hypothesis/samplesRunning Tools as ModuleIf for some reason you want to run the tools as a module, just invoke:usage: icontract_hypothesis [-h] {test,ghostwrite} ...

Combine property-based testing with contracts of a Python module.

positional arguments:
  {test,ghostwrite}  Commands
    test             Test the functions automatically by inferring search
                     strategies from preconditions
    ghostwrite       Ghostwrite the unit tests with inferred search strategies

optional arguments:
  -h, --help         show this help message and exitInstallationicontract-hypothesis is available on PyPI athttps://pypi.org/project/icontract-hypothesis, so you can usepip:pip3 install icontract-hypothesisSearch StrategiesA naive approach to fuzzy testing is to randomly sample input data, filter it
based on pre-conditions and ensure post-conditions after the run. However,
if your acceptable band of input values is narrow, the rejection sampling
will become impractically slow.For example, assume a pre-condition5 < x < 10.
Sampling from all possible integers forxwill rarely hit
the pre-condition (if ever) thus wasting valuable computational time.
The problem is exacerbated as the number of arguments grow due tothe curse of dimensionality.Icontract-hypothesis tries to address the search strategies
a bit more intelligently:The pre-conditions are matched against common code patterns to define
the strategies. For example,5 < x < 10gives a search strategyhypothesis.strategies.integers(min=6, max=9).We currently match bounds on all available Hypothesis types
(int,float,datetime.dateetc.).
We also match regular expressions onstrarguments.Pre-conditions which could not be matched, but operate on a single argument
are inferred based on the type hint and composed with HypothesisFilteredStrategy.The remainder of the pre-conditions are enforced by filtering on the whole
fixed dictionary which is finally passed into the function as keyword arguments.There is an ongoing effort to move the strategy matching code into Hypothesis and
develop it further to include many more cases. Seethis Hypothesis issue.Note that static analysis of the source code may not determine all the defined names in various
scopes as they can also be injected dynamically (e.g., setting__globals__attribute orglobals()[random.choice("abc")]= 1).
As long as you keep fancy dynamic acrobatics out of your contracts,
the strategy inference by icontract-hypothesis should work fine.ClassesHypothesis automatically builds composite input arguments (classes, dataclasses,
named tuplesetc.). If your class enforces pre-conditions in the constructor
method (__init__), make sure that it inherits fromicontract.DBC.That way icontract-hypothesis will usehypothesis.strategies.register_type_strategyto register your class with Hypothesis and consider pre-conditions when building
its instances.It is important that you shouldnotusehypothesis.strategies.buildswith the classes using contracts in their constructors asbuildswill disregard the registered strategy. You should usehypothesis.strategies.from_typeinstead. Seethis comment on an Hypothesis issueandthe corresponding answer.Many times default inferred strategies for the constructors should be enough, though you
are of course not restricted to them. You can register your own strategies withhypothesis.strategies.register_type_strategy. Icontract-hypothesis will respect the previous registrations and will not overwrite them.IDE Plug-insicontract-hypothesis-vimforVIMicontract-hypothesis-pycharmforPyCharmicontract-hypothesis-vscodeforVisual Studio CodeRelated LibrariesPython design-by-contract librariesdealanddpcontractsintegrate directly with Hypothesis (seethis pageandthat page,
respectively).As of 2020-12-16:Behavioral subtyping.Neither of the two libraries handles behavioral sub-typing correctly
(i.e., they do not weaken and strengthen the pre-conditions, and
post-conditions and invariants, respectively).
Hence they can not be used with class hierarchies as the contracts are not
properly inherited.This is not strictly related to property-based testing,
but presents an inherent flaw in how they implement contracts.
Hence even if you manually supply a search strategy that
fulfills behavioral subtyping, these two libraries would
report (or ignore) an error.Consider this example withdeal:classA:@deal.post(lambdaresult:result%2==0)defsome_func(self)->int:return2classB(A):@deal.post(lambdaresult:result%3==0)defsome_func(self)->int:# The result 9 satisfies the postcondition of B.some_func,# but not the postcondition of A.some_func thus# breaking the behavioral subtyping.return9b=B()# The correct behavior would be to throw an exception here.b.some_func()Rejection sampling.The two libraries only provide rejection sampling which is insufficient
for many practical use cases. For example, the computational time grows exponentially with the
number of arguments (see Section “Search Strategies”).Propagation of contracts.Finally, the existing libraries do not automatically propagate
pre-conditions of constructors to Hypothesis so automatic testing with composite inputs
(such as instances of classes) is currently not possible with these two libraries. The user can,
of course, manually design search strategies that satisfy the contracts.
In contrast, icontract-hypothesis does that hard-lifting for you automatically.BenchmarksWe run benchmarks againstdealanddpcontractslibraries as part of our continuous integration.We benchmark against functions using 1, 2 and 3 arguments, respectively, with the precondition that
the argument should be positive (e.g.,a > 0). We sampled 100 inputs per each run.The following scripts were run:benchmarks/compare_with_others.pyThe benchmarks were executed on Intel(R) Xeon(R) E-2276M  CPU @ 2.80GHz.
We used Python 3.8.5, icontract 2.4.1, deal 4.4.0 and dpcontracts 0.6.0.The following tables summarize the results.Benchmarking Hypothesis testing:Argument count: 1CaseTotal timeTime per runRelative time per runbenchmark_icontract_inferred_strategy0.48 s48.29 ms100%benchmark_icontract_assume_preconditions0.79 s78.75 ms163%benchmark_dpcontracts1.06 s106.17 ms220%benchmark_deal0.83 s82.63 ms171%Argument count: 2CaseTotal timeTime per runRelative time per runbenchmark_icontract_inferred_strategy0.63 s63.45 ms100%benchmark_icontract_assume_preconditions1.65 s165.05 ms260%benchmark_dpcontracts2.10 s209.51 ms330%benchmark_deal1.61 s161.09 ms254%Argument count: 3CaseTotal timeTime per runRelative time per runbenchmark_icontract_inferred_strategy0.72 s71.66 ms100%benchmark_icontract_assume_preconditions3.30 s330.20 ms461%benchmark_dpcontracts4.23 s423.31 ms591%benchmark_deal3.20 s319.57 ms446%VersioningWe followSemantic Versioning.
The version X.Y.Z indicates:X is the major version (backward-incompatible),Y is the minor version (backward-compatible), andZ is the patch version (backward-compatible bug fix).