Repository: HypothesisWorks/hypothesis

Issue:
Generate gufunc input shapes with a `gufunc=` arg to `mutually_broadcastable_shapes`
This is the tracking issue for a long-awaited feature: generating a collection of array shapes for a Numpy 'generalised ufunc'.  It's fairly niche, but *very* useful to anyone in that niche.

As background, a strategy for `gufunc_shapes` was first proposed in #1760/#1784 and later released as the [`hypothesis-gufunc` extension](https://github.com/uber/hypothesis-gufunc).  Around the same time, #1829 proposed our `boradcastable_shapes` strategy (for single shapes), and we later decided (#1969) to release `mutually_broadcastable_shapes` without gufunc support in the first instance.  That was added by #2153/#2173, so we're finally ready for gufunc support!

-------------

The implementation isn't *too* bad, but there are a bunch of little details we'll need to handle carefully.

- The `gufunc` argument is keyword-only, defaults to `not_set`, and exactly one of it or `num_shapes` must be passed.
- If a `gufunc` callable is passed, use its `.signature` attribute. IMO we should reject None signatures, since the ufunc is not generalised and it simply gives us `num_shapes`.
- Parse the `gufunc` signature string ~~using `numpy.lib.function_base._parse_gufunc_signature`~~ something which is actually [NEP-20](https://numpy.org/neps/nep-0020-gufunc-signature-enhancement.html) compliant, i.e. our own regex-based parser.
- When drawing, we sort the size names (`m` and `n` in `'(m,n),(n)->(m)'` for matrix-vector multiplication), then draw an allowed size for each, and construct our initial tuples for each shape - the "core dimensions".
- Also handle fixed-size dimensions and optional dimensions ~~and broadcastable dimensions~~ (NEP rejected).  Fiddly but no worse than our existing logic for other dimensions.
- `min_dims` and `max_dims` are interpreted (and documented!) as applying to the *"loop dimensions" **only***.  In the example above, `max_dims=1` would mean that we could generate a three-dimensional output `(loop,m,n)` and the result shape would be `(loop,m)`.
- The exception is that we (silently) stop adding to a shape when it reaches 32 dimensions even if this means that it will have fewer than `min_dims` dimensions.  We should also adjust the biased_coin probability in these cases to avoid squashing our uniform distribution against the upper bound.
  If for example the cap were two dimensions, it would be impossible to use any loop dimensions or have a 2D result from matrix-vector multiplication without supporting this (or ugly per-dimension size limits).  It's very unlikely to arise in practice, but if *Hypothesis* doesn't handle edge cases who will?
  I believe that optional dimensions (eg in matmul `(m?,n),(n,p?)->(m?,p?)`) *do* count towards the limit of 32 dimensions even if unused, but we should check this and generate them if not.

We'll also want to make sure it's well-tested and thoroughly documented before merging, but I think that's everything we need in for a viable implementation 😄 







Discussion:
For @rsokl - parsing signatures is mostly done, and that's the hard part... https://github.com/HypothesisWorks/hypothesis/compare/master...Zac-HD:ufunc-shapes
You read my mind! I was going to ask you to ping me if you had been cooking up the parser 😄 

Resolving patch:
diff --git a/hypothesis-python/docs/conf.py b/hypothesis-python/docs/conf.py
--- a/hypothesis-python/docs/conf.py
+++ b/hypothesis-python/docs/conf.py
@@ -86,6 +86,7 @@
     "pull": (_repo + "pull/%s", "pull request #"),
     "pypi": ("https://pypi.org/project/%s", ""),
     "bpo": ("https://bugs.python.org/issue%s", "bpo-"),
+    "np-ref": ("https://docs.scipy.org/doc/numpy/reference/%s", ""),
 }
 
 # -- Options for HTML output ----------------------------------------------
diff --git a/hypothesis-python/src/hypothesis/extra/numpy.py b/hypothesis-python/src/hypothesis/extra/numpy.py
--- a/hypothesis-python/src/hypothesis/extra/numpy.py
+++ b/hypothesis-python/src/hypothesis/extra/numpy.py
@@ -18,6 +18,7 @@
 from __future__ import absolute_import, division, print_function
 
 import math
+import re
 from collections import namedtuple
 
 import numpy as np
@@ -27,13 +28,19 @@
 from hypothesis import Verbosity, assume
 from hypothesis._settings import note_deprecation
 from hypothesis.errors import InvalidArgument
-from hypothesis.internal.compat import PY2, hrange, integer_types
+from hypothesis.internal.compat import (
+    PY2,
+    hrange,
+    integer_types,
+    quiet_raise,
+    string_types,
+)
 from hypothesis.internal.coverage import check_function
 from hypothesis.internal.reflection import proxies, reserved_means_kwonly_star
 from hypothesis.internal.validation import check_type, check_valid_interval
 from hypothesis.reporting import current_verbosity
 from hypothesis.searchstrategy import SearchStrategy
-from hypothesis.utils.conventions import not_set
+from hypothesis.utils.conventions import UniqueIdentifier, not_set
 
 if PY2:
     BroadcastableShapes = namedtuple(
@@ -844,6 +851,7 @@ class MutuallyBroadcastableShapesStrategy(SearchStrategy):
     def __init__(
         self,
         num_shapes,
+        signature=None,
         base_shape=(),
         min_dims=0,
         max_dims=None,
@@ -856,6 +864,7 @@ def __init__(
         self.base_shape = base_shape
         self.side_strat = st.integers(min_side, max_side)
         self.num_shapes = num_shapes
+        self.signature = signature
         self.min_dims = min_dims
         self.max_dims = max_dims
         self.min_side = min_side
@@ -864,11 +873,62 @@ def __init__(
         self.size_one_allowed = self.min_side <= 1 <= self.max_side
 
     def do_draw(self, data):
+        # We don't usually have a gufunc signature; do the common case first & fast.
+        if self.signature is None:
+            return self._draw_loop_dimensions(data)
+
+        # When we *do*, draw the core dims, then draw loop dims, and finally combine.
+        core_in, core_res = self._draw_core_dimensions(data)
+
+        # If some core shape has omitted optional dimensions, it's an error to add
+        # loop dimensions to it.  We never omit core dims if min_dims >= 1.
+        # This ensures that we respect Numpy's gufunc broadcasting semantics and user
+        # constraints without needing to check whether the loop dims will be
+        # interpreted as an invalid substitute for the omitted core dims.
+        # We may implement this check later!
+        use = [None not in shp for shp in core_in]
+        loop_in, loop_res = self._draw_loop_dimensions(data, use=use)
+
+        def add_shape(loop, core):
+            return tuple(x for x in (loop + core)[-32:] if x is not None)
+
+        return BroadcastableShapes(
+            input_shapes=tuple(add_shape(l, c) for l, c in zip(loop_in, core_in)),
+            result_shape=add_shape(loop_res, core_res),
+        )
+
+    def _draw_core_dimensions(self, data):
+        # Draw gufunc core dimensions, with None standing for optional dimensions
+        # that will not be present in the final shape.  We track omitted dims so
+        # that we can do an accurate per-shape length cap.
+        dims = {}
+        shapes = []
+        for shape in self.signature.input_shapes + (self.signature.result_shape,):
+            shapes.append([])
+            for name in shape:
+                if name.isdigit():
+                    shapes[-1].append(int(name))
+                    continue
+                if name not in dims:
+                    dim = name.strip("?")
+                    dims[dim] = data.draw(self.side_strat)
+                    if self.min_dims == 0 and not data.draw_bits(3):
+                        dims[dim + "?"] = None
+                    else:
+                        dims[dim + "?"] = dims[dim]
+                shapes[-1].append(dims[name])
+        return tuple(tuple(s) for s in shapes[:-1]), tuple(shapes[-1])
+
+    def _draw_loop_dimensions(self, data, use=None):
         # All shapes are handled in column-major order; i.e. they are reversed
         base_shape = self.base_shape[::-1]
         result_shape = list(base_shape)
         shapes = [[] for _ in range(self.num_shapes)]
-        use = [True for _ in range(self.num_shapes)]
+        if use is None:
+            use = [True for _ in range(self.num_shapes)]
+        else:
+            assert len(use) == self.num_shapes
+            assert all(isinstance(x, bool) for x in use)
 
         for dim_count in range(1, self.max_dims + 1):
             dim = dim_count - 1
@@ -928,18 +988,100 @@ def do_draw(self, data):
         )
 
 
+# See https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html
+# Implementation based on numpy.lib.function_base._parse_gufunc_signature
+# with minor upgrades to handle numeric and optional dimensions.  Examples:
+#
+#     add       (),()->()                   binary ufunc
+#     sum1d     (i)->()                     reduction
+#     inner1d   (i),(i)->()                 vector-vector multiplication
+#     matmat    (m,n),(n,p)->(m,p)          matrix multiplication
+#     vecmat    (n),(n,p)->(p)              vector-matrix multiplication
+#     matvec    (m,n),(n)->(m)              matrix-vector multiplication
+#     matmul    (m?,n),(n,p?)->(m?,p?)      combination of the four above
+#     cross1d   (3),(3)->(3)                cross product with frozen dimensions
+#
+# Note that while no examples of such usage are given, Numpy does allow
+# generalised ufuncs that have *multiple output arrays*.  This is not
+# currently supported by Hypothesis - please contact us if you would use it!
+#
+# We are unsure if gufuncs allow frozen dimensions to be optional, but it's
+# easy enough to support here - and so we will unless we learn otherwise.
+#
+_DIMENSION = r"\w+\??"  # Note that \w permits digits too!
+_SHAPE = r"\((?:{0}(?:,{0})".format(_DIMENSION) + r"{0,31})?\)"
+_ARGUMENT_LIST = "{0}(?:,{0})*".format(_SHAPE)
+_SIGNATURE = r"^{}->{}$".format(_ARGUMENT_LIST, _SHAPE)
+_SIGNATURE_MULTIPLE_OUTPUT = r"^{0}->{0}$".format(_ARGUMENT_LIST)
+
+_GUfuncSig = namedtuple("_GUfuncSig", ["input_shapes", "result_shape"])
+
+
+def _hypothesis_parse_gufunc_signature(signature, all_checks=True):
+    # Disable all_checks to better match the Numpy version, for testing
+    if not re.match(_SIGNATURE, signature):
+        if re.match(_SIGNATURE_MULTIPLE_OUTPUT, signature):
+            raise InvalidArgument(
+                "Hypothesis does not yet support generalised ufunc signatures "
+                "with multiple output arrays - mostly because we don't know of "
+                "anyone who uses them!  Please get in touch with us to fix that."
+                "\n (signature=%r)" % (signature,)
+            )
+        if re.match(np.lib.function_base._SIGNATURE, signature):
+            raise InvalidArgument(
+                "signature=%r matches Numpy's regex for gufunc signatures, but "
+                "contains shapes with more than 32 dimensions and is thus invalid."
+                % (signature,)
+            )
+        raise InvalidArgument("%r is not a valid gufunc signature" % (signature,))
+    input_shapes, output_shapes = (
+        tuple(tuple(re.findall(_DIMENSION, a)) for a in re.findall(_SHAPE, arg_list))
+        for arg_list in signature.split("->")
+    )
+    assert len(output_shapes) == 1
+    result_shape = output_shapes[0]
+    if all_checks:
+        # Check that there are no names in output shape that do not appear in inputs.
+        # (kept out of parser function for easier generation of test values)
+        # We also disallow frozen optional dimensions - this is ambiguous as there is
+        # no way to share an un-named dimension between shapes.  Maybe just padding?
+        # Anyway, we disallow it pending clarification from upstream.
+        frozen_optional_err = (
+            "Got dimension %r, but handling of frozen optional dimensions "
+            "is ambiguous.  If you known how this should work, please "
+            "contact us to get this fixed and documented (signature=%r)."
+        )
+        only_out_err = (
+            "The %r dimension only appears in the output shape, and is "
+            "not frozen, so the size is not determined (signature=%r)."
+        )
+        names_in = {n.strip("?") for shp in input_shapes for n in shp}
+        names_out = {n.strip("?") for n in result_shape}
+        for shape in input_shapes + (result_shape,):
+            for name in shape:
+                try:
+                    int(name.strip("?"))
+                    if "?" in name:
+                        raise InvalidArgument(frozen_optional_err % (name, signature))
+                except ValueError:
+                    if name.strip("?") in (names_out - names_in):
+                        quiet_raise(InvalidArgument(only_out_err % (name, signature)))
+    return _GUfuncSig(input_shapes=input_shapes, result_shape=result_shape)
+
+
 @st.defines_strategy
 @reserved_means_kwonly_star
 def mutually_broadcastable_shapes(
-    __reserved=not_set,
-    num_shapes=not_set,
-    base_shape=(),
-    min_dims=0,
-    max_dims=None,
-    min_side=1,
-    max_side=None,
+    __reserved=not_set,  # type: Any
+    num_shapes=not_set,  # type: Union[UniqueIdentifier, int]
+    signature=not_set,  # type: Union[UniqueIdentifier, str]
+    base_shape=(),  # type: Shape
+    min_dims=0,  # type: int
+    max_dims=None,  # type: int
+    min_side=1,  # type: int
+    max_side=None,  # type: int
 ):
-    # type: (Any, Any, Shape , int, int, int, int) -> st.SearchStrategy[BroadcastableShapes]
+    # type: (...) -> st.SearchStrategy[BroadcastableShapes]
     """Return a strategy for generating a specified number of shapes, N, that are
     mutually-broadcastable with one another and with the provided "base-shape".
 
@@ -978,13 +1120,66 @@ def mutually_broadcastable_shapes(
         BroadcastableShapes(input_shapes=((3,), (1, 3), (2, 3)), result_shape=(2, 3))
         BroadcastableShapes(input_shapes=((), (), ()), result_shape=(2, 3))
         BroadcastableShapes(input_shapes=((3,), (), (3,)), result_shape=(2, 3))
+
+    **Use with Generalised Universal Function signatures**
+
+    A :np-ref:`universal function <ufuncs.html>` (or ufunc for short) is a function
+    that operates on ndarrays in an element-by-element fashion, supporting array
+    broadcasting, type casting, and several other standard features.
+    A :np-ref:`generalised ufunc <c-api.generalized-ufuncs.html>` operates on
+    sub-arrays rather than elements, based on the "signature" of the function.
+    Compare e.g. :obj:`numpy:numpy.add` (ufunc) to :obj:`numpy:numpy.matmul` (gufunc).
+
+    To generate shapes for a gufunc, you can pass the ``signature`` argument instead of
+    ``num_shapes``.  This must be a gufunc signature string; which you can write by
+    hand or access as e.g. ``np.matmul.signature`` on generalised ufuncs.
+
+    In this case, the ``side`` arguments are applied to the 'core dimensions' as well,
+    ignoring any frozen dimensions.  ``base_shape``  and the ``dims`` arguments are
+    applied to the 'loop dimensions', and if necessary, the dimensionality of each
+    shape is silently capped to respect the 32-dimension limit.
+
+    The generated ``result_shape`` is the real result shape of applying the gufunc
+    to arrays of the generated ``input_shapes``, even where this is different to
+    broadcasting the loop dimensions.
+
+    gufunc-compatible shapes shrink their loop dimensions as above, towards omitting
+    optional core dimensions, and smaller-size core dimensions.
+
+    .. code-block:: pycon
+
+        >>> # np.matmul.signature == "(m?,n),(n,p?)->(m?,p?)"
+        >>> for _ in range(3):
+        ...     mutually_broadcastable_shapes(signature=np.matmul.signature).example()
+        BroadcastableShapes(input_shapes=((2,), (2,)), result_shape=())
+        BroadcastableShapes(input_shapes=((3, 4, 2), (1, 2)), result_shape=(3, 4))
+        BroadcastableShapes(input_shapes=((4, 2), (1, 2, 3)), result_shape=(4, 3))
     """
     if __reserved is not not_set:
         raise InvalidArgument("Do not pass the __reserved argument.")
 
-    check_type(integer_types, num_shapes, "num_shapes")
-    if num_shapes < 1:
-        raise InvalidArgument("num_shapes=%s must be at least 1." % (num_shapes,))
+    arg_msg = "Pass either the `num_shapes` or the `signature` argument, but not both."
+    if num_shapes is not not_set:
+        check_argument(signature is not_set, arg_msg)
+        check_type(integer_types, num_shapes, "num_shapes")
+        assert isinstance(num_shapes, integer_types)  # for mypy
+        check_argument(num_shapes >= 1, "num_shapes={} must be at least 1", num_shapes)
+        parsed_signature = None
+        sig_dims = 0
+    else:
+        check_argument(signature is not not_set, arg_msg)
+        if signature is None:
+            raise InvalidArgument(
+                "Expected a string, but got invalid signature=None.  "
+                "(maybe .signature attribute of an element-wise ufunc?)"
+            )
+        check_type(string_types, signature, "signature")
+        parsed_signature = _hypothesis_parse_gufunc_signature(signature)
+        sig_dims = min(
+            map(len, parsed_signature.input_shapes + (parsed_signature.result_shape,))
+        )
+        num_shapes = len(parsed_signature.input_shapes)
+        assert num_shapes >= 1
 
     check_type(tuple, base_shape, "base_shape")
     strict_check = max_dims is not None
@@ -992,7 +1187,7 @@ def mutually_broadcastable_shapes(
     check_type(integer_types, min_dims, "min_dims")
 
     if max_dims is None:
-        max_dims = min(32, max(len(base_shape), min_dims) + 2)
+        max_dims = min(32 - sig_dims, max(len(base_shape), min_dims) + 2)
     else:
         check_type(integer_types, max_dims, "max_dims")
 
@@ -1004,8 +1199,13 @@ def mutually_broadcastable_shapes(
     order_check("dims", 0, min_dims, max_dims)
     order_check("side", 0, min_side, max_side)
 
-    if 32 < max_dims:
-        raise InvalidArgument("max_dims cannot exceed 32")
+    if 32 - sig_dims < max_dims:
+        if sig_dims == 0:
+            raise InvalidArgument("max_dims cannot exceed 32")
+        raise InvalidArgument(
+            "max_dims=%r would exceed the 32-dimension limit given signature=%r"
+            % (signature, parsed_signature)
+        )
 
     dims, bnd_name = (max_dims, "max_dims") if strict_check else (min_dims, "min_dims")
 
@@ -1039,6 +1239,7 @@ def mutually_broadcastable_shapes(
 
     return MutuallyBroadcastableShapesStrategy(
         num_shapes=num_shapes,
+        signature=parsed_signature,
         base_shape=base_shape,
         min_dims=min_dims,
         max_dims=max_dims,


Test patch:
diff --git a/hypothesis-python/tests/numpy/test_argument_validation.py b/hypothesis-python/tests/numpy/test_argument_validation.py
--- a/hypothesis-python/tests/numpy/test_argument_validation.py
+++ b/hypothesis-python/tests/numpy/test_argument_validation.py
@@ -28,7 +28,8 @@
 
 
 def e(a, **kwargs):
-    return (a, kwargs)
+    rep = "%s(%s)" % (a.__name__, ", ".join("%s=%r" % it for it in kwargs.items()))
+    return pytest.param(a, kwargs, id=rep)
 
 
 @pytest.mark.parametrize(
@@ -117,70 +118,59 @@ def e(a, **kwargs):
             min_side=2,
             max_side=3,
         ),
+        e(nps.mutually_broadcastable_shapes),
         e(nps.mutually_broadcastable_shapes, num_shapes=0),
         e(nps.mutually_broadcastable_shapes, num_shapes="a"),
         e(nps.mutually_broadcastable_shapes, num_shapes=2, base_shape="a"),
         e(
             nps.mutually_broadcastable_shapes,  # min_side is invalid type
             num_shapes=2,
-            base_shape=(2, 2),
             min_side="a",
         ),
         e(
             nps.mutually_broadcastable_shapes,  # min_dims is invalid type
             num_shapes=2,
-            base_shape=(2, 2),
             min_dims="a",
         ),
         e(
             nps.mutually_broadcastable_shapes,  # max_side is invalid type
             num_shapes=2,
-            base_shape=(2, 2),
             max_side="a",
         ),
         e(
             nps.mutually_broadcastable_shapes,  # max_side is invalid type
             num_shapes=2,
-            base_shape=(2, 2),
             max_dims="a",
         ),
         e(
             nps.mutually_broadcastable_shapes,  # min_side is out of domain
             num_shapes=2,
-            base_shape=(2, 2),
             min_side=-1,
         ),
         e(
             nps.mutually_broadcastable_shapes,  # min_dims is out of domain
             num_shapes=2,
-            base_shape=(2, 2),
             min_dims=-1,
         ),
         e(
             nps.mutually_broadcastable_shapes,  # min_dims is out of domain
             num_shapes=2,
-            base_shape=(2, 2),
             min_dims=33,
-            max_dims=None,
         ),
         e(
             nps.mutually_broadcastable_shapes,  # max_dims is out of domain
             num_shapes=2,
-            base_shape=(2, 2),
-            min_dims=1,
             max_dims=33,
         ),
         e(
             nps.mutually_broadcastable_shapes,  # max_side < min_side
             num_shapes=2,
-            base_shape=(2, 2),
             min_side=1,
             max_side=0,
         ),
         e(
             nps.mutually_broadcastable_shapes,  # max_dims < min_dims
             num_shapes=2,
-            base_shape=(2, 2),
             min_dims=1,
             max_dims=0,
         ),
@@ -220,6 +210,49 @@ def e(a, **kwargs):
             min_side=2,
             max_side=3,
         ),
+        e(
+            nps.mutually_broadcastable_shapes,  # valid to pass num_shapes xor gufunc
+            num_shapes=2,
+            signature="()->()",
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # element-wise ufunc has signature=None
+            signature=numpy.add.signature,
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # multiple outputs not yet supported
+            signature="()->(),()",
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # output has dimension not in inputs
+            signature="()->(i)",
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # frozen-optional is ambiguous & banned
+            signature="(2?)->()",
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # signature must be in string format
+            signature=([(), ()], [()]),
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # string must match signature regex
+            signature="this string isn't a valid signature",
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # shape with too many dimensions
+            signature="(" + ",".join("d{}".format(n) for n in range(33)) + ")->()",
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # max_dims too large given ufunc
+            signature=numpy.matmul.signature,
+            max_dims=32,
+        ),
+        e(
+            nps.mutually_broadcastable_shapes,  # least valid max_dims is < min_dims
+            signature=numpy.matmul.signature,
+            min_dims=32,
+        ),
         e(nps.basic_indices, shape=0),
         e(nps.basic_indices, shape=("1", "2")),
         e(nps.basic_indices, shape=(0, -1)),
diff --git a/hypothesis-python/tests/numpy/test_gufunc.py b/hypothesis-python/tests/numpy/test_gufunc.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/numpy/test_gufunc.py
@@ -0,0 +1,229 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2019 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import absolute_import, division, print_function
+
+import numpy as np
+import pytest
+from pytest import param
+
+import hypothesis.extra.numpy as nps
+import hypothesis.strategies as st
+from hypothesis import example, given, note, settings
+from hypothesis.errors import InvalidArgument
+from tests.common.debug import find_any, minimal
+
+
+def use_signature_examples(func):
+    for sig in [
+        "(),()->()",
+        "(i)->()",
+        "(i),(i)->()",
+        "(m,n),(n,p)->(m,p)",
+        "(n),(n,p)->(p)",
+        "(m,n),(n)->(m)",
+        "(m?,n),(n,p?)->(m?,p?)",
+        "(3),(3)->(3)",
+    ]:
+        func = example(sig)(func)
+    return func
+
+
+def hy_sig_2_np_sig(hy_sig):
+    return (
+        [tuple(d.strip("?") for d in shape) for shape in hy_sig.input_shapes],
+        [tuple(d.strip("?") for d in hy_sig.result_shape)],
+    )
+
+
+@use_signature_examples
+@example("()->(%s),()" % ",".join(33 * "0"))
+@given(st.from_regex(np.lib.function_base._SIGNATURE))
+def test_numpy_signature_parses(sig):
+    if sig == "(m?,n),(n,p?)->(m?,p?)":  # matmul example
+        return
+
+    np_sig = np.lib.function_base._parse_gufunc_signature(sig)
+    try:
+        hy_sig = nps._hypothesis_parse_gufunc_signature(sig, all_checks=False)
+        assert np_sig == hy_sig_2_np_sig(hy_sig)
+    except InvalidArgument:
+        shape_too_long = any(len(s) > 32 for s in np_sig[0] + np_sig[1])
+        multiple_outputs = len(np_sig[1]) > 1
+        assert shape_too_long or multiple_outputs
+
+        # Now, if we can fix this up does it validate?
+        in_, out = sig.split("->")
+        sig = in_ + "->" + out.split(",(")[0]
+        np_sig = np.lib.function_base._parse_gufunc_signature(sig)
+        if all(len(s) <= 32 for s in np_sig[0] + np_sig[1]):
+            hy_sig = nps._hypothesis_parse_gufunc_signature(sig, all_checks=False)
+            assert np_sig == hy_sig_2_np_sig(hy_sig)
+
+
+@use_signature_examples
+@given(st.from_regex(nps._SIGNATURE))
+def test_hypothesis_signature_parses(sig):
+    hy_sig = nps._hypothesis_parse_gufunc_signature(sig, all_checks=False)
+    try:
+        np_sig = np.lib.function_base._parse_gufunc_signature(sig)
+        assert np_sig == hy_sig_2_np_sig(hy_sig)
+    except ValueError:
+        assert "?" in sig
+        # We can always fix this up, and it should then always validate.
+        sig = sig.replace("?", "")
+        hy_sig = nps._hypothesis_parse_gufunc_signature(sig, all_checks=False)
+        np_sig = np.lib.function_base._parse_gufunc_signature(sig)
+        assert np_sig == hy_sig_2_np_sig(hy_sig)
+
+
+def test_frozen_dims_signature():
+    nps._hypothesis_parse_gufunc_signature("(2),(3)->(4)")
+
+
+@st.composite
+def gufunc_arrays(draw, shape_strat, **kwargs):
+    """An example user strategy built on top of mutually_broadcastable_shapes."""
+    input_shapes, result_shape = draw(shape_strat)
+    arrays_strat = st.tuples(*(nps.arrays(shape=s, **kwargs) for s in input_shapes))
+    return draw(arrays_strat), result_shape
+
+
+@given(
+    gufunc_arrays(
+        nps.mutually_broadcastable_shapes(signature=np.matmul.signature),
+        dtype="float64",
+        elements=st.floats(0, 1000),
+    )
+)
+def test_matmul_gufunc_shapes(everything):
+    arrays, result_shape = everything
+    out = np.matmul(*arrays)
+    assert out.shape == result_shape
+
+
+@settings(deadline=None, max_examples=10)
+@pytest.mark.parametrize(
+    "target_sig",
+    ("(i),(i)->()", "(m,n),(n,p)->(m,p)", "(n),(n,p)->(p)", "(m,n),(n)->(m)"),
+)
+@given(data=st.data())
+def test_matmul_signature_can_exercise_all_combination_of_optional_dims(
+    target_sig, data
+):
+    target_shapes = data.draw(
+        nps.mutually_broadcastable_shapes(signature=target_sig, max_dims=0)
+    )
+    find_any(
+        nps.mutually_broadcastable_shapes(
+            signature="(m?,n),(n,p?)->(m?,p?)", max_dims=0
+        ),
+        lambda shapes: shapes == target_shapes,
+        settings(max_examples=10 ** 6),
+    )
+
+
+@settings(deadline=None, max_examples=50)
+@given(
+    min_dims=st.integers(0, 4),
+    min_side=st.integers(2, 3),
+    n_fixed=st.booleans(),
+    data=st.data(),
+)
+def test_matmul_sig_shrinks_as_documented(min_dims, min_side, n_fixed, data):
+    sig = "(m?,n),(n,p?)->(m?,p?)"
+    if n_fixed:
+        n_value = data.draw(st.integers(0, 4))
+        sig = sig.replace("n", str(n_value))
+    else:
+        n_value = min_side
+
+    note("signature: {}".format(sig))
+    max_dims = data.draw(st.none() | st.integers(min_dims, 4), label="max_dims")
+    max_side = data.draw(st.none() | st.integers(min_side, 6), label="max_side")
+
+    smallest_shapes, result = minimal(
+        nps.mutually_broadcastable_shapes(
+            signature=sig,
+            min_side=min_side,
+            max_side=max_side,
+            min_dims=min_dims,
+            max_dims=max_dims,
+        )
+    )
+    note("(smallest_shapes, result): {}".format((smallest_shapes, result)))
+
+    # if min_dims >= 1 then core dims are never excluded
+    # otherwise, should shrink towards excluding all optional dims
+    expected_input_0 = (
+        (n_value,) if min_dims == 0 else (min_side,) * min_dims + (min_side, n_value)
+    )
+    assert expected_input_0 == smallest_shapes[0]
+
+    expected_input_1 = (
+        (n_value,) if min_dims == 0 else (min_side,) * min_dims + (n_value, min_side)
+    )
+    assert expected_input_1 == smallest_shapes[1]
+
+
+def gufunc_sig_to_einsum_sig(gufunc_sig):
+    """E.g. (i,j),(j,k)->(i,k) becomes ...ij,...jk->...ik"""
+
+    def einlabels(labels):
+        assert "x" not in labels, "we reserve x for fixed-dimensions"
+        return "..." + "".join(i if not i.isdigit() else "x" for i in labels)
+
+    gufunc_sig = nps._hypothesis_parse_gufunc_signature(gufunc_sig)
+    input_sig = ",".join(map(einlabels, gufunc_sig.input_shapes))
+    return input_sig + "->" + einlabels(gufunc_sig.result_shape)
+
+
+@pytest.mark.parametrize(
+    ("gufunc_sig"),
+    [
+        param("()->()", id="unary sum"),
+        param("(),()->()", id="binary sum"),
+        param("(),(),()->()", id="trinary sum"),
+        param("(i)->()", id="sum1d"),
+        param("(i,j)->(j)", id="sum rows"),
+        param("(i),(i)->()", id="inner1d"),
+        param("(i),(i),(i)->()", id="trinary inner1d"),
+        param("(m,n),(n,p)->(m,p)", id="matmat"),
+        param("(n),(n,p)->(p)", id="vecmat"),
+        param("(i,t),(j,t)->(i,j)", id="outer-inner"),
+        param("(3),(3)->(3)", id="cross1d"),
+        param("(i,j)->(j,i)", id="transpose"),
+        param("(i),(j)->(i,j)", id="outer"),
+        param("(i,3),(3,k)->(3,i,k)", id="fixed dim outer product"),
+        param("(i),(j),(k)->(i,j,k)", id="trinary outer"),
+        param("(i,i)->(i)", id="trace"),
+        param("(j,i,i,j)->(i,j)", id="bigger trace"),
+        param("(k),(j,i,k,i,j),(j,i)->(i,j)", id="trace product"),
+    ],
+)
+@given(data=st.data())
+def test_einsum_gufunc_shapes(gufunc_sig, data):
+    arrays, result_shape = data.draw(
+        gufunc_arrays(
+            nps.mutually_broadcastable_shapes(signature=gufunc_sig),
+            dtype="float64",
+            elements=st.floats(0, 1000),
+        ),
+        label="arrays, result_shape",
+    )
+    out = np.einsum(gufunc_sig_to_einsum_sig(gufunc_sig), *arrays)
+    assert out.shape == result_shape