Repository: HypothesisWorks/hypothesis

Issue:
Teach the Ghostwriter about `@staticmethod` and `@classmethod` methods
The [Hypothesis Ghostwriter](https://hypothesis.readthedocs.io/en/latest/ghostwriter.html) currently only writes tests for functions, because they're a natural starting point for property-based tests and the primary design goal is to *make it incredibly easy to get started*.

It's also usually unclear what properties you could guess for classes or instances, but in conversation at PyCon today we worked out that there are a few special cases in which we can test something after all:

- [ ] a `@staticmethod` is just like a function (albeit glued onto a class), so we could treat it just like we do any other function, replacing the dot with triple-underscore for name-based analyses.
- [ ] `@classmethod`s are similar in that they don't require a pre-build instance.  In addition to the "fuzz" tests which just check for internal errors, we can also:
  - [ ] assert that `from_*` classmethods return an instance of the class (by strong convention) 
  - [ ] and assert that the matching `to_*` method (if one exists) inverts this.


Discussion:
This looks really interesting. Do you mind if I dip my toe into it? (I don't know if it will be a good issue for me but I am happy to spend time on it, tell me if I should go something else instead)
Go for it!  Just expect to also need a while to read through the module, there's A Lot Of Code.  You might also find it helpful to read "callers to callees" rather than "top to bottom", starting with the `equivalent` function?  Also remember that you don't need to understand the whole thing to make useful changes to it!  (yay tests 😁)
quick report of what I found by testing the `equivalent` function. It actually does not gives an error when using a `staticmethod` or `classmethod` directly:

```python
from hypothesis.extra import ghostwriter

def just_func(variable: int):
    return varibale

class A:

    @staticmethod
    def stat_func(variable: int):
        return varibale

    @classmethod
    def class_func(cls, variable: int):
        return varibale

print(ghostwriter.equivalent(A.class_func, just_func))
print("================")
print(ghostwriter.equivalent(A.class_func, just_func))
```

Will gives:

```python
# This test code was written by the `hypothesis.extra.ghostwriter` module
# and is provided under the Creative Commons Zero public domain dedication.

from hypothesis import given, strategies as st


@given(variable=st.integers())
def test_equivalent_A_class_func_just_func(variable):
    result_class_func = A.class_func(variable=variable)
    result_just_func = just_func(variable=variable)
    assert result_class_func == result_just_func, (result_class_func, result_just_func)

================
# This test code was written by the `hypothesis.extra.ghostwriter` module
# and is provided under the Creative Commons Zero public domain dedication.

from hypothesis import given, strategies as st


@given(variable=st.integers())
def test_equivalent_A_class_func_just_func(variable):
    result_class_func = A.class_func(variable=variable)
    result_just_func = just_func(variable=variable)
    assert result_class_func == result_just_func, (result_class_func, result_just_func)
```

So I guess for the 1st task it just need to rename the test to `test_equivalent_A___class_func_just_func`? (I don't quite get where we need to replace the dot)
and the 2nd task to add in the extra checks?
It got more complicated when using with CLI as `.` is always interperted as module. I guess the triple underscore you mean here? so someone should type `A___class_func` instead of `A.class_func`?
- triple-underscore is for [this part of the `magic()` ghostwriter](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/extra/ghostwriter.py#L933-L952), where it's trying to detect round-trips based on the name.
  - Actually, we don't need to replace the dot, we just need to make sure that we include the class name *as well as* the method name.  (otherwise if we had two classes with a `from_dict()` classmethod in the same module, we'd only test one of them!)
  - Similarly, we'll need to improve the 'magic discovery' for a module, so that it checks classes for testable methods
  - Based on a quick glance, I think the current system seems fine for test names.
- Yep, the CLI seems a bit tricky; we can revisit that in a future PR.
- FYI, [here's how to syntax-highlight code blocks on GitHub](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks); I tend to use `python` and `python-traceback` a lot (and they don't have to be fully valid to highlight well, either, so the latter works for pytest outputs)
Ah sorry, I actually know the syntax highlighting but am too lazy to use it. My bad
For the CLI stuff actually I was working on it so don't mind including it

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/extra/cli.py b/hypothesis-python/src/hypothesis/extra/cli.py
--- a/hypothesis-python/src/hypothesis/extra/cli.py
+++ b/hypothesis-python/src/hypothesis/extra/cli.py
@@ -37,7 +37,9 @@
 
 import builtins
 import importlib
+import inspect
 import sys
+import types
 from difflib import get_close_matches
 from functools import partial
 from multiprocessing import Pool
@@ -84,6 +86,7 @@ def obj_name(s: str) -> object:
             return importlib.import_module(s)
         except ImportError:
             pass
+        classname = None
         if "." not in s:
             modulename, module, funcname = "builtins", builtins, s
         else:
@@ -91,20 +94,56 @@ def obj_name(s: str) -> object:
             try:
                 module = importlib.import_module(modulename)
             except ImportError as err:
+                try:
+                    modulename, classname = modulename.rsplit(".", 1)
+                    module = importlib.import_module(modulename)
+                except ImportError:
+                    raise click.UsageError(
+                        f"Failed to import the {modulename} module for introspection.  "
+                        "Check spelling and your Python import path, or use the Python API?"
+                    ) from err
+
+        def describe_close_matches(
+            module_or_class: types.ModuleType, objname: str
+        ) -> str:
+            public_names = [
+                name for name in vars(module_or_class) if not name.startswith("_")
+            ]
+            matches = get_close_matches(objname, public_names)
+            if matches:
+                return f"  Closest matches: {matches!r}"
+            else:
+                return ""
+
+        if classname is None:
+            try:
+                return getattr(module, funcname)
+            except AttributeError as err:
                 raise click.UsageError(
-                    f"Failed to import the {modulename} module for introspection.  "
-                    "Check spelling and your Python import path, or use the Python API?"
+                    f"Found the {modulename!r} module, but it doesn't have a "
+                    f"{funcname!r} attribute."
+                    + describe_close_matches(module, funcname)
+                ) from err
+        else:
+            try:
+                func_class = getattr(module, classname)
+            except AttributeError as err:
+                raise click.UsageError(
+                    f"Found the {modulename!r} module, but it doesn't have a "
+                    f"{classname!r} class." + describe_close_matches(module, classname)
+                ) from err
+            try:
+                return getattr(func_class, funcname)
+            except AttributeError as err:
+                if inspect.isclass(func_class):
+                    func_class_is = "class"
+                else:
+                    func_class_is = "attribute"
+                raise click.UsageError(
+                    f"Found the {modulename!r} module and {classname!r} {func_class_is}, "
+                    f"but it doesn't have a {funcname!r} attribute."
+                    + describe_close_matches(func_class, funcname)
                 ) from err
-        try:
-            return getattr(module, funcname)
-        except AttributeError as err:
-            public_names = [name for name in vars(module) if not name.startswith("_")]
-            matches = get_close_matches(funcname, public_names)
-            raise click.UsageError(
-                f"Found the {modulename!r} module, but it doesn't have a "
-                f"{funcname!r} attribute."
-                + (f"  Closest matches: {matches!r}" if matches else "")
-            ) from err
 
     def _refactor(func, fname):
         try:
diff --git a/hypothesis-python/src/hypothesis/extra/ghostwriter.py b/hypothesis-python/src/hypothesis/extra/ghostwriter.py
--- a/hypothesis-python/src/hypothesis/extra/ghostwriter.py
+++ b/hypothesis-python/src/hypothesis/extra/ghostwriter.py
@@ -891,10 +891,15 @@ def magic(
     for thing in modules_or_functions:
         if callable(thing):
             functions.add(thing)
+            # class need to be added for exploration
+            if inspect.isclass(thing):
+                funcs: List[Optional[Any]] = [thing]
+            else:
+                funcs = []
         elif isinstance(thing, types.ModuleType):
             if hasattr(thing, "__all__"):
                 funcs = [getattr(thing, name, None) for name in thing.__all__]
-            else:
+            elif hasattr(thing, "__package__"):
                 pkg = thing.__package__
                 funcs = [
                     v
@@ -906,22 +911,35 @@ def magic(
                 ]
                 if pkg and any(getattr(f, "__module__", pkg) == pkg for f in funcs):
                     funcs = [f for f in funcs if getattr(f, "__module__", pkg) == pkg]
-            for f in funcs:
-                try:
-                    if (
-                        (not is_mock(f))
-                        and callable(f)
-                        and _get_params(f)
-                        and not isinstance(f, enum.EnumMeta)
-                    ):
-                        functions.add(f)
-                        if getattr(thing, "__name__", None):
-                            KNOWN_FUNCTION_LOCATIONS[f] = thing.__name__
-                except (TypeError, ValueError):
-                    pass
         else:
             raise InvalidArgument(f"Can't test non-module non-callable {thing!r}")
 
+        for f in list(funcs):
+            if inspect.isclass(f):
+                funcs += [
+                    v.__get__(f)
+                    for k, v in vars(f).items()
+                    if hasattr(v, "__func__")
+                    and not is_mock(v)
+                    and not k.startswith("_")
+                ]
+        for f in funcs:
+            try:
+                if (
+                    (not is_mock(f))
+                    and callable(f)
+                    and _get_params(f)
+                    and not isinstance(f, enum.EnumMeta)
+                ):
+                    functions.add(f)
+                    if getattr(thing, "__name__", None):
+                        if inspect.isclass(thing):
+                            KNOWN_FUNCTION_LOCATIONS[f] = thing.__module__
+                        else:
+                            KNOWN_FUNCTION_LOCATIONS[f] = thing.__name__
+            except (TypeError, ValueError):
+                pass
+
     imports = set()
     parts = []
 


Test patch:
diff --git a/hypothesis-python/tests/ghostwriter/recorded/fuzz_staticmethod.txt b/hypothesis-python/tests/ghostwriter/recorded/fuzz_staticmethod.txt
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/recorded/fuzz_staticmethod.txt
@@ -0,0 +1,10 @@
+# This test code was written by the `hypothesis.extra.ghostwriter` module
+# and is provided under the Creative Commons Zero public domain dedication.
+
+import test_expected_output
+from hypothesis import given, strategies as st
+
+
+@given(arg=st.integers())
+def test_fuzz_A_Class_a_staticmethod(arg):
+    test_expected_output.A_Class.a_staticmethod(arg=arg)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/hypothesis_module_magic.txt b/hypothesis-python/tests/ghostwriter/recorded/hypothesis_module_magic.txt
--- a/hypothesis-python/tests/ghostwriter/recorded/hypothesis_module_magic.txt
+++ b/hypothesis-python/tests/ghostwriter/recorded/hypothesis_module_magic.txt
@@ -96,6 +96,21 @@ def test_fuzz_settings(
     )
 
 
+@given(name=st.text())
+def test_fuzz_settings_get_profile(name):
+    hypothesis.settings.get_profile(name=name)
+
+
+@given(name=st.text())
+def test_fuzz_settings_load_profile(name):
+    hypothesis.settings.load_profile(name=name)
+
+
+@given(name=st.text(), parent=st.one_of(st.none(), st.builds(settings)))
+def test_fuzz_settings_register_profile(name, parent):
+    hypothesis.settings.register_profile(name=name, parent=parent)
+
+
 @given(observation=st.one_of(st.floats(), st.integers()), label=st.text())
 def test_fuzz_target(observation, label):
     hypothesis.target(observation=observation, label=label)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/magic_builtins.txt b/hypothesis-python/tests/ghostwriter/recorded/magic_builtins.txt
--- a/hypothesis-python/tests/ghostwriter/recorded/magic_builtins.txt
+++ b/hypothesis-python/tests/ghostwriter/recorded/magic_builtins.txt
@@ -31,6 +31,16 @@ def test_fuzz_bin(number):
     bin(number)
 
 
+@given(frm=st.nothing(), to=st.nothing())
+def test_fuzz_bytearray_maketrans(frm, to):
+    bytearray.maketrans(frm, to)
+
+
+@given(frm=st.nothing(), to=st.nothing())
+def test_fuzz_bytes_maketrans(frm, to):
+    bytes.maketrans(frm, to)
+
+
 @given(obj=st.nothing())
 def test_fuzz_callable(obj):
     callable(obj)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/magic_class.txt b/hypothesis-python/tests/ghostwriter/recorded/magic_class.txt
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/recorded/magic_class.txt
@@ -0,0 +1,20 @@
+# This test code was written by the `hypothesis.extra.ghostwriter` module
+# and is provided under the Creative Commons Zero public domain dedication.
+
+import test_expected_output
+from hypothesis import given, strategies as st
+
+
+@given()
+def test_fuzz_A_Class():
+    test_expected_output.A_Class()
+
+
+@given(arg=st.integers())
+def test_fuzz_A_Class_a_classmethod(arg):
+    test_expected_output.A_Class.a_classmethod(arg=arg)
+
+
+@given(arg=st.integers())
+def test_fuzz_A_Class_a_staticmethod(arg):
+    test_expected_output.A_Class.a_staticmethod(arg=arg)
diff --git a/hypothesis-python/tests/ghostwriter/test_expected_output.py b/hypothesis-python/tests/ghostwriter/test_expected_output.py
--- a/hypothesis-python/tests/ghostwriter/test_expected_output.py
+++ b/hypothesis-python/tests/ghostwriter/test_expected_output.py
@@ -65,6 +65,10 @@ class A_Class:
     def a_classmethod(cls, arg: int):
         pass
 
+    @staticmethod
+    def a_staticmethod(arg: int):
+        pass
+
 
 def add(a: float, b: float) -> float:
     return a + b
@@ -86,6 +90,7 @@ def divide(a: int, b: int) -> float:
         ("fuzz_sorted", ghostwriter.fuzz(sorted)),
         ("fuzz_with_docstring", ghostwriter.fuzz(with_docstring)),
         ("fuzz_classmethod", ghostwriter.fuzz(A_Class.a_classmethod)),
+        ("fuzz_staticmethod", ghostwriter.fuzz(A_Class.a_staticmethod)),
         ("fuzz_ufunc", ghostwriter.fuzz(numpy.add)),
         ("magic_gufunc", ghostwriter.magic(numpy.matmul)),
         ("magic_base64_roundtrip", ghostwriter.magic(base64.b64encode)),
@@ -176,6 +181,7 @@ def divide(a: int, b: int) -> float:
                 style="unittest",
             ),
         ),
+        ("magic_class", ghostwriter.magic(A_Class)),
         pytest.param(
             ("magic_builtins", ghostwriter.magic(builtins)),
             marks=[
diff --git a/hypothesis-python/tests/ghostwriter/test_ghostwriter.py b/hypothesis-python/tests/ghostwriter/test_ghostwriter.py
--- a/hypothesis-python/tests/ghostwriter/test_ghostwriter.py
+++ b/hypothesis-python/tests/ghostwriter/test_ghostwriter.py
@@ -257,6 +257,34 @@ def test_invalid_func_inputs(gw, args):
         gw(*args)
 
 
+class A:
+    @classmethod
+    def to_json(cls, obj: Union[dict, list]) -> str:
+        return json.dumps(obj)
+
+    @classmethod
+    def from_json(cls, obj: str) -> Union[dict, list]:
+        return json.loads(obj)
+
+    @staticmethod
+    def static_sorter(seq: Sequence[int]) -> List[int]:
+        return sorted(seq)
+
+
+@pytest.mark.parametrize(
+    "gw,args",
+    [
+        (ghostwriter.fuzz, [A.static_sorter]),
+        (ghostwriter.idempotent, [A.static_sorter]),
+        (ghostwriter.roundtrip, [A.to_json, A.from_json]),
+        (ghostwriter.equivalent, [A.to_json, json.dumps]),
+    ],
+)
+def test_class_methods_inputs(gw, args):
+    source_code = gw(*args)
+    get_test_function(source_code)()
+
+
 def test_run_ghostwriter_fuzz():
     # Our strategy-guessing code works for all the arguments to sorted,
     # and we handle positional-only arguments in calls correctly too.
diff --git a/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py b/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py
--- a/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py
+++ b/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py
@@ -9,6 +9,7 @@
 # obtain one at https://mozilla.org/MPL/2.0/.
 
 import ast
+import itertools
 import json
 import operator
 import re
@@ -43,6 +44,8 @@
         ),
         # Imports submodule (importlib.import_module passes; __import__ fails)
         ("hypothesis.errors.StopTest", lambda: fuzz(StopTest)),
+        # We can write tests for classes even without classmethods or staticmethods
+        ("hypothesis.errors.StopTest", lambda: fuzz(StopTest)),
         # Search for identity element does not print e.g. "You can use @seed ..."
         ("--binary-op operator.add", lambda: binary_operation(operator.add)),
     ],
@@ -116,6 +119,128 @@ def test_can_import_from_scripts_in_working_dir(tmpdir):
     assert "Error: " not in result.stderr
 
 
+CLASS_CODE_TO_TEST = """
+from typing import Sequence, List
+
+def my_func(seq: Sequence[int]) -> List[int]:
+    return sorted(seq)
+
+class MyClass:
+
+    @staticmethod
+    def my_staticmethod(seq: Sequence[int]) -> List[int]:
+        return sorted(seq)
+
+    @classmethod
+    def my_classmethod(cls, seq: Sequence[int]) -> List[int]:
+        return sorted(seq)
+"""
+
+
+@pytest.mark.parametrize("func", ["my_staticmethod", "my_classmethod"])
+def test_can_import_from_class(tmpdir, func):
+    (tmpdir / "mycode.py").write(CLASS_CODE_TO_TEST)
+    result = subprocess.run(
+        f"hypothesis write mycode.MyClass.{func}",
+        capture_output=True,
+        shell=True,
+        text=True,
+        cwd=tmpdir,
+    )
+    assert result.returncode == 0
+    assert "Error: " not in result.stderr
+
+
+@pytest.mark.parametrize(
+    "classname,thing,kind",
+    [
+        ("XX", "", "class"),
+        ("MyClass", " and 'MyClass' class", "attribute"),
+        ("my_func", " and 'my_func' attribute", "attribute"),
+    ],
+)
+def test_error_import_from_class(tmpdir, classname, thing, kind):
+    (tmpdir / "mycode.py").write(CLASS_CODE_TO_TEST)
+    result = subprocess.run(
+        f"hypothesis write mycode.{classname}.XX",
+        capture_output=True,
+        shell=True,
+        text=True,
+        cwd=tmpdir,
+    )
+    msg = f"Error: Found the 'mycode' module{thing}, but it doesn't have a 'XX' {kind}."
+    assert result.returncode == 2
+    assert msg in result.stderr
+
+
+def test_magic_discovery_from_module(tmpdir):
+    (tmpdir / "mycode.py").write(CLASS_CODE_TO_TEST)
+    result = subprocess.run(
+        f"hypothesis write mycode",
+        capture_output=True,
+        shell=True,
+        text=True,
+        cwd=tmpdir,
+    )
+    assert result.returncode == 0
+    assert "my_func" in result.stdout
+    assert "MyClass.my_staticmethod" in result.stdout
+    assert "MyClass.my_classmethod" in result.stdout
+
+
+ROUNDTRIP_CODE_TO_TEST = """
+from typing import Union
+import json
+
+def to_json(json: Union[dict,list]) -> str:
+    return json.dumps(json)
+
+def from_json(json: str) -> Union[dict,list]:
+    return json.loads(json)
+
+class MyClass:
+
+    @staticmethod
+    def to_json(json: Union[dict,list]) -> str:
+        return json.dumps(json)
+
+    @staticmethod
+    def from_json(json: str) -> Union[dict,list]:
+        return json.loads(json)
+
+class OtherClass:
+
+    @classmethod
+    def to_json(cls, json: Union[dict,list]) -> str:
+        return json.dumps(json)
+
+    @classmethod
+    def from_json(cls, json: str) -> Union[dict,list]:
+        return json.loads(json)
+"""
+
+
+def test_roundtrip_correct_pairs(tmpdir):
+    (tmpdir / "mycode.py").write(ROUNDTRIP_CODE_TO_TEST)
+    result = subprocess.run(
+        f"hypothesis write mycode",
+        capture_output=True,
+        shell=True,
+        text=True,
+        cwd=tmpdir,
+    )
+    assert result.returncode == 0
+    for scope1, scope2 in itertools.product(
+        ["mycode.MyClass", "mycode.OtherClass", "mycode"], repeat=2
+    ):
+        round_trip_code = f"""value0 = {scope1}.to_json(json=json)
+    value1 = {scope2}.from_json(json=value0)"""
+        if scope1 == scope2:
+            assert round_trip_code in result.stdout
+        else:
+            assert round_trip_code not in result.stdout
+
+
 def test_empty_module_is_not_error(tmpdir):
     (tmpdir / "mycode.py").write("# Nothing to see here\n")
     result = subprocess.run(