Repository: numpy/numpy

Issue:
MAINT: ignore `.hypothesis/`
<!-- Please be sure you are following the instructions in the dev guidelines
http://www.numpy.org/devdocs/dev/development_workflow.html
-->

<!-- We'd appreciate it if your commit message is properly formatted
http://www.numpy.org/devdocs/dev/development_workflow.html#writing-the-commit-message
-->

I ran the test using `np.test()`. Then the `.hypothesis` directory was created. This seems to a byproduct of the test.


Discussion:
Thanks @marload. This looks like a bug that should not just be ignored. `np.test()` can be run on installed versions of `numpy`, and it should not leave behind any stray files.
ping @Zac-HD 
Should the same be applied to `.pytest_cache` which is already in `.gitignore`?
> Should the same be applied to `.pytest_cache` which is already in `.gitignore`?

Yes, `pytest` should clean up after itself as well, and we should remove it from `.gitignore` to ensure we catch regressions. Unless there's some setting for enabling the cache, that we could use to disable by default and let devs enable for speedups if that helps (which is hard to imagine, test discovery is very fast compared to running the tests).
I would suggest adding `.hypothesis/` to `.gitignore`, but also using a different settings profile when invoked via `np.test()`.  

Because this mode is really just checking for packaging or config related regressions [rather than looking for bugs](https://blog.nelhage.com/post/two-kinds-of-testing/), it should probably set use the `derandomize=True` and `database=None` settings to ensure determinism and avoid persisting any data.  That's presently configured here:

https://github.com/numpy/numpy/blob/c31ba764540a12843b3caf711b447fe3e08c5a46/numpy/conftest.py#L16-L20

I'd be happy to write the PR for that, if someone can show me how to tell if we're running `np.test()` v.s. running the tests in development or CI.
> how to tell if we're running `np.test()` v.s. running the tests in development or CI.

Calling `np.test()` uses the `__call__` method of `PytestTester` from `numpy/_pytesttester.py`. That may be a place to set the profile. Do you know if pytest also uses a profile?
> I would suggest adding .hypothesis/ to .gitignore

Another problem with this approach is that it doesn't catch the case where `pytest` is called from a subdirectory, e.g.

```bash
cd numpy/lib/tests
pytest test_function_base.py
```
This creates a `.hypothesis` folder in `numpy/lib/tests`. It'd be good if this use-case were handled as well.
> Calling `np.test()` uses the `__call__` method of `PytestTester` from `numpy/_pytesttester.py`. That may be a place to set the profile. Do you know if pytest also uses a profile?

Sounds good!

Assuming you mean "does invoking `pytest test_foo.py` in Numpy use a Hypothesis profile", the answer is yes - that's set up in the root `conftest.py` I linked to above.

> > I would suggest adding .hypothesis/ to .gitignore
> 
> Another problem with this approach is that it doesn't catch the case where `pytest` is called from a subdirectory, ... this creates a `.hypothesis` folder in `numpy/lib/tests`. It'd be good if this use-case were handled as well.

gitignoring `.hypothesis/` does in fact ignore subdirectories as well; I think you're conflating it with `/.hypothesis/` which would only ignore it in the root directory of the repo.

You're correct though that it would be better to force the database dir to always be at the repo root.  I'll set that up too, so that the cache is shared regardless of the working directory from which `pytest` is invoked.

Resolving patch:
diff --git a/numpy/_pytesttester.py b/numpy/_pytesttester.py
--- a/numpy/_pytesttester.py
+++ b/numpy/_pytesttester.py
@@ -140,6 +140,9 @@ def __call__(self, label='fast', verbose=1, extra_argv=None,
         import pytest
         import warnings
 
+        # Imported after pytest to enable assertion rewriting
+        import hypothesis
+
         module = sys.modules[self.module_name]
         module_path = os.path.abspath(module.__path__[0])
 
@@ -202,6 +205,14 @@ def __call__(self, label='fast', verbose=1, extra_argv=None,
 
         pytest_args += ["--pyargs"] + list(tests)
 
+        # This configuration is picked up by numpy.conftest, and ensures that
+        # running `np.test()` is deterministic and does not write any files.
+        # See https://hypothesis.readthedocs.io/en/latest/settings.html
+        hypothesis.settings.register_profile(
+            name="np.test() profile",
+            deadline=None, print_blob=True, database=None, derandomize=True,
+            suppress_health_check=hypothesis.HealthCheck.all(),
+        )
 
         # run tests.
         _show_numpy_info()
diff --git a/numpy/conftest.py b/numpy/conftest.py
--- a/numpy/conftest.py
+++ b/numpy/conftest.py
@@ -2,6 +2,7 @@
 Pytest configuration and fixtures for the Numpy test suite.
 """
 import os
+import tempfile
 
 import hypothesis
 import pytest
@@ -13,11 +14,23 @@
 _old_fpu_mode = None
 _collect_results = {}
 
+# Use a known and persistent tmpdir for hypothesis' caches, which
+# can be automatically cleared by the OS or user.
+hypothesis.configuration.set_hypothesis_home_dir(
+    os.path.join(tempfile.gettempdir(), ".hypothesis")
+)
 # See https://hypothesis.readthedocs.io/en/latest/settings.html
 hypothesis.settings.register_profile(
     name="numpy-profile", deadline=None, print_blob=True,
 )
-hypothesis.settings.load_profile("numpy-profile")
+# We try loading the profile defined by np.test(), which disables the
+# database and forces determinism, and fall back to the profile defined
+# above if we're running pytest directly.  The odd dance is required
+# because np.test() executes this file *after* its own setup code.
+try:
+    hypothesis.settings.load_profile("np.test() profile")
+except hypothesis.errors.InvalidArgument:  # profile not found
+    hypothesis.settings.load_profile("numpy-profile")
 
 
 def pytest_configure(config):


Test patch: