Repository: HypothesisWorks/hypothesis

Issue:
`IOError` on `import hypothesis.strategies`
I'm running Hypothesis in a Docker container. On the first run, everything works fine. One the second run I get this:

```python-traceback
Traceback (most recent call last):
[…]
  File "[my_test].py", line 18, in <module>
    import hypothesis.strategies as st
  File "/usr/local/lib/python2.7/dist-packages/hypothesis/strategies.py", line 71, in <module>
    @defines_strategy
  File "/usr/local/lib/python2.7/dist-packages/hypothesis/strategies.py", line 51, in defines_strategy
    @proxies(strategy_definition)
  File "/usr/local/lib/python2.7/dist-packages/hypothesis/internal/reflection.py", line 497, in accept
    copy_argspec(target.__name__, getargspec(target))(proxy)))
  File "/usr/local/lib/python2.7/dist-packages/hypothesis/internal/reflection.py", line 465, in accept
    u'invocation': u', '.join(invocation_parts)
  File "/usr/local/lib/python2.7/dist-packages/hypothesis/internal/reflection.py", line 372, in source_exec_as_module
    f = open(temporary_filepath, u'w')
IOError: [Errno 2] No such file or directory: u'/.hypothesis/eval_source/hypothesis_temporary_module_9abd2800900986980ad8f68a32f0005fc05800d8_53a57e8c-7a3a-4e9a-9a0c-8a8641d521d4.py'
```

The immediate cause of the `IOError` appears to be that Hypothesis tries to create a directory directly beneath `/`. This has to do with my setup, so I don't blame you. My concern is that Hypothesis tries to create a directory somewhere in the file system on module load. This strikes me as unorthodox.

Specifically, [`hypothesis_home_dir()`](https://github.com/DRMacIver/hypothesis/blob/v1.13.0/src/hypothesis/settings.py#L51) creates a `.hypothesis` directory either under a directory specified by the undocumented environment variable `HYPOTHESIS_STORAGE_DIRECTORY` or under the current working directory. The latter case is what ultimately causes the above exception. The former case I now use to redirect the `.hypothesis` creation to some other place. This, however, is a nasty workaround.

Hypothesis should work on read-only filesystems
- [ ] We should have sensible and useful behavior when the examples database cannot be written (#242)
- [x] We should handle other cases where writes (and subsequent reads) may fail, such as #767 
- [ ] We need a new test run for this starting with `chmod a-w -R .`

The tricky part of this is deciding what *should* happen when the example DB cannot be written to disk.  Do we explicitly fail (with an error message diagnosing the problem and noting how to use the in-memory DB from the command line), or do we log a warning and fail over to the in-memory DB?


Discussion:
Hm, yes, the behaviour when cwd is root isn't great.

Creating a directory on module load is indeed unfortunate. There are semi-good reasons (or, at least, probably unavoidable ones) for that, but it would be nice not to.

I'm rather surprised everything works on the first run. What's different there?

I don't know. For now, I didn't manage to make a minimal example, either. Maybe later.

I guess there are also reasons that `.hypothesis` can't be a temporary directory? I expect it's about the example database.

It's partly about the example database, partly that if you try to clean up the generated files on exit it breaks coverage so you'll end up spamming /tmp/ with new directories.

However the reasons, there should be documentation and a proper exception text in order to lessen the suprise for the users. If you want, I could contribute that, but I'm not planning to make more contributions unless I find another itch.

As to why it works the first time and not the second: (found this rather stupid quirk after some tries at building a minimal example) when I run a fresh container, it starts out in `/var/lib/something`, but on subsequent runs it's in `/`, where the user doesn't have write permission anymore. Don't know why the directory change; I haven't built it.

Proposed solution:

- document creation of this directory and where it goes
- document the environment variable
- delay creating the directory until it's actually going to be used (so you can work around it by using an in-memory example database)
- give a more useful error if it fails anyway
#767 (or a very closely related bug) can be caused by a similar problem, where files can't be written at runtime.

I think we should handle this case, and test that hypothesis works on a read-only filesystem.  Obviously this precludes saving examples, but everything else should be possible.
Try`$TMPDIR`, then `/tmp` , then `$HOME` , then `/dev/shm` in that order first.  Then log a warning and fail over to in-memory DB.
Personally I prefer the explicit failure - not super keen on writing to someone's `$HOME` despite config, and tmpdirs are equivalent to the in-memory DB but with more serialization and IO.
> tmpdirs are equivalent to the in-memory DB but with more serialization and IO

Depends if we can ever read it again – a file on disk persists between multiple test runs, an in-memory database doesn’t.
In general use terms, I'd rather hypothesis try to go in blindly in worst case scenario over it failing entirely. Maybe add warnings if not possible at all?
\@alexwlchan, I guess tempfiles *might* persist but I'm reluctant to add OS-specific and hard-to-test fallback logic.

\@CodyKochmann, I'll take that as a vote for 'log warning and fail over to the in-memory DB'

Resolving patch:
diff --git a/src/hypothesis/_settings.py b/src/hypothesis/_settings.py
--- a/src/hypothesis/_settings.py
+++ b/src/hypothesis/_settings.py
@@ -32,7 +32,6 @@
 import attr
 
 from hypothesis.errors import InvalidArgument, HypothesisDeprecationWarning
-from hypothesis.configuration import hypothesis_home_dir
 from hypothesis.internal.compat import text_type
 from hypothesis.utils.conventions import UniqueIdentifier, not_set
 from hypothesis.internal.reflection import proxies, \
@@ -51,9 +50,6 @@
 all_settings = {}
 
 
-_db_cache = {}
-
-
 class settingsProperty(object):
 
     def __init__(self, name, show_default):
@@ -121,7 +117,7 @@ class settings(settingsMeta('settings', (object,), {})):
     """
 
     _WHITELISTED_REAL_PROPERTIES = [
-        '_database', '_construction_complete', 'storage'
+        '_construction_complete', 'storage'
     ]
     __definitions_are_locked = False
     _profiles = {}
@@ -135,14 +131,17 @@ def __getattr__(self, name):
         else:
             raise AttributeError('settings has no attribute %s' % (name,))
 
-    def __init__(
-            self,
-            parent=None,
-            **kwargs
-    ):
+    def __init__(self, parent=None, **kwargs):
+        if (
+            kwargs.get('database', not_set) is not_set and
+            kwargs.get('database_file', not_set) is not not_set
+        ):
+            if kwargs['database_file'] is None:
+                kwargs['database'] = None
+            else:
+                from hypothesis.database import ExampleDatabase
+                kwargs['database'] = ExampleDatabase(kwargs['database_file'])
         self._construction_complete = False
-        self._database = kwargs.pop('database', not_set)
-        database_file = kwargs.get('database_file', not_set)
         deprecations = []
         defaults = parent or settings.default
         if defaults is not None:
@@ -156,12 +155,10 @@ def __init__(
                     if setting.validator:
                         kwargs[setting.name] = setting.validator(
                             kwargs[setting.name])
-            if self._database is not_set and database_file is not_set:
-                self._database = defaults.database
         for name, value in kwargs.items():
             if name not in all_settings:
                 raise InvalidArgument(
-                    'Invalid argument %s' % (name,))
+                    'Invalid argument: %r is not a valid setting' % (name,))
             setattr(self, name, value)
         self.storage = threading.local()
         self._construction_complete = True
@@ -237,7 +234,7 @@ def new_test(*args, **kwargs):
 
     @classmethod
     def define_setting(
-        cls, name, description, default, options=None, deprecation=None,
+        cls, name, description, default, options=None,
         validator=None, show_default=True, future_default=not_set,
         deprecation_message=None,
     ):
@@ -275,12 +272,6 @@ def lock_further_definitions(cls):
     def __setattr__(self, name, value):
         if name in settings._WHITELISTED_REAL_PROPERTIES:
             return object.__setattr__(self, name, value)
-        elif name == 'database':
-            assert self._construction_complete
-            raise AttributeError(
-                'settings objects are immutable and may not be assigned to'
-                ' after construction.'
-            )
         elif name in all_settings:
             if self._construction_complete:
                 raise AttributeError(
@@ -310,27 +301,6 @@ def __repr__(self):
         bits.sort()
         return 'settings(%s)' % ', '.join(bits)
 
-    @property
-    def database(self):
-        """An ExampleDatabase instance to use for storage of examples. May be
-        None.
-
-        If this was explicitly set at settings instantiation then that
-        value will be used (even if it was None). If not and the
-        database_file setting is not None this will be lazily loaded as
-        an ExampleDatabase, using that file the first time that this
-        property is accessed on a particular thread.
-        """
-        if self._database is not_set and self.database_file is not None:
-            from hypothesis.database import ExampleDatabase
-            if self.database_file not in _db_cache:
-                _db_cache[self.database_file] = (
-                    ExampleDatabase(self.database_file))
-            return _db_cache[self.database_file]
-        if self._database is not_set:
-            self._database = None
-        return self._database
-
     def __enter__(self):
         default_context_manager = default_variable.with_value(self)
         self.defaults_stack().append(default_context_manager)
@@ -521,18 +491,49 @@ def _validate_timeout(n):
     future_default=False,
 )
 
+
+def _validate_database(db, __from_db_file=False):
+    from hypothesis.database import ExampleDatabase
+    if db is None or isinstance(db, ExampleDatabase):
+        return db
+    if __from_db_file or db is not_set:
+        return ExampleDatabase(db)
+    raise InvalidArgument(
+        'Arguments to the database setting must be None or an instance of '
+        'ExampleDatabase.  Try passing database=ExampleDatabase(%r), or '
+        'construct and use one of the specific subclasses in '
+        'hypothesis.database' % (db,)
+    )
+
+
 settings.define_setting(
-    'database_file',
-    default=lambda: (
-        os.getenv('HYPOTHESIS_DATABASE_FILE') or
-        os.path.join(hypothesis_home_dir(), 'examples')
-    ),
+    'database',
+    default=lambda: _validate_database(not_set),
     show_default=False,
     description="""
-    An instance of hypothesis.database.ExampleDatabase that will be
+An instance of hypothesis.database.ExampleDatabase that will be
 used to save examples to and load previous examples from. May be None
-in which case no storage will be used.
-"""
+in which case no storage will be used, `:memory:` for an in-memory
+database, or any path for a directory-based example database.
+""",
+    validator=_validate_database,
+)
+
+settings.define_setting(
+    'database_file',
+    default=not_set,
+    show_default=False,
+    description="""
+The file or directory location to save and load previously tried examples;
+`:memory:` for an in-memory cache or None to disable caching entirely.
+""",
+    validator=lambda f: _validate_database(f, __from_db_file=True),
+    deprecation_message="""
+The `database_file` setting is deprecated in favor of the `database`
+setting, and will be removed in a future version.  It only exists at
+all for complicated historical reasons and you should just use
+`database` instead.
+""",
 )
 
 
diff --git a/src/hypothesis/database.py b/src/hypothesis/database.py
--- a/src/hypothesis/database.py
+++ b/src/hypothesis/database.py
@@ -20,19 +20,48 @@
 import os
 import re
 import binascii
+import warnings
 import threading
 from hashlib import sha1
 from contextlib import contextmanager
 
+from hypothesis.errors import HypothesisWarning
 from hypothesis._settings import note_deprecation
+from hypothesis.configuration import storage_directory
 from hypothesis.internal.compat import FileNotFoundError, hbytes, \
     b64decode, b64encode
+from hypothesis.utils.conventions import not_set
 
 sqlite3 = None
 SQLITE_PATH = re.compile(r"\.\(db|sqlite|sqlite3\)$")
 
 
 def _db_for_path(path=None):
+    if path is not_set:
+        path = os.getenv('HYPOTHESIS_DATABASE_FILE')
+        if path is not None:  # pragma: no cover
+            # Note: we should retain an explicit deprecation warning for a
+            # further period after this is removed, to ease debugging for
+            # anyone migrating to a new version.
+            note_deprecation(
+                'The $HYPOTHESIS_DATABASE_FILE environment variable is '
+                'deprecated, and will be ignored by a future version of '
+                'Hypothesis.  Configure your database location via a '
+                'settings profile instead.'
+            )
+            return _db_for_path(path)
+        # Note: storage_directory attempts to create the dir in question, so
+        # if os.access fails there *must* be a fatal permissions issue.
+        path = storage_directory('examples')
+        if os.access(path, os.R_OK | os.W_OK | os.X_OK):
+            return _db_for_path(path)
+        else:  # pragma: no cover
+            warnings.warn(HypothesisWarning(
+                'The database setting is not configured, and the default '
+                'location is unusable - falling back to an in-memory '
+                'database for this session.  path=%r' % (path,)
+            ))
+            return InMemoryExampleDatabase()
     if path in (None, ':memory:'):
         return InMemoryExampleDatabase()
     path = str(path)


Test patch:
diff --git a/tests/cover/test_duplication.py b/tests/cover/test_duplication.py
--- a/tests/cover/test_duplication.py
+++ b/tests/cover/test_duplication.py
@@ -38,6 +38,7 @@ def test_does_not_duplicate_blocks(n):
     counts = Counter()
 
     @given(Blocks(n))
+    @settings(database=None)
     def test(b):
         counts[b] += 1
     test()
diff --git a/tests/cover/test_settings.py b/tests/cover/test_settings.py
--- a/tests/cover/test_settings.py
+++ b/tests/cover/test_settings.py
@@ -229,10 +229,21 @@ def test_can_have_none_database():
     assert settings(database=None).database is None
 
 
+@pytest.mark.parametrize('db', [None, ExampleDatabase(':memory:')])
+def test_database_type_must_be_ExampleDatabase(db):
+    with settings(database=db):
+        settings_property_db = settings.database
+        with pytest.raises(InvalidArgument):
+            settings(database='.hypothesis/examples')
+        assert settings.database is settings_property_db
+
+
+@checks_deprecated_behaviour
 def test_can_have_none_database_file():
     assert settings(database_file=None).database is None
 
 
+@checks_deprecated_behaviour
 def test_can_override_database_file():
     f = mkdtemp()
     x = settings(database_file=f)
diff --git a/tests/cover/test_shrinking_limits.py b/tests/cover/test_shrinking_limits.py
--- a/tests/cover/test_shrinking_limits.py
+++ b/tests/cover/test_shrinking_limits.py
@@ -33,6 +33,6 @@ def tracktrue(s):
 
     find(
         st.binary(min_size=100, max_size=100), tracktrue,
-        settings=settings(max_shrinks=1)
+        settings=settings(max_shrinks=1, database=None)
     )
     assert len(seen) == 2
diff --git a/tests/cover/test_verbosity.py b/tests/cover/test_verbosity.py
--- a/tests/cover/test_verbosity.py
+++ b/tests/cover/test_verbosity.py
@@ -60,7 +60,7 @@ def test_foo(x):
 
 def test_includes_progress_in_verbose_mode():
     with capture_verbosity(Verbosity.verbose) as o:
-        with settings(verbosity=Verbosity.verbose):
+        with settings(verbosity=Verbosity.verbose, database=None):
             find(lists(integers()), lambda x: sum(x) >= 1000000)
 
     out = o.getvalue()
diff --git a/tests/cover/test_weird_settings.py b/tests/cover/test_weird_settings.py
--- a/tests/cover/test_weird_settings.py
+++ b/tests/cover/test_weird_settings.py
@@ -19,8 +19,10 @@
 
 from hypothesis import given, settings
 from hypothesis import strategies as st
+from tests.common.utils import checks_deprecated_behaviour
 
 
+@checks_deprecated_behaviour
 def test_setting_database_to_none_disables_the_database():
     @given(st.booleans())
     @settings(database_file=None)