Repository: HypothesisWorks/hypothesis

Issue:
Initial calculation of unicode table is not atomic
If you're using e.g. pytest-xdist this can result in a corrupt unicode table, which is bad. Instead the file should be created under a temporary name then moved atomically.



Discussion:
Relevant code is https://github.com/DRMacIver/hypothesis/blob/master/src/hypothesis/internal/charmap.py#L56

I think we hit this in CI all the time. Here's the stack trace if helpful:

```
reinfer/models/voc/label/tests/test_batch.py:17: in <module>
    from ...shared.tests.test_prediction_store import s_predictions
/env/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py:171: in load_module
    py.builtin.exec_(co, mod.__dict__)
reinfer/models/voc/shared/tests/test_prediction_store.py:46: in <module>
    s.tuples(s.text(min_size=1, max_size=3).map(lambda t: t.encode('utf-8')),
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/strategies.py:130: in map
    pack=pack, strategy=self
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/strategies.py:245: in __init__
    self.is_empty = strategy.is_empty
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py:59: in is_empty
    return self.wrapped_strategy.is_empty
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py:67: in wrapped_strategy
    **self.__kwargs
/env/local/lib/python2.7/site-packages/hypothesis/strategies.py:655: in text
    max_size=max_size
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/strings.py:104: in __init__
    strategy=list_of_one_char_strings_strategy
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/strategies.py:245: in __init__
    self.is_empty = strategy.is_empty
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py:59: in is_empty
    return self.wrapped_strategy.is_empty
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py:67: in wrapped_strategy
    **self.__kwargs
/env/local/lib/python2.7/site-packages/hypothesis/strategies.py:439: in lists
    if elements.is_empty:
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py:59: in is_empty
    return self.wrapped_strategy.is_empty
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py:67: in wrapped_strategy
    **self.__kwargs
/env/local/lib/python2.7/site-packages/hypothesis/strategies.py:620: in characters
    max_codepoint=max_codepoint)
/env/local/lib/python2.7/site-packages/hypothesis/searchstrategy/strings.py:46: in __init__
    max_codepoint=max_codepoint,
/env/local/lib/python2.7/site-packages/hypothesis/internal/charmap.py:143: in query
    catkey = _category_key(exclude_categories, include_categories)
/env/local/lib/python2.7/site-packages/hypothesis/internal/charmap.py:104: in _category_key
    cs = categories()
/env/local/lib/python2.7/site-packages/hypothesis/internal/charmap.py:70: in categories
    cm = charmap()
/env/local/lib/python2.7/site-packages/hypothesis/internal/charmap.py:59: in charmap
    _charmap = dict(pickle.loads(i.read()))
/usr/lib/python2.7/pickle.py:1388: in loads
    return Unpickler(file).load()
/usr/lib/python2.7/pickle.py:864: in load
    dispatch[key](self)
/usr/lib/python2.7/pickle.py:886: in load_eof
    raise EOFError
E   EOFError
```


Resolving patch:
diff --git a/src/hypothesis/internal/charmap.py b/src/hypothesis/internal/charmap.py
--- a/src/hypothesis/internal/charmap.py
+++ b/src/hypothesis/internal/charmap.py
@@ -20,6 +20,8 @@
 import sys
 import gzip
 import pickle
+import shutil
+import tempfile
 import unicodedata
 
 from hypothesis.configuration import storage_directory
@@ -53,8 +55,13 @@ def charmap():
             data = sorted(
                 (k, tuple((map(tuple, v))))
                 for k, v in _charmap.items())
-            with gzip.GzipFile(f, 'wb', mtime=1) as o:
+
+            # Write the Unicode table atomically
+            _, tmpfile = tempfile.mkstemp()
+            with gzip.GzipFile(tmpfile, 'wb', mtime=1) as o:
                 o.write(pickle.dumps(data, pickle.HIGHEST_PROTOCOL))
+            shutil.move(tmpfile, f)
+
         with gzip.open(f, 'rb') as i:
             _charmap = dict(pickle.loads(i.read()))
     assert _charmap is not None


Test patch: