Repository: HypothesisWorks/hypothesis

Issue:
Replace pickle with json and tuple in charmap.py
As described in #1511, we use the `pickle` module to cache some data about unicode categories between runs, because it's relatively expensive to calculate.

However, `pickle` can store arbitrary Python objects - and it's also harder for other tools to deal with.  Instead, we could replace it with the `json` module, and just add a loop or comprehension that converted between lists and tuples as appropriate on each end.

For bonus points, you could also add some assertions that every key in the dictionary is a valid unicode category name, and that each value is a tuple of 2-tuples (that is, tuples of length 2) and that both elements of *that* tuple are integers.  It would even be possible to check that concatenating the 2-tuples gives a sorted sequence of integers, but that might be overkill :wink: 

[Here's the code in question.](https://github.com/HypothesisWorks/hypothesis/blob/026fa3598e10be651178c035272c1016a329ad45/hypothesis-python/src/hypothesis/internal/charmap.py#L59-L84)


Discussion:

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/charmap.py b/hypothesis-python/src/hypothesis/internal/charmap.py
--- a/hypothesis-python/src/hypothesis/internal/charmap.py
+++ b/hypothesis-python/src/hypothesis/internal/charmap.py
@@ -20,7 +20,7 @@
 import os
 import sys
 import gzip
-import pickle
+import json
 import tempfile
 import unicodedata
 
@@ -37,7 +37,7 @@
 def charmap_file():
     return os.path.join(
         storage_directory('unicodedata', unicodedata.unidata_version),
-        'charmap.pickle.gz'
+        'charmap.json.gz'
     )
 
 
@@ -59,7 +59,7 @@ def charmap():
         f = charmap_file()
         try:
             with gzip.GzipFile(f, 'rb') as i:
-                _charmap = dict(pickle.load(i))
+                tmp_charmap = dict(json.loads(i))
 
         except Exception:
             tmp_charmap = {}
@@ -70,8 +70,6 @@ def charmap():
                     rs[-1][-1] += 1
                 else:
                     rs.append([i, i])
-            _charmap = {k: tuple(tuple(pair) for pair in pairs)
-                        for k, pairs in tmp_charmap.items()}
 
             try:
                 # Write the Unicode table atomically
@@ -79,11 +77,24 @@ def charmap():
                 os.close(fd)
                 # Explicitly set the mtime to get reproducible output
                 with gzip.GzipFile(tmpfile, 'wb', mtime=1) as o:
-                    pickle.dump(sorted(_charmap.items()), o,
-                                pickle.HIGHEST_PROTOCOL)
+                    result = json.dumps(sorted(tmp_charmap.items()))
+                    o.write(result.encode())
+
                 os.rename(tmpfile, f)
             except Exception:
                 pass
+
+        # convert between lists and tuples
+        _charmap = {k: tuple(tuple(pair) for pair in pairs)
+                    for k, pairs in tmp_charmap.items()}
+        # each value is a tuple of 2-tuples (that is, tuples of length 2)
+        # and that both elements of that tuple are integers.
+        for vs in _charmap.values():
+            ints = list(sum(vs, ()))
+            assert all([isinstance(x, int) for x in ints])
+            assert ints == sorted(ints)
+            assert all([len(tup) == 2 for tup in vs])
+
     assert _charmap is not None
     return _charmap
 


Test patch: