Repository: dwavesystems/dimod

Issue:
`bqm.to_serializable(use_bytes=True)` produces an offset with numpy type
**Description**
The offset value produced by `bqm.to_serializable(use_bytes=True)` is a numpy type, which is not generally serializable.

**Steps To Reproduce**
```python
>>> bqm = dimod.BinaryQuadraticModel.from_ising({}, {"ab": 1})
>>> type(bqm.to_serializable(use_bytes=True)["offset"])
numpy.float32
```

**Expected Behavior**
All the types produced in `bqm.to_serializable(use_bytes=True)` should be bson serializable.


Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=h1) Report
> Merging [#384](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=desc) into [master](https://codecov.io/gh/dwavesystems/dimod/commit/72188236007a1225294a994f9684560de657d865?src=pr&el=desc) will **increase** coverage by `1.34%`.
> The diff coverage is `89.47%`.

[![Impacted file tree graph](https://codecov.io/gh/dwavesystems/dimod/pull/384/graphs/tree.svg?width=650&token=KyxipO4fvT&height=150&src=pr)](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #384      +/-   ##
==========================================
+ Coverage   91.63%   92.97%   +1.34%     
==========================================
  Files          40       40              
  Lines        2450     2464      +14     
==========================================
+ Hits         2245     2291      +46     
+ Misses        205      173      -32
```


| [Impacted Files](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [dimod/serialization/bson.py](https://codecov.io/gh/dwavesystems/dimod/pull/384/diff?src=pr&el=tree#diff-ZGltb2Qvc2VyaWFsaXphdGlvbi9ic29uLnB5) | `96% <89.47%> (+82.11%)` | :arrow_up: |
| [dimod/binary\_quadratic\_model.py](https://codecov.io/gh/dwavesystems/dimod/pull/384/diff?src=pr&el=tree#diff-ZGltb2QvYmluYXJ5X3F1YWRyYXRpY19tb2RlbC5weQ==) | `95.72% <0%> (+0.35%)` | :arrow_up: |
| [dimod/serialization/json.py](https://codecov.io/gh/dwavesystems/dimod/pull/384/diff?src=pr&el=tree#diff-ZGltb2Qvc2VyaWFsaXphdGlvbi9qc29uLnB5) | `95.32% <0%> (+0.93%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=footer). Last update [7218823...2dad440](https://codecov.io/gh/dwavesystems/dimod/pull/384?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

arcondello: Marking as WIP for now, just comment here when you want me to take a look again!

Patch:
diff --git a/dimod/binary_quadratic_model.py b/dimod/binary_quadratic_model.py
index 86003d9a..55f1ecb2 100644
--- a/dimod/binary_quadratic_model.py
+++ b/dimod/binary_quadratic_model.py
@@ -1656,13 +1656,24 @@ class BinaryQuadraticModel(abc.Sized, abc.Container, abc.Iterable):
 
         return coo.load(obj, cls=cls, vartype=vartype)
 
-    def to_serializable(self, use_bytes=False):
+    def to_serializable(self, use_bytes=False, bias_dtype=np.float32,
+                        bytes_type=bytes):
         """Convert the binary quadratic model to a serializable object.
 
         Args:
             use_bytes (bool, optional, default=False):
                 If True, a compact representation representing the biases as bytes is used.
 
+            bias_dtype (numpy.dtype, optional, default=numpy.float32):
+                If `use_bytes` is True, this numpy dtype will be used to
+                represent the bias values in the serialized format.
+
+            bytes_types (class, optional, default=bytes):
+                This class will be used to wrap the bytes objects in the
+                serialization if `use_bytes` is true. Useful for when using
+                Python 2 and using BSON encoding, which will not accept the raw
+                `bytes` type, so `bson.Binary` can be used instead.
+
         Returns:
             dict: An object that can be serialized.
 
@@ -1692,12 +1703,7 @@ class BinaryQuadraticModel(abc.Sized, abc.Container, abc.Iterable):
             >>> import bson
             ...
             >>> bqm = dimod.BinaryQuadraticModel({'a': -1.0, 'b': 1.0}, {('a', 'b'): -1.0}, 0.0, dimod.SPIN)
-            >>> doc = bqm.to_serializable(use_bytes=True)
-            >>> for key in ['linear', 'quadratic_vals', 'quadratic_head', 'quadratic_tail']:
-            >>>     if key in doc:
-            >>>         doc[key] = bson.binary.Binary(doc[key])
-            >>>     else:
-            >>>         assert doc['as_complete']
+            >>> doc = bqm.to_serializable(use_bytes=True, bytes_type=bson.Binary)
             >>> b = bson.BSON.encode(doc)
 
         See also:
@@ -1713,7 +1719,8 @@ class BinaryQuadraticModel(abc.Sized, abc.Container, abc.Iterable):
         if use_bytes:
             from dimod.serialization.bson import bqm_bson_encoder
 
-            return bqm_bson_encoder(self)
+            return bqm_bson_encoder(self, bias_dtype=bias_dtype,
+                                    bytes_type=bytes_type)
         else:
             # we we don't use bytes then use json encoder
             from dimod.serialization.json import DimodEncoder
diff --git a/dimod/serialization/bson.py b/dimod/serialization/bson.py
index e4fd3e90..ee0da44b 100644
--- a/dimod/serialization/bson.py
+++ b/dimod/serialization/bson.py
@@ -13,6 +13,7 @@
 #    limitations under the License.
 #
 # ================================================================================================
+from __future__ import absolute_import, division
 import itertools
 
 import numpy as np
@@ -20,27 +21,32 @@ import numpy as np
 from dimod.binary_quadratic_model import BinaryQuadraticModel
 
 
-def bqm_bson_encoder(bqm):
+def bqm_bson_encoder(bqm, bytes_type=bytes, bias_dtype=np.float32):
     """todo"""
+
     num_variables = len(bqm)
 
     index_dtype = np.uint32
     if num_variables <= 2**16:
         index_dtype = np.uint16
 
-    variable_order = sorted(bqm.linear)
-    num_possible_edges = num_variables*(num_variables - 1) // 2
+    try:
+        variable_order = sorted(bqm.variables)
+    except TypeError:
+        variable_order = list(bqm.variables)
+    num_possible_edges = max(num_variables*(num_variables - 1) // 2, 1)
     density = len(bqm.quadratic) / num_possible_edges
     as_complete = density >= 0.5
 
     lin, (i, j, _vals), off = bqm.to_numpy_vectors(
-        dtype=np.float32,
+        dtype=bias_dtype,
         index_dtype=index_dtype,
         sort_indices=as_complete,
         variable_order=variable_order)
+    off = float(off)
 
     if as_complete:
-        vals = np.zeros(num_possible_edges, dtype=np.float32)
+        vals = np.zeros(num_possible_edges, dtype=bias_dtype)
 
         def mul(a, b):
             return np.multiply(a, b, dtype=np.int64)
@@ -52,28 +58,32 @@ def bqm_bson_encoder(bqm):
     else:
         vals = _vals
 
+    lvals, qvals = bytes_type(lin.tobytes()), bytes_type(vals.tobytes())
+
     doc = {
         "as_complete": as_complete,
-        "linear": lin.tobytes(),
-        "quadratic_vals": vals.tobytes(),
+        "linear": lvals,
+        "quadratic_vals": qvals,
         "variable_type": "SPIN" if bqm.vartype == bqm.SPIN else "BINARY",
         "offset": off,
         "variable_order": variable_order,
         "index_dtype": np.dtype(index_dtype).str,
+        "bias_dtype": np.dtype(bias_dtype).str,
     }
 
     if not as_complete:
-        doc["quadratic_head"] = i.tobytes()
-        doc["quadratic_tail"] = j.tobytes()
+        ii, jj = bytes_type(i.tobytes()), bytes_type(j.tobytes())
+        doc["quadratic_head"] = ii
+        doc["quadratic_tail"] = jj
 
     return doc
 
 
 def bqm_bson_decoder(doc, cls=BinaryQuadraticModel):
-    lin = np.frombuffer(doc["linear"], dtype=np.float32)
+    bias_dtype, index_dtype = doc["bias_dtype"], doc["index_dtype"]
+    lin = np.frombuffer(doc["linear"], dtype=bias_dtype)
     num_variables = len(lin)
-    vals = np.frombuffer(doc["quadratic_vals"], dtype=np.float32)
-    index_dtype = doc["index_dtype"]
+    vals = np.frombuffer(doc["quadratic_vals"], dtype=bias_dtype)
     if doc["as_complete"]:
         i, j = zip(*itertools.combinations(range(num_variables), 2))
     else:


Test patch:
diff --git a/tests/test_binary_quadratic_model.py b/tests/test_binary_quadratic_model.py
index 1d65fbc2..a41831ce 100644
--- a/tests/test_binary_quadratic_model.py
+++ b/tests/test_binary_quadratic_model.py
@@ -2020,3 +2020,10 @@ class TestConvert(unittest.TestCase):
         self.assertIn('tag', new_bqm.info)
         self.assertEqual(new_bqm.info['tag'], 5)
         self.assertIn(('a', "complex key"), new_bqm.linear)
+
+    def test_bson_serialization(self):
+        linear = {'a': -1, 4: 1, ('a', "complex key"): 3}
+        quadratic = {('a', 'c'): 1.2, ('b', 'c'): .3, ('a', 3): -1}
+        bqm = dimod.BinaryQuadraticModel(linear, quadratic, 3, dimod.SPIN,
+                                         tag=5)
+        byte_serializable = bqm.to_serializable(use_bytes=True)
diff --git a/tests/test_serialization_bson.py b/tests/test_serialization_bson.py
new file mode 100644
index 00000000..ca12925a
--- /dev/null
+++ b/tests/test_serialization_bson.py
@@ -0,0 +1,131 @@
+# Copyright 2019 D-Wave Systems Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+# ================================================================================================
+import unittest
+import dimod
+from dimod.serialization.bson import bqm_bson_decoder, bqm_bson_encoder
+import numpy as np
+from six import PY2
+
+
+try:
+    import bson
+    _bson_imported = True
+except ImportError:
+    _bson_imported = False
+
+
+class TestBSONSerialization(unittest.TestCase):
+    def test_empty_bqm(self):
+        bqm = dimod.BinaryQuadraticModel.from_qubo({})
+        encoded = bqm_bson_encoder(bqm)
+        expected_encoding = {
+            'as_complete': False,
+            'linear': b'',
+            'quadratic_vals': b'',
+            'variable_type': 'BINARY',
+            'offset': 0.0,
+            'variable_order': [],
+            'index_dtype': '<u2',
+            'bias_dtype': '<f4',
+            'quadratic_head': b'',
+            'quadratic_tail': b'',
+        }
+        self.assertDictEqual(encoded, expected_encoding)
+        decoded = bqm_bson_decoder(encoded)
+        self.assertEqual(bqm, decoded)
+
+    def test_single_variable_bqm(self):
+        bqm = dimod.BinaryQuadraticModel.from_ising({"a": -1}, {})
+        encoded = bqm_bson_encoder(bqm)
+        expected_encoding = {
+            'as_complete': False,
+            'linear': b'\x00\x00\x80\xbf',
+            'quadratic_vals': b'',
+            'variable_type': 'SPIN',
+            'offset': 0.0,
+            'variable_order': ['a'],
+            'index_dtype': '<u2',
+            'bias_dtype': '<f4',
+            'quadratic_head': b'',
+            'quadratic_tail': b'',
+        }
+        self.assertDictEqual(encoded, expected_encoding)
+        decoded = bqm_bson_decoder(encoded)
+        self.assertEqual(bqm, decoded)
+
+    def test_small_bqm(self):
+        bqm = dimod.BinaryQuadraticModel.from_ising(
+            {"a": 1, "b": 3, "c": 4.5, "d": 0},
+            {"ab": -3, "cd": 3.5, "ad": 2}
+        )
+        encoded = bqm_bson_encoder(bqm)
+        decoded = bqm_bson_decoder(encoded)
+
+        # no easy way to directly check if the bqm objects are equal (b/c float
+        # precision, missing edges), so for now check if the qubo matrices are
+        # the same
+        var_order = sorted(bqm)
+        np.testing.assert_almost_equal(bqm.to_numpy_matrix(var_order),
+                                       decoded.to_numpy_matrix(var_order))
+
+    def test_complex_variable_names(self):
+        linear = {'a': -1, 4: 1, ('a', "complex key"): 3}
+        quadratic = {('a', 'c'): 1.2, ('b', 'c'): .3, ('a', 3): -1}
+        bqm = dimod.BinaryQuadraticModel(linear, quadratic, 3, dimod.SPIN,
+                                         tag=5)
+        encoded = bqm_bson_encoder(bqm)
+        decoded = bqm_bson_decoder(encoded)
+
+        # no easy way to directly check if the bqm objects are equal (b/c float
+        # precision, missing edges), so for now check if the qubo matrices are
+        # the same
+        var_order = list(bqm.variables)
+        np.testing.assert_almost_equal(bqm.to_numpy_matrix(var_order),
+                                       decoded.to_numpy_matrix(var_order),
+                                       decimal=6)
+
+    @unittest.skipUnless(_bson_imported, "no pymongo bson installed")
+    def test_bsonable(self):
+        bqm = dimod.BinaryQuadraticModel.from_ising(
+            {"a": 1, "b": 3, "c": 4.5, "d": 0},
+            {"ab": -3, "cd": 3.5, "ad": 2}
+        )
+        encoded = bqm_bson_encoder(bqm,
+                                   bytes_type=(bson.Binary if PY2 else bytes))
+        bson.BSON.encode(encoded)
+
+    def test_bias_dtype(self):
+        bqm = dimod.BinaryQuadraticModel.from_ising(
+            {"a": 1, "b": 3, "c": 4.5, "d": 0},
+            {"ab": -3, "cd": 3.5, "ad": 2},
+        )
+        encoded = bqm_bson_encoder(bqm, bias_dtype=np.float16)
+        expected_encoding = {
+            'as_complete': True,
+            'linear': b'\x00<\x00B\x80D\x00\x00',
+            'quadratic_vals': b'\x00\xc2\x00\x00\x00@\x00\x00\x00\x00\x00C',
+            'variable_type': 'SPIN',
+            'offset': 0.0,
+            'variable_order': ['a', 'b', 'c', 'd'],
+            'index_dtype': '<u2',
+            'bias_dtype': '<f2'
+        }
+        self.assertDictEqual(encoded, expected_encoding)
+        decoded = bqm_bson_decoder(encoded)
+        var_order = list(bqm.variables)
+        np.testing.assert_almost_equal(bqm.to_numpy_matrix(var_order),
+                                       decoded.to_numpy_matrix(var_order),
+                                       decimal=6)