Repository: TurboGears/Ming

Issue:
Support for Decimal128 type in Ming
MongoDB 3.4 introduced support for the Decimal type (https://docs.mongodb.com/manual/tutorial/model-monetary-data/#numeric-decimal). Pymongo supports it from v3.7, I think Ming should be able to use it as well. 
The main issue I see with this is the `ming.base._safe_bson` function, that explicitly casts `decimal.Decimal` values to `float`. Refactoring it would probably be a breaking change for some application relying on that cast to ensure consistency in the value type of that field.
@amol- any suggestions on how to safely implement this, considering also retrocompatibility with MongoDB < 3.4?

Discussion:
coveralls: ## Pull Request Test Coverage Report for [Build 39](https://coveralls.io/builds/21597751)

* **63** of **64**   **(98.44%)**  changed or added relevant lines in **3** files are covered.
* **12** unchanged lines in **4** files lost coverage.
* Overall coverage decreased (**-0.1%**) to **88.244%**

---

|  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [ming/schema.py](https://coveralls.io/builds/21597751/source?filename=ming%2Fschema.py#L745) | 18 | 19 | 94.74%
<!-- | **Total:** | **63** | **64** | **98.44%** | -->

|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [ming/mim.py](https://coveralls.io/builds/21597751/source?filename=ming%2Fmim.py#L1364) | 1 | 76.52% |
| [ming/odm/mapper.py](https://coveralls.io/builds/21597751/source?filename=ming%2Fodm%2Fmapper.py#L225) | 1 | 89.17% |
| [ming/tests/test_base.py](https://coveralls.io/builds/21597751/source?filename=ming%2Ftests%2Ftest_base.py#L49) | 1 | 95.45% |
| [ming/odm/icollection.py](https://coveralls.io/builds/21597751/source?filename=ming%2Fodm%2Ficollection.py#L167) | 9 | 84.44% |
<!-- | **Total:** | **12** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/21597751/badge)](https://coveralls.io/builds/21597751) |
| :-- | --: |
| Change from base [Build 38](https://coveralls.io/builds/21540625): |  -0.1% |
| Covered Lines: | 5690 |
| Relevant Lines: | 6448 |

---
##### 💛  - [Coveralls](https://coveralls.io)

amol-: I'll make a `0.8.1` release with current master that includes the fixes to connection management and then a `0.9.0` that includes this changes which break backward compatibility in `make_safe`
devilicecream: > I'll make a `0.8.1` release with current master that includes the fixes to connection management and then a `0.9.0` that includes this changes which break backward compatibility in `make_safe`

I actually just added a deprecation warning on `make_safe`, so that backwards compatibility wouldn't be broken with this release and we can figure out when to actually remove it based on users feedback. Does that make sense?

Patch:
diff --git a/.travis.yml b/.travis.yml
index 401150e..8b15975 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -6,7 +6,7 @@ python:
   - "3.5"
   - "3.6"
   - "3.7-dev"
-  - "pypy"
+  - "pypy2.7-6.0"
 
 install:
   - "travis_retry pip install --upgrade pip"
diff --git a/docs/news.rst b/docs/news.rst
index ead9713..b0bd3cd 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -3,6 +3,11 @@ Ming News / Release Notes
 
 The latest releases are 0.7.x and support PyMongo 3.  The 0.5.x releases support PyMongo 2.  Both work on Python 2 and 3.
 
+0.8.1 (Feb 22, 2019)
+--------------------
+
+* Fix for connection string when seedlist is used
+
 0.8.0 (Jan 15, 2019)
 --------------------
 
diff --git a/ming/base.py b/ming/base.py
index 18e5287..d2a9264 100644
--- a/ming/base.py
+++ b/ming/base.py
@@ -1,11 +1,13 @@
 """Ming Base module.  Good stuff here.
 """
 import decimal
+import warnings
 from collections import defaultdict
 from datetime import datetime
 
 import bson
 import six
+from bson import Decimal128
 
 from ming.exc import MingException
 
@@ -52,6 +54,9 @@ class Object(dict):
             return bson
 
     def make_safe(self):
+        warnings.warn("make_safe is now deprecated. "
+                      "If your code relies on make_safe for validation, "
+                      "you should consider adding your own layer of validation", DeprecationWarning)
         safe_self = _safe_bson(self)
         self.update(safe_self)
 
@@ -131,20 +136,24 @@ class Cursor(object):
         return self
 
 NoneType = type(None)
-def _safe_bson(obj):
+def _safe_bson(obj, _no_warning=False):
     '''Verify that the obj is safe for bsonification (in particular, no tuples or
     Decimal objects
     '''
+    if not _no_warning:
+        warnings.warn("_safe_bson is now deprecated. "
+                      "If your code relies on _safe_bson for validation, "
+                      "you should consider adding your own layer of validation", DeprecationWarning)
     if isinstance(obj, list):
-        return [ _safe_bson(o) for o in obj ]
+        return [ _safe_bson(o, True) for o in obj ]
     elif isinstance(obj, dict):
-        return Object((k, _safe_bson(v)) for k,v in six.iteritems(obj))
+        return Object((k, _safe_bson(v, True)) for k,v in six.iteritems(obj))
     elif isinstance(obj, six.string_types + six.integer_types + (
             float, datetime, NoneType,
             bson.ObjectId)):
         return obj
     elif isinstance(obj, decimal.Decimal):
-        return float(obj)
+        return Decimal128(obj)
     else:
         assert False, '%s is not safe for bsonification: %r' % (
             type(obj), obj)
diff --git a/ming/schema.py b/ming/schema.py
index b0469c1..c62e7e3 100644
--- a/ming/schema.py
+++ b/ming/schema.py
@@ -3,11 +3,13 @@ import logging
 
 from copy import deepcopy
 from datetime import datetime, date
+from decimal import Decimal, ROUND_HALF_DOWN, Context
 
 import bson
 import pymongo
 import pytz
 import six
+from bson import Decimal128
 
 from .utils import LazyProperty
 from .base import Object as BaseObject, Missing, NoDefault
@@ -708,10 +710,53 @@ class ObjectId(Scalar):
         except Exception as ex:
             raise Invalid(str(ex), value, None)
 
+
+class NumberDecimal(ParticularScalar):
+    """Validates value is compatible with :class:`bson.Decimal128`.
+
+    Useful for financial data that need arbitrary precision.
+
+    The argument `precision` specifies the number of decimal digits to evaluate.
+    Defaults to 6.
+
+    The argument `rounding` specifies the kind of rounding to be performed.
+    Valid values are `decimal.ROUND_DOWN`, `decimal.ROUND_HALF_UP`, `decimal.ROUND_HALF_EVEN`,
+    `decimal.ROUND_CEILING`, `decimal.ROUND_FLOOR`, `decimal.ROUND_UP`,
+    `decimal.ROUND_HALF_DOWN`, `decimal.ROUND_05UP`.
+    Defaults to `decimal.ROUND_HALF_DOWN`.
+
+    """
+    type = (int, float, Decimal, Decimal128)
+
+    def __init__(
+        self,
+        precision=None,
+        rounding=ROUND_HALF_DOWN,
+        **kwargs
+    ):
+        super(NumberDecimal, self).__init__(**kwargs)
+        self.precision = precision
+        self.rounding = rounding
+        self._context = Context(prec=34, rounding=rounding)  # Max Decimal128 precision
+        if self.precision:
+            self._quantizing = self._context.create_decimal(Decimal(str(10 ** -precision)))
+
+    def _validate(self, value, **kw):
+        value = super(NumberDecimal, self)._validate(value, **kw)
+        if isinstance(value, Decimal128):
+            value = value.to_decimal()
+        value = self._context.create_decimal(value)
+        if self.precision:
+            value = value.quantize(self._quantizing, rounding=self.rounding)
+        return Decimal128(value)
+
+
 # Shorthand for various SchemaItems
-SHORTHAND={
-    int:Int,
-    str:String,
-    float:Float,
-    bool:Bool,
-    datetime:DateTime}
+SHORTHAND = {
+    int: Int,
+    str: String,
+    float: Float,
+    bool: Bool,
+    datetime: DateTime,
+    Decimal128: NumberDecimal,
+}
diff --git a/ming/session.py b/ming/session.py
index 3d5d74d..cc2215b 100644
--- a/ming/session.py
+++ b/ming/session.py
@@ -144,7 +144,6 @@ class Session(object):
         hook = doc.m.before_save
         if hook: hook(doc)
         if validate:
-            doc.make_safe()
             if doc.m.schema is None:
                 data = dict(doc)
             else:
@@ -201,7 +200,6 @@ class Session(object):
         immediately
         """
         fields_values = Object.from_bson(fields_values)
-        fields_values.make_safe()
         for k,v in six.iteritems(fields_values):
             self._set(doc, k.split('.'), v)
         impl = self._impl(doc)
diff --git a/ming/version.py b/ming/version.py
index d939c5f..ad18e51 100644
--- a/ming/version.py
+++ b/ming/version.py
@@ -1,2 +1,2 @@
-__version_info__ = ('0', '8', '0')
+__version_info__ = ('0', '8', '1')
 __version__ = '.'.join(__version_info__)


Test patch:
diff --git a/ming/tests/test_schema.py b/ming/tests/test_schema.py
index c56f5ee..0702224 100644
--- a/ming/tests/test_schema.py
+++ b/ming/tests/test_schema.py
@@ -1,6 +1,9 @@
+from decimal import Decimal, ROUND_DOWN, ROUND_HALF_DOWN
 from unittest import TestCase, main
 from datetime import datetime, date
 
+from bson import Decimal128
+
 import ming.datastore
 import pytz
 from ming import Document, Field
@@ -155,5 +158,68 @@ class TestSchemaItem(TestCase):
     def test_nodefault(self):
         self.assertEqual(repr(S.NoDefault), '<NoDefault>')
 
+
+class TestNumberDecimal(TestCase):
+    def test_dont_allow_none(self):
+        si = S.NumberDecimal(allow_none=False)
+        self.assertRaises(S.Invalid, si.validate, None)
+
+    def test_allow_none(self):
+        si = S.NumberDecimal()
+        assert si.validate(None) is None
+
+    def test_specify_precision(self):
+        si = S.NumberDecimal(precision=2)
+        assert si.validate(0.123456789) == Decimal128("0.12")
+        assert si.validate(0.125432109) == Decimal128("0.13")
+
+    def test_specify_rounding(self):
+        si = S.NumberDecimal(precision=2, rounding=ROUND_DOWN)
+        assert si.validate(0.129999999) == Decimal128("0.12")
+        assert si.validate(0.123456789) == Decimal128("0.12")
+
+    def test_input_int_precision_explicit(self):
+        si = S.NumberDecimal(precision=2)
+        assert si.validate(0) == Decimal128("0.00")
+        assert si.validate(1) == Decimal128("1.00")
+
+    def test_input_float_precision_explicit(self):
+        si = S.NumberDecimal(precision=2)
+        assert si.validate(12.42) == Decimal128("12.42")
+
+    def test_input_decimal_precision_explicit(self):
+        si = S.NumberDecimal(precision=2)
+        assert si.validate(Decimal("12.42")) == Decimal128("12.42")
+
+    def test_input_decimal128_precision_explicit(self):
+        si = S.NumberDecimal(precision=2)
+        assert si.validate(Decimal128("12.42")) == Decimal128("12.42")
+
+    def test_input_int(self):
+        si = S.NumberDecimal()
+        assert si.validate(0) == Decimal128("0")
+        assert si.validate(1) == Decimal128("1")
+
+    def test_input_float(self):
+        si = S.NumberDecimal()
+        validated = si.validate(12.42)
+        assert isinstance(validated, Decimal128)
+        quantized = validated.to_decimal().quantize(Decimal("0.01"),
+                                                    rounding=ROUND_HALF_DOWN)
+        assert quantized == Decimal("12.42")
+
+    def test_input_decimal(self):
+        si = S.NumberDecimal()
+        assert si.validate(Decimal("12.42")) == Decimal128("12.42")
+
+    def test_input_decimal128(self):
+        si = S.NumberDecimal()
+        assert si.validate(Decimal128("12.42")) == Decimal128("12.42")
+
+    def test_input_str(self):
+        si = S.NumberDecimal(precision=2)
+        self.assertRaises(S.Invalid, si.validate, "12.42")
+
+
 if __name__ == '__main__':
     main()