Repository: radiasoft/pykern

Issue:
pkjson support standard better on numbers
Sirepo uses JSON to store numbers. There is a problem value `"verticalFocalLength": 100000000000000000000` in an SRW lens, which is 1e20, which is larger than 1e16. This note explains the problem. The solution needs to be overriding how the json decoder works.

Python supports `int` objects with arbitrary magnitude. This is incompatible with JSON, which has a vague definition of [Numbers](https://datatracker.ietf.org/doc/html/rfc8259#section-6): 

> Since software that implements IEEE 754 binary64 (double precision) numbers [IEEE754](https://datatracker.ietf.org/doc/html/rfc8259#ref-IEEE754) is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these  provide, in the sense that implementations will approximate JSON numbers within the expected precision.

I think we should assume JSON, like Javascript, does not distinguish between integers and floats. All [Numbers](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#number_type) are 64 bit IEEE 754 values. Number defines [MAX_SAFE_INTEGER](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER) and [MIN_SAFE_INTEGER](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MIN_SAFE_INTEGER), which are 53 bits or just shy of 16 decimal digits. 

Javascript encodes 1e20 (problematic number) in JSON as:

```sh
> JSON.stringify(1e20)
'100000000000000000000'
```

Python's JSON interpreter uses Python parsing rules instead of JSON standard parsing rules. It interprets a number without a `.` as an `int`, which JSON has no concept of. Instead, once parse_int converts the number, it should a float if the number is larger than MAX_SAFE_INTEGER or smaller than MIN_SAFE_INTEGER.


Discussion:

Patch:
diff --git a/pykern/pkcollections.py b/pykern/pkcollections.py
index 88ca5dd..1f6c4b7 100644
--- a/pykern/pkcollections.py
+++ b/pykern/pkcollections.py
@@ -329,6 +329,10 @@ Dict = PKDict
 DictNameError = PKDictNameError
 
 
+_JSON_INT_MAX = 2**53 - 1
+_JSON_INT_MIN = -(2**53) + 1
+
+
 def json_load_any(obj, *args, **kwargs):
     """Parse json with `PKDict` for object pairs
 
@@ -341,6 +345,19 @@ def json_load_any(obj, *args, **kwargs):
         object: parsed JSON
     """
 
+    def _parse_int(value):
+        if len(value) > 64:
+            # 64 is pretty arbitrary, but reasonable in Python. Nothing in
+            # JSON can be this large.
+            raise ValueError(f"number={value:60s}... unreasonably large")
+        if len(value) > 17:
+            return float(value)
+        res = int(value)
+        if _JSON_INT_MIN <= res <= _JSON_INT_MAX:
+            return res
+        return float(res)
+
+    kwargs.setdefault("parse_int", _parse_int)
     kwargs.setdefault("object_pairs_hook", object_pairs_hook)
     return json.loads(obj.read() if hasattr(obj, "read") else obj, *args, **kwargs)
 


Test patch:
diff --git a/tests/pkjson_test.py b/tests/pkjson_test.py
index 3c5fd46..71ec670 100644
--- a/tests/pkjson_test.py
+++ b/tests/pkjson_test.py
@@ -4,8 +4,6 @@
 :copyright: Copyright (c) 2017 RadiaSoft LLC.  All Rights Reserved.
 :license: http://www.apache.org/licenses/LICENSE-2.0.html
 """
-from __future__ import absolute_import, division, print_function
-import pytest
 
 
 def test_load_any():
@@ -18,6 +16,24 @@ def test_load_any():
     pkeq("a", j2[0])
 
 
+def test_load_any_parse_int():
+    from pykern import pkjson
+    from pykern.pkunit import pkeq, pkexcept
+
+    for c, e in (
+        ("9007199254740992", float(2**53)),
+        ("9007199254740991", int(2**53 - 1)),
+        ("-9007199254740991", int(-(2**53) + 1)),
+        ("-9007199254740992", float(-(2**53))),
+        ("1" * 64, 1.1111111111111112e63),
+    ):
+        a = pkjson.load_any(c)
+        pkeq(type(e), type(a))
+        pkeq(e, a)
+    with pkexcept("unreasonably large"):
+        pkjson.load_any("1" * 65)
+
+
 def test_dump_bytes():
     import json
     from pykern import pkjson, pkcompat