Repository: stan-dev/cmdstanpy
Pull request #398 (2021-07-01 19:21:04)

Support json dumping for more collections
e.g. pandas or other numpy wrappers

See #397 

#### Submission Checklist

- [x] Run unit tests
- [x] Declare copyright holder and open-source license: see below

#### Summary

#### Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Brian Ward, Flatiron Institute



By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)



mitzimorris: hi @WardBrian, see wiki, "Developers" section for details on PyLint and flake8.
we don't reformat automatically - I know the Stan C++ repos do this - should we?

WardBrian: It might be a good idea to include a .flake8 file and use something like [pre-commit](https://pre-commit.com/) to enforce those. I believe currently it is failing based on files I haven't touched, so this suggests to me that the main branch of the repo is not in a correct state
WardBrian: I've checked any the errors generated during my tests are also generated if you run flake8 and pylint on a fresh clone of develop. Some of the code it is complaining about has been in the package for over 2 years. I have been trying to puzzle out why this means all PRs aren't failing, but I frankly have no idea.
codecov-commenter: # [Codecov](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev) Report
> Merging [#398](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev) (16279b7) into [develop](https://codecov.io/gh/stan-dev/cmdstanpy/commit/289c35f62b00e1f19e59c4ffc88da5e549909b9f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev) (289c35f) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398/graphs/tree.svg?width=650&height=150&src=pr&token=u5kGw6uPHT&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev)](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev)

```diff
@@           Coverage Diff            @@
##           develop     #398   +/-   ##
========================================
  Coverage    75.62%   75.63%           
========================================
  Files           27       27           
  Lines         7803     7805    +2     
========================================
+ Hits          5901     5903    +2     
  Misses        1902     1902           
```


| [Impacted Files](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev) | Coverage Δ | |
|---|---|---|
| [...tanpy/cmdstanpy/cmdstanpy/install\_cxx\_toolchain.py](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev#diff-YS9jbWRzdGFucHkvY21kc3RhbnB5L2NtZHN0YW5weS9pbnN0YWxsX2N4eF90b29sY2hhaW4ucHk=) | `22.61% <0.00%> (-0.39%)` | :arrow_down: |
| [a/cmdstanpy/cmdstanpy/cmdstanpy/utils.py](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev#diff-YS9jbWRzdGFucHkvY21kc3RhbnB5L2NtZHN0YW5weS91dGlscy5weQ==) | `76.68% <0.00%> (+0.03%)` | :arrow_up: |
| [cmdstanpy/cmdstanpy/utils.py](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev#diff-Y21kc3RhbnB5L2NtZHN0YW5weS91dGlscy5weQ==) | `72.99% <0.00%> (+0.04%)` | :arrow_up: |
| [...runner/work/cmdstanpy/cmdstanpy/cmdstanpy/utils.py](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev#diff-VXNlcnMvcnVubmVyL3dvcmsvY21kc3RhbnB5L2NtZHN0YW5weS9jbWRzdGFucHkvdXRpbHMucHk=) | `72.99% <0.00%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev). Last update [289c35f...16279b7](https://codecov.io/gh/stan-dev/cmdstanpy/pull/398?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stan-dev).


Patch:
diff --git a/cmdstanpy/utils.py b/cmdstanpy/utils.py
index bc19594..d70493a 100644
--- a/cmdstanpy/utils.py
+++ b/cmdstanpy/utils.py
@@ -11,7 +11,7 @@ import subprocess
 import sys
 import tempfile
 from collections import OrderedDict
-from collections.abc import Sequence
+from collections.abc import Sequence, Collection
 from numbers import Integral, Real
 from typing import Dict, List, TextIO, Tuple, Union
 
@@ -371,8 +371,10 @@ def jsondump(path: str, data: Dict) -> None:
     data = data.copy()
     for key, val in data.items():
         if type(val).__module__ == 'numpy':
-            val = val.tolist()
-            data[key] = val
+            data[key] = val.tolist()
+        elif isinstance(val, Collection):
+            data[key] = np.asarray(val).tolist()
+
     with open(path, 'w') as fd:
         json.dump(data, fd)
 


Test patch:
diff --git a/test/test_utils.py b/test/test_utils.py
index 0c5ae98..4becfb8 100644
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -9,8 +9,10 @@ import stat
 import string
 import tempfile
 import unittest
+import collections.abc
 
 import numpy as np
+import pandas as pd
 
 from cmdstanpy import _DOT_CMDSTAN, _DOT_CMDSTANPY, _TMPDIR
 from cmdstanpy.model import CmdStanModel
@@ -212,8 +214,9 @@ class DataFilesTest(unittest.TestCase):
             for k in d1:
                 data_1 = d1[k]
                 data_2 = d2[k]
-                if isinstance(data_2, np.ndarray):
-                    data_2 = data_2.tolist()
+                if isinstance(data_2, collections.abc.Collection):
+                    data_2 = np.asarray(data_2).tolist()
+
                 self.assertEqual(data_1, data_2)
 
         dict_list = {'a': [1.0, 2.0, 3.0]}
@@ -222,12 +225,20 @@ class DataFilesTest(unittest.TestCase):
         with open(file_list) as fd:
             cmp(json.load(fd), dict_list)
 
-        dict_vec = {'a': np.repeat(3, 4)}
+        arr = np.repeat(3, 4)
+        dict_vec = {'a': arr}
         file_vec = os.path.join(_TMPDIR, 'vec.json')
         jsondump(file_vec, dict_vec)
         with open(file_vec) as fd:
             cmp(json.load(fd), dict_vec)
 
+        series = pd.Series(arr)
+        dict_vec_pd = {'a': series}
+        file_vec_pd = os.path.join(_TMPDIR, 'vec_pd.json')
+        jsondump(file_vec_pd, dict_vec_pd)
+        with open(file_vec_pd) as fd:
+            cmp(json.load(fd), dict_vec_pd)
+
         dict_zero_vec = {'a': []}
         file_zero_vec = os.path.join(_TMPDIR, 'empty_vec.json')
         jsondump(file_zero_vec, dict_zero_vec)