Repository: lmfit/lmfit-py

Issue:
ModelResult.dumps() broken for lbfgsb method
#### Description
When I use the limited BFGS (lbfgsb) method to fit, I cannot use dumps().
It works fine when I do not specify a method, or if i specify the "bfgs" method, but the 'lbfgsb' method fails.


###### A Minimal, Complete, and Verifiable example
```Python
import matplotlib.pyplot as plt
from numpy import exp, loadtxt, pi, sqrt

from lmfit import Model

data = loadtxt('model1d_gauss.dat')
x = data[:, 0]
y = data[:, 1]

def gaussian(x, amp, cen, wid):
    """1-d gaussian: gaussian(x, amp, cen, wid)"""
    return (amp / (sqrt(2*pi) * wid)) * exp(-(x-cen)**2 / (2*wid**2))

gmodel = Model(gaussian)
result = gmodel.fit(y, x=x, amp=5, cen=5, wid=1, method='lbfgsb')

print(result.fit_report())

s = result.dumps()
```

###### Error message:

```
TypeError                                 Traceback (most recent call last)
<ipython-input-9-b8a48d1fdbd7> in <module>
     17 result = gmodel.fit(y, x=x, amp=5, cen=5, wid=1, method='lbfgsb')
     18 
---> 19 s = result.dumps()

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lmfit/model.py in dumps(self, **kws)
   1629                 val = bool(val)
   1630             out[attr] = encode4js(val)
-> 1631         return json.dumps(out, **kws)
   1632 
   1633     def dump(self, fp, **kws):

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    229         cls is None and indent is None and separators is None and
    230         default is None and not sort_keys and not kw):
--> 231         return _default_encoder.encode(obj)
    232     if cls is None:
    233         cls = JSONEncoder

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py in encode(self, o)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py in iterencode(self, o, _one_shot)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
    258 
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py in default(self, o)
    177 
    178         """
--> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
    180                         f'is not JSON serializable')
    181 

TypeError: Object of type bytes is not JSON serializable
```

###### Version information
```
Python: 3.8.6 (default, Oct  8 2020, 14:07:53) 
[Clang 11.0.0 (clang-1100.0.33.17)]

lmfit: 1.0.1, scipy: 1.5.2, numpy: 1.19.2, asteval: 0.9.19, uncertainties: 3.1.4
```


ModelResult.dumps() broken for lbfgsb method
#### Description
When I use the limited BFGS (lbfgsb) method to fit, I cannot use dumps().
It works fine when I do not specify a method, or if i specify the "bfgs" method, but the 'lbfgsb' method fails.


###### A Minimal, Complete, and Verifiable example
```Python
import matplotlib.pyplot as plt
from numpy import exp, loadtxt, pi, sqrt

from lmfit import Model

data = loadtxt('model1d_gauss.dat')
x = data[:, 0]
y = data[:, 1]

def gaussian(x, amp, cen, wid):
    """1-d gaussian: gaussian(x, amp, cen, wid)"""
    return (amp / (sqrt(2*pi) * wid)) * exp(-(x-cen)**2 / (2*wid**2))

gmodel = Model(gaussian)
result = gmodel.fit(y, x=x, amp=5, cen=5, wid=1, method='lbfgsb')

print(result.fit_report())

s = result.dumps()
```

###### Error message:

```
TypeError                                 Traceback (most recent call last)
<ipython-input-9-b8a48d1fdbd7> in <module>
     17 result = gmodel.fit(y, x=x, amp=5, cen=5, wid=1, method='lbfgsb')
     18 
---> 19 s = result.dumps()

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lmfit/model.py in dumps(self, **kws)
   1629                 val = bool(val)
   1630             out[attr] = encode4js(val)
-> 1631         return json.dumps(out, **kws)
   1632 
   1633     def dump(self, fp, **kws):

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    229         cls is None and indent is None and separators is None and
    230         default is None and not sort_keys and not kw):
--> 231         return _default_encoder.encode(obj)
    232     if cls is None:
    233         cls = JSONEncoder

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py in encode(self, o)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py in iterencode(self, o, _one_shot)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
    258 
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py in default(self, o)
    177 
    178         """
--> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
    180                         f'is not JSON serializable')
    181 

TypeError: Object of type bytes is not JSON serializable
```

###### Version information
```
Python: 3.8.6 (default, Oct  8 2020, 14:07:53) 
[Clang 11.0.0 (clang-1100.0.33.17)]

lmfit: 1.0.1, scipy: 1.5.2, numpy: 1.19.2, asteval: 0.9.19, uncertainties: 3.1.4
```




Discussion:
@leonfoks Hm, yeah I can reproduce that - sorry for the trouble.  

Ugh -- not your fault at all, but why (the heck) would a scipy.optimize method return a "bytes" object as a message from a fitting method?  That seems either a misunderstanding of what bytes and strings are for.  I will complain over at scipy.  I don't expect they'll pay attention.

In the meantime, I guess we need a workaround so that we can serialize their idiotic use of bytes for a string.  Of course, that means either saving the bytes themselves or asserting that we know the encoding.  Well, we *do* know the encoding because they (scipy) would have known the encoding because the data is not actually bytes, it's a string.   Just allow me to state that clearly:  this is completely an error in scipy.optimize, and we have to work around their mess.

A PR would be most welcome - just assuming that bytes are in utf-8 in `encode4js` - and don't bother to decode to bytes: the message from scipy.optimize is a string and should not be bytes.  I think we should also add a test when moving attributes from scipy.OptimizerResult to lmfit.MinimizerResult that `message` is not a `bytes` just because scipy was too lazy to properly convert C strings to Python.   Sheesh.



 
No worries!  Thanks for pointing me in the right direction.  I was not sure where to start looking.

I have a fix, will submit a PR soon.
@leonfoks Hm, yeah I can reproduce that - sorry for the trouble.  

Ugh -- not your fault at all, but why (the heck) would a scipy.optimize method return a "bytes" object as a message from a fitting method?  That seems either a misunderstanding of what bytes and strings are for.  I will complain over at scipy.  I don't expect they'll pay attention.

In the meantime, I guess we need a workaround so that we can serialize their idiotic use of bytes for a string.  Of course, that means either saving the bytes themselves or asserting that we know the encoding.  Well, we *do* know the encoding because they (scipy) would have known the encoding because the data is not actually bytes, it's a string.   Just allow me to state that clearly:  this is completely an error in scipy.optimize, and we have to work around their mess.

A PR would be most welcome - just assuming that bytes are in utf-8 in `encode4js` - and don't bother to decode to bytes: the message from scipy.optimize is a string and should not be bytes.  I think we should also add a test when moving attributes from scipy.OptimizerResult to lmfit.MinimizerResult that `message` is not a `bytes` just because scipy was too lazy to properly convert C strings to Python.   Sheesh.



 
No worries!  Thanks for pointing me in the right direction.  I was not sure where to start looking.

I have a fix, will submit a PR soon.

Resolving patch:
diff --git a/lmfit/model.py b/lmfit/model.py
--- a/lmfit/model.py
+++ b/lmfit/model.py
@@ -1624,7 +1624,13 @@ def dumps(self, **kws):
                 continue
             if isinstance(val, np.bool_):
                 val = bool(val)
+
             out[attr] = encode4js(val)
+
+        val = out.get('message', '')
+        if isinstance(val, bytes):
+            out['message'] = str(val, encoding='ASCII')
+
         return json.dumps(out, **kws)
 
     def dump(self, fp, **kws):


Test patch:
diff --git a/tests/test_saveload.py b/tests/test_saveload.py
--- a/tests/test_saveload.py
+++ b/tests/test_saveload.py
@@ -186,7 +186,7 @@ def test_saveload_modelresult_exception():
 
 
 @pytest.mark.parametrize("method", ['leastsq', 'nelder', 'powell', 'cobyla',
-                                    'bfgsb', 'differential_evolution', 'brute',
+                                    'bfgs', 'lbfgsb', 'differential_evolution', 'brute',
                                     'basinhopping', 'ampgo', 'shgo',
                                     'dual_annealing'])
 def test_saveload_modelresult_roundtrip(method):