Repository: statsmodels/statsmodels

Issue:
Can't get attribute 'ResettableCache' on module 'statsmodels.tools.decorators'
#### Describe the bug
Trying to load a model saved with pickle module, and the previous version of statmodels (0.9.0), I got this error...

```AttributeError: Can't get attribute 'ResettableCache' on <module 'statsmodels.tools.decorators' from '/usr/local/lib/python3.6/dist-packages/statsmodels/tools/decorators.py'>```


```python
import pickle
model = pickle.load(open('data/modelA.model','rb'))
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A statsmodels model was loaded in the variable ```model```. 

#### Output of ``import statsmodels.api as sm; sm.show_versions()``
Okay, this is weird...

<details>
```import statsmodels.api as sm; sm.show_versions()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-411ef5239fad> in <module>()
----> 1 import statsmodels.api as sm; sm.show_versions()

/usr/local/lib/python3.6/dist-packages/statsmodels/api.py in <module>()
      7 from . import regression
      8 from .regression.linear_model import OLS, GLS, WLS, GLSAR
----> 9 from .regression.recursive_ls import RecursiveLS
     10 from .regression.quantile_regression import QuantReg
     11 from .regression.mixed_linear_model import MixedLM

/usr/local/lib/python3.6/dist-packages/statsmodels/regression/recursive_ls.py in <module>()
     12 from statsmodels.compat import unicode
     13 from statsmodels.tools.data import _is_using_pandas
---> 14 from statsmodels.tsa.statespace.mlemodel import (
     15     MLEModel, MLEResults, MLEResultsWrapper, PredictionResults,
     16     PredictionResultsWrapper)

/usr/local/lib/python3.6/dist-packages/statsmodels/tsa/statespace/mlemodel.py in <module>()
     29 import statsmodels.tsa.base.tsa_model as tsbase
     30 
---> 31 from .simulation_smoother import SimulationSmoother
     32 from .kalman_smoother import SmootherResults
     33 from .kalman_filter import INVERT_UNIVARIATE, SOLVE_LU

/usr/local/lib/python3.6/dist-packages/statsmodels/tsa/statespace/simulation_smoother.py in <module>()
      8 
      9 import numpy as np
---> 10 from .kalman_smoother import KalmanSmoother
     11 from . import tools
     12 

/usr/local/lib/python3.6/dist-packages/statsmodels/tsa/statespace/kalman_smoother.py in <module>()
      9 import numpy as np
     10 
---> 11 from statsmodels.tsa.statespace.representation import OptionWrapper
     12 from statsmodels.tsa.statespace.kalman_filter import (KalmanFilter,
     13                                                       FilterResults)

/usr/local/lib/python3.6/dist-packages/statsmodels/tsa/statespace/representation.py in <module>()
      8 
      9 import numpy as np
---> 10 from .tools import (
     11     find_best_blas_type, validate_matrix_shape, validate_vector_shape
     12 )

/usr/local/lib/python3.6/dist-packages/statsmodels/tsa/statespace/tools.py in <module>()
     14 from statsmodels.tools.data import _is_using_pandas
     15 from scipy.linalg.blas import find_best_blas_type
---> 16 from . import (_initialization, _representation, _kalman_filter,
     17                _kalman_smoother, _simulation_smoother, _tools)
     18 

__init__.pxd in init statsmodels.tsa.statespace._initialization()

ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject```

</details>



Discussion:
Try setting “sm.tools.decorators.ResettableCache = dict=“ before trying to unpickle.  Does that solve the problem?
Doesn't work.
sm.tools.decorators.ResettableCache = dict=
 File "<ipython-input-5-1cbd5064d685>", line 1
   sm.tools.decorators.ResettableCache = dict=
                                              ^
SyntaxError: invalid syntax

This doesn't work either.
In [6]: sm.tools.decorators.ResettableCache = dict

In [7]: model = pickle.load(open('data/modelA.model','rb'))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-f4a0ddf78667> in <module>()
----> 1 model = pickle.load(open('data/modelA.model','rb'))

AttributeError: 'dict' object has no attribute '__dict__'

On Sun, Jun 30, 2019 at 9:49 PM jbrockmendel <notifications@github.com>
wrote:

> Try setting “sm.tools.decorators.ResettableCache = dict=“ before trying to
> unpickle. Does that solve the problem?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/5899?email_source=notifications&email_token=AACXSPB7OPT25QFQFO4L4X3P5FICJA5CNFSM4H4M424KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY4XLJY#issuecomment-507082151>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACXSPHZPXTJM7W6F6J4DNTP5FICJANCNFSM4H4M424A>
> .
>


-- 
Joaquín Bogado
Equipo de Soporte y Desarrollo
Lihuen -- GNU/Linux
Lethe -- Linux Partition Freezing Software
EELA-UNLP -- Grid Site Admin

Typo (was on a phone).  That should be `sm.tools.decorators.ResettableCache = dict`
I suspected that, but the corrected version also doesn't work, as I send
you in the previous email.
Any other suggestions? I will have no problems to try. Workaround is to
downgrade to 0.9.0.

On Mon, Jul 1, 2019 at 5:28 PM jbrockmendel <notifications@github.com>
wrote:

> Typo (was on a phone). That should be sm.tools.decorators.ResettableCache
> = dict
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/5899?email_source=notifications&email_token=AACXSPBRABSA3KKLBAWTZE3P5JSGZA5CNFSM4H4M424KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7IF6I#issuecomment-507413241>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACXSPAA4FBQD7WKTW3DLNTP5JSGZANCNFSM4H4M424A>
> .
>


-- 
Joaquín Bogado
Equipo de Soporte y Desarrollo
Lihuen -- GNU/Linux
Lethe -- Linux Partition Freezing Software
EELA-UNLP -- Grid Site Admin

OK, let's troubleshoot.

Try writing in that module:

```
class ResettableCache(dict):
     def __init__(self, *args, **kwargs):
         super(ResettableCache, self).__init__(*args, **kwargs)
         self.__dict__ = self
```

Is this something you need to do once for a particular model, or something you'll anticipate doing many times?
> Is this something you need to do once for a particular model, or something you'll anticipate doing many times?

If its a just-once thing, then one workaround may be to downgrade to 0.9.0, load it there, then replace `model._cache = dict(model._cache)`, then re-pickle and load in 0.10.0
hahahaha, I have 512 different models...
I saw the class ResetttableCache was removed in the newer version of
statsmodels. If it's something you spect, at least statsmodels should warn
about you will not be able to load your models if saved with a previous
version of the library.
You want me to edit the sm.tools.decorators.py file?

On Mon, Jul 1, 2019 at 5:41 PM jbrockmendel <notifications@github.com>
wrote:

> OK, let's troubleshoot.
>
> Try writing in that module:
>
> class ResettableCache(dict):
>      def __init__(self, *args, **kwargs):
>          super(ResettableCache, self).__init__(*args, **kwargs)
>          self.__dict__ = self
>
> Is this something you need to do once for a particular model, or something
> you'll anticipate doing many times?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/5899?email_source=notifications&email_token=AACXSPHNBZHUNM4BI7ARV43P5JTYRA5CNFSM4H4M424KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7JJWY#issuecomment-507417819>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACXSPDPVJ73JM4L4C7ZRQ3P5JTYRANCNFSM4H4M424A>
> .
>


-- 
Joaquín Bogado
Equipo de Soporte y Desarrollo
Lihuen -- GNU/Linux
Lethe -- Linux Partition Freezing Software
EELA-UNLP -- Grid Site Admin

I labeled this a bug but I'm not 100% sure it is.  AFAIK there is no official support for saving models across version, and I suspect that this case isn't worth the extra effort and constraints now. Pickling is necessary for some contexts, e.g., using multiprocess, and so much be supported, but these all use the same version of all libraries. 
Maybe my "just do what pandas does" bias is showing, but I think going forward trying to avoid this problem is worthwhile.
FWIW I had the same problems with pickled results from a Model. My workaround was fitting the model in `0.10.0`. Alternatively I could have used @jbrockmendel  approach as suggested above. Would indeed be great if this kind of problem could be avoided going fwd. 
Perhaps a compromise would be to ensure that a model pickled with 0.n.x should be unpicklable with 0.n+1.x, but not guarantee about 0.n+m.x for m>1. Constraining pickling support will eliminate a lot fo code clean up, and sm isn't at that point.

@jbrockmendel Any chance you want to do a PR that will allow these models to be loaded in 0.10.1?

For me this was only a very minor inconvenience and I am all for simple code so I fully understand which ever way you guys decide to go. BTW thanks very much for the fantastic work!
Yah, if one of the commenters here doesn't beat me to it (hint hint) I'll take a look at what it will take to fix this in 0.10.1

Resolving patch:
diff --git a/statsmodels/tools/decorators.py b/statsmodels/tools/decorators.py
--- a/statsmodels/tools/decorators.py
+++ b/statsmodels/tools/decorators.py
@@ -2,7 +2,14 @@
 from statsmodels.tools.sm_exceptions import CacheWriteWarning
 import warnings
 
-__all__ = ['cache_readonly', 'cache_writable', 'deprecated_alias']
+__all__ = ['cache_readonly', 'cache_writable', 'deprecated_alias',
+           'ResettableCache']
+
+
+class ResettableCache(dict):
+    def __init__(self, *args, **kwargs):
+        super(ResettableCache, self).__init__(*args, **kwargs)
+        self.__dict__ = self
 
 
 def deprecated_alias(old_name, new_name, remove_version=None, msg=None,


Test patch: