Repository: statsmodels/statsmodels

Issue:
Caching behavior for KDEUnivariate icdf 
I've noticed some strange behaviour with the caching in the KDEUnivariate class. The problem can be reproduced with the following code:

```
import statsmodels.nonparametric.api as smkern
import numpy as np

data1 = np.linspace(0,1,100)
pdf = smkern.KDEUnivariate(data1)
pdf.fit()

data2 = np.linspace(1,2,100)
pdf2 = smkern.KDEUnivariate(data2)
pdf2.fit()

print pdf.icdf[0]
print pdf2.icdf[0]
```

which outputs

```
0.0
0.0
```

While if instead the last two lines are:

```
print pdf2.icdf[0]
print pdf.icdf[0]
```

then it outputs

```
1.0
1.0
```

It seems as though the cache is persistent across different KDEUnivariate objects. Perhaps this is the intended behaviour, but at the very least it is a little confusing.



Discussion:
That's because `cache_readonly` is a class instance and not a class. 

I would remove the whole caching mechanism, and let the user use `joblib.Memory` if he wants to. If you want to keep it, at least consider making `cache_readonly` and co. proper signature-preserving decorators (use the `decorator` package).

Also, caching `cdf` is enough in any case. Caching `icdf` and `sf` as well only results in additional memory use.

On Fri, Jun 21, 2013 at 6:38 PM, Ralf Gommers notifications@github.comwrote:

> Also, caching cdf is enough in any case. Caching icdf and sf as well only
> results in additional memory use.

icdf (and potentially sf, I haven't looked into it) is calculated from the
sample, not the cdf (which is calculated as an integral of the kernel
estimator). Not sure I agree with this behavior, just pointing it out.

Padarn

> —
> Reply to this email directly or view it on GitHubhttps://github.com/statsmodels/statsmodels/issues/885#issuecomment-19804282
> .

Oh right, I was just staring at `sf` and assuming something about `icdf`.

> That's because cache_readonly is a class instance and not a class. 

`_cache  = resettable_cache()` needs to be in the `__init__` to get the cache for each instance. That's how we have it in RegressionResults, and I guess everywhere else.

I haven't looked at the cache decorator in 3 or 4 years. It was working pretty well for all the models, and I haven't seen any problems with it, IIRC.

I still need to check which results are added to the cache.

Also, based on reading the code: If fit is called a second time, with a different bandwith for example, then the decorated cached properties are not updated.

We might have to create/wipe the `resettable_cache()` in the fit method.

I don't know how this escaped us so long, essentially KDEUnivariate can only be used once. ???

On Fri, Jun 21, 2013 at 9:37 PM, Josef Perktold notifications@github.comwrote:

> I don't know how this escaped us so long, essentially KDEUnivariate can
> only be used once. ???
> 
> Some elements of the object don't seem to survive in the cache. I am still
> able to fit new data and evaluate the kernel estimator. So it isn't as
> broken as it could be, which is probably why it hasn't been obviously.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/statsmodels/statsmodels/issues/885#issuecomment-19811083
> .

Too little reuse: If fit is called with a different bandwidth, in a cross-validation loop for example, then we redo the binning each time in `kdensityfft`

> Some elements of the object don't seem to survive in the cache. I am still
> able to fit new data and evaluate the kernel estimator. So it isn't as
> broken as it could be, which is probably why it hasn't been obviously.

In my reading, only `density` should be updated. But the original example
from the mailing list might not be a good case to check this.

@Padarn can you try to put this in the fit method

`self_cache = resettable_cache()`

(I will soon be offline for the next several hours.)

On Fri, Jun 21, 2013 at 10:02 PM, Josef Perktold
notifications@github.comwrote:

> @Padarn https://github.com/Padarn can you try to put this in the fit
> method
> 
> self_cache = resettable_cache()
> 
> (I will soon be offline for the next several hours.)
> 
> Will do, but will be half a day before I get back to my work computer
> (middle of the night here)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/statsmodels/statsmodels/issues/885#issuecomment-19811936
> .

related: missing test coverage, 
only `density` and bw='silverman' are tested. 
kernels, weights and fft options for density are tested

Creating `self._cache` inside `fit()` solves @Padarn's issue, but there's still a lot of weirdness left. For example:

```
In [20]: pdf = smkern.KDEUnivariate(data1)

In [21]: hasattr(pdf, 'cdf')
Out[21]: False

In [22]: pdf.fit()

In [23]: hasattr(pdf, 'cdf')
Out[23]: True

In [24]: type(pdf.cdf)
Out[24]: numpy.ndarray
```

Hmm, this looks bad.

Isn't that ok though? We shouldn't have the other results until the model's been fit.

But why make it so complicated then? It would be much clearer to just say in the class docstring that cdf/icdf/sf are attributes, and create them directly as ndarrays within `fit`. Now they're coded as and documented to be methods, which they clearly aren't anymore due to the use of `cache_readonly`.

I'm not sure I follow that they're coded and documented as methods. They're just lazy properties like we have everywhere. If you just want the cdf, we don't calculate the other stuff, for instance.

Look under the heading `Methods`: http://statsmodels.sourceforge.net/devel/generated/statsmodels.nonparametric.kde.KDEUnivariate.html

The `def cdf():` with docstrings (now hidden and not introspectable by the way) also looks a lot like a method to me. The docstrings even start with "Returns ....".

Oh, this a long-standing issue with documenting properties. All of our lazy-properties are documented as methods like this. It's why we have this ugly stuff in places.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/linear_model.py#L720

We moved away from it to just document the docstrings like this but not necessarily duplicating them in the class docstring. Any suggestions for fixing it and retaining lazy evaluation are welcome. I've banged my head against it a few times but never came up with anything good.

I don't think it's fixable in an acceptable way. You could let `CachedAttribute.__get__` return something that acted like an ndarray but with the docstring you want, but that's very ugly.

Hmm, given it's already pretty magical (from where I sit), I wouldn't mind looking at fixing it like this. I don't see what kind of object it would be though. Maybe another issue for this (if there isn't already one).

An ndarray subclass with its docstring created from the decorated method. 

Yes, that would fix the introspection, but I don't know that it would "fool" sphinx. Would it?

No it wouldn't. I suspect that that's only fixable by modifying the autodoc extension to be `cache_readonly`-aware. Not fun.

I looked at the sphinx doc generation again two weeks ago. I didn't see any way to get rid of the `()` around the cached attributes, short of coding a special case in the sphinx extensions.

My recent issue was with the docstrings for cache decorated properties and regular attributes of results of hypothesis tests.

I haven't managed to understand part of the discussion above yet. However, KDE is a bit different from the models because with those the results instance doesn't exist to inspect before `fit()` has been called. In the KDE case, we attach or have decorated methods at the "model" instance.

Independent of the implementation details, the lazy properties/attributes are some of the best design decisions we have with "calculate on demand and don't recalculate unless necessary".

And to point out more fun: The Results classes of the models don't actually have the lazy attribute and it's docstring directly, the user actually has the wrapper instance, with possibly a pandas series or dataframe as returned data, that fetches the "real" `_results`.

But except for the docstring, sphinx documentation issue I haven't seen a problem there in a long time.

properties/descriptors (?) docstrings are accessed through the class not the instance in python, (IIRC, I don't find a quick reference)

```
help(smkern.KDEUnivariate.cdf)
```

maybe this (copied from above)

```
In [20]: pdf = smkern.KDEUnivariate(data1)

In [21]: hasattr(pdf, 'cdf')
Out[21]: False

In [22]: pdf.fit()

In [23]: hasattr(pdf, 'cdf')
Out[23]: True

In [24]: type(pdf.cdf)
Out[24]: numpy.ndarray
```

is "exactly" what it is supposed to be, 
before calling fit, `cdf` is not available as attributes, after calling fit() it is.
the attribute is a numpy array, when it exists.

However, introspection on the attribute, cashed property, triggers the calculation

```
>>> pdf
<statsmodels.nonparametric.kde.KDEUnivariate object at 0x052D7490>
>>> pdf._cache.keys()
[]
>>> pdf.fit()
>>> pdf._cache.keys()
[]
>>> hasattr(pdf, 'cdf')
True
>>> pdf._cache.keys()
['cdf']
```

My guess is that it's largely a feature of python properties and descriptors. (And I never went to figure out the tricky parts, to see if there is a way to cheat around it.)

Fernando had written some decorators that created descriptors for nipy that had a lot more magic than ours. But they stopped using it after a year or two.

I opened issue #886 for decorator question

It looks like this issue can be fixed/closed with

put `self_cache = resettable_cache()` in the `fit` method
add unit tests


Resolving patch:
diff --git a/statsmodels/nonparametric/kde.py b/statsmodels/nonparametric/kde.py
--- a/statsmodels/nonparametric/kde.py
+++ b/statsmodels/nonparametric/kde.py
@@ -78,7 +78,6 @@ class KDEUnivariate(object):
     >>> plt.show()
 
     """
-    _cache = resettable_cache()
 
     def __init__(self, endog):
         self.endog = np.asarray(endog)
@@ -150,6 +149,8 @@ def fit(self, kernel="gau", bw="scott", fft=True, weights=None,
         self.bw = bw
         self.kernel = kernel_switch[kernel](h=bw) # we instantiate twice,
                                                 # should this passed to funcs?
+        # put here to ensure empty cache after re-fit with new options
+        self._cache = resettable_cache()
 
     @cache_readonly
     def cdf(self):


Test patch:
diff --git a/statsmodels/nonparametric/tests/test_kde.py b/statsmodels/nonparametric/tests/test_kde.py
--- a/statsmodels/nonparametric/tests/test_kde.py
+++ b/statsmodels/nonparametric/tests/test_kde.py
@@ -88,6 +88,19 @@ def setupClass(cls):
         rfname2 = os.path.join(curdir,'results','results_kde_fft.csv')
         cls.res_density = np.genfromtxt(open(rfname2, 'rb'))
 
+class test_kde_refit():
+    np.random.seed(12345)
+    data1 = np.random.randn(100) * 100
+    pdf = KDE(data1)
+    pdf.fit()
+
+    data2 = np.random.randn(100) * 100
+    pdf2 = KDE(data2)
+    pdf2.fit()
+
+    for attr in ['icdf', 'cdf', 'sf']:
+        npt.assert_(not np.allclose(getattr(pdf, attr)[:10],
+                                    getattr(pdf2, attr)[:10]))
 
 if __name__ == "__main__":
     import nose