Fixes usage of CacheableOffset
State: closed

`CacheableOffset` should be first parent class when sub-classing both `CacheableOffset` and `DateOffset`

Closes #4609


Labels:

Comments:
pls add a release notes entry

Where do I do that?
On Aug 20, 2013 9:23 AM, "jreback" notifications@github.com wrote:

> pls add a release notes entry
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4614#issuecomment-22944152
> .

edit `pandas/docs/source/release.rst` and add a line in the Bugs section (with the issue reference)

also after release notes, pls squash down to 1 commit

e.g

```
git rebase -i <commit before your first>
```

then `s` or `f` the commits

```
git push origin yourbranch -f
```

Do you want me to squash the commit with the test identifying the issue with the commit containing the fix?

yes...when you are done squash down to 1 commit (you can leave as 2 if you want..up2u)

take a look at `core/index.py/_cached_range` where this is all used

I think you mean  `pandas/tseries/index/_cached_range` ? Which I know about; one of the refactorings in this branch is around calling that method.

Looking at that method, this line:

```
cachedRange[startLoc:endLoc]
```

even when called with the same `startLoc` and `endLoc` returns both a different range object AND the items with the range are not the same.

@cancan101 yep..that's what I meant!

so the caching should set this range and then retrieve the already cached objects from the cache (if they are in range)....

It should be a different range object (as its a new instance), BUT the objects inside should not be recreated (I think)

so if you created a date_range twice with overlapping ranges, then you should get identical objects (e.g. Period/Timestamp), though the actual indicies would be different

@jreback That is what I meant when I was complaining about this not working:

```
f = pd.DatetimeIndex(start="2013-01", end="2013-03", freq="M")
# This fails:
assert f[0] is f[0] 
```

I _think_ that both these should be True; could be a bug

```
In [1]: f1 = pd.DatetimeIndex(start="2013-01", end="2013-03", freq="M")

In [2]: f2 = pd.DatetimeIndex(start="2013-01", end="2013-03", freq="M")

In [3]: f1[0] == f2[0]
Out[3]: True

In [4]: f1[0] is f2[0]
Out[4]: False
```

Okay but should:
`f1[0] is f2[0]` be `True`? I would think yes.

I would thing that this should definitely be `True`:
`f1[0] is f1[0]`

Hmm....Not sure if Timestamp's are 'cached'....I think they may need a `__nonzero__` or maybe `__hash__` (though I think they have hash)....method to provide the `is` identity testing

I actually think that `is` cannot be overridden / does not need to be overridden to work.

See: http://stackoverflow.com/a/3647786/2638485

true....

@cancan101 can you trace a cache and see if it could work?

So looking at the code for `DateTimeIndex`, a scalar access to leads to:

```
return Timestamp(val, offset=self.offset, tz=self.tz)
```

`Timestamp` does have a  `__new__` method which calls up to `datetime`. I don't think there is any effort here to cache / intern.

There is a chance that slicing as opposed to reading one element will use the same objects. I can give that a try:

```
f1 = pd.DatetimeIndex(start="2013-01", end="2013-03", freq="M")[0:-1]
f2 = pd.DatetimeIndex(start="2013-01", end="2013-03", freq="M")[0:-1]
```

and then go from there.

It looks like numpy is not particularly helpful here with caching:

```
a = np.array([1,2,3,])
In [16]: a[0] is a[0]
Out[16]: False

In [17]: a.item(0) is a.item(0)
Out[17]: True
```

numpy is different because what you are comparing are 2 different views, however integers are cached (by python) so that's why `1 is 1` will always be True

I suppose `Timestamps` could be cached, not sure there is a whole lot to gain (which is why it may have been put in place, but turned off)

@wesm ?

Ah, yes. That example from numpy is no good. The brackets return `numpy.int64` and the `item` returns `int`.

@jreback This PR should be squashed and good to merge

one more thing....check perf

`vb_suite/test_perf.sh -b <commit before yours> -t <last commit in yours>`

check if anything comes up say > 1.2 ratio

Timestamp boxes are created when you request an element. NumPy does the same thing with scalar values:

```
In [1]: arr = np.arange(10)

In [2]: arr[2]
Out[2]: 2

In [3]: arr[2] is arr[2]
Out[3]: False
```

@wesm That makes sense and is consistent with what I have observed. That being said, what is the point of this entire `_cached_range` infrastructure? i.e. what allocation / object creation is it attempting to avoid?

The point was to avoid the regeneration of `datetime.datetime` objects (and the slow date offset logic). At the time I was building applications in pandas in which hundreds of distinct date ranges would be created and creating a single array for each frequency and retrieving views on that was a big performance and memory usage win overall.

@wesm That totally makes sense...if that is in fact happening. In my attempts to test whether caching is actually working, I could not tell if when I slice the DateTimeIndex (i.e. the call to `cachedRange[startLoc:endLoc]`) if new objects are being created for each slice or not. All of my sttempts to prove referential equality on the items in slice failed.

Check the datetime object ids (`id(obj)`)

That does not seem to work:

```
In [71]: id(pd.tseries.index._daterange_cache[pd.offsets.QuarterEnd(startingMonth=9)][0:10][0])
Out[71]: 43833552

In [72]: id(pd.tseries.index._daterange_cache[pd.offsets.QuarterEnd(startingMonth=9)][0:10][0])
Out[72]: 43834832

In [73]: id(pd.tseries.index._daterange_cache[pd.offsets.QuarterEnd(startingMonth=9)][0:10][0])
Out[73]: 43834576

In [74]: id(pd.tseries.index._daterange_cache[pd.offsets.QuarterEnd(startingMonth=9)][0:10][0])
Out[74]: 43833552

In [75]: id(pd.tseries.index._daterange_cache[pd.offsets.QuarterEnd(startingMonth=9)][0:10][0])
Out[75]: 43834832

In [76]: id(pd.tseries.index._daterange_cache[pd.offsets.QuarterEnd(startingMonth=9)][0:10][0])
Out[76]: 43834576
```