Repository: pandas-dev/pandas

Issue:
"AssertionError: Index length did not match values" when resampling with kind='period'
Should raise that `kind='period'` is not accepted for `DatetimeIndex` when resampling
Possible issue with period index resampling hanging (see @cpcloud example below)

version = 0.12.0.dev-f61d7e3

This bug also exists in 0.11.
## The bug

``` Python
In [20]: s.resample('T', kind='period')
-----------------
AssertionError  
Traceback (most recent call last)
<ipython-input-79-c290c0578332> in <module>()
----> 1 s.resample('T', kind='period')

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/core/generic.py in resample(self, rule, how, axis, fill_method, closed, label, convention, kind, loffset, limit, base)
    255                               fill_method=fill_method, convention=convention,
    256                               limit=limit, base=base)
--> 257         return sampler.resample(self)
    258 
    259     def first(self, offset):

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/tseries/resample.py in resample(self, obj)
     81 
     82         if isinstance(axis, DatetimeIndex):
---> 83             rs = self._resample_timestamps(obj)
     84         elif isinstance(axis, PeriodIndex):
     85             offset = to_offset(self.freq)

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/tseries/resample.py in _resample_timestamps(self, obj)
    224             # Irregular data, have to use groupby
    225             grouped = obj.groupby(grouper, axis=self.axis)
--> 226             result = grouped.aggregate(self._agg_method)
    227 
    228             if self.fill_method is not None:

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/core/groupby.py in aggregate(self, func_or_funcs, *args, **kwargs)
   1410         if isinstance(func_or_funcs, basestring):
-> 1411             return getattr(self, func_or_funcs)(*args, **kwargs)
   1412 
   1413         if hasattr(func_or_funcs, '__iter__'):

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/core/groupby.py in mean(self)
    356         except Exception:  # pragma: no cover
    357             f = lambda x: x.mean(axis=self.axis)
--> 358             return self._python_agg_general(f)
    359 
    360     def median(self):

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/core/groupby.py in _python_agg_general(self, func, *args, **kwargs)
    498                 output[name] = self._try_cast(values[mask],result)
    499 
--> 500         return self._wrap_aggregated_output(output)
    501 
    502     def _wrap_applied_output(self, *args, **kwargs):

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/core/groupby.py in _wrap_aggregated_output(self, output, names)
   1473             return DataFrame(output, index=index, columns=names)
   1474         else:
-> 1475             return Series(output, index=index, name=self.name)
   1476 
   1477     def _wrap_applied_output(self, keys, values, not_indexed_same=False):

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/core/series.py in __new__(cls, data, index, dtype, name, copy)
    494         else:
    495             subarr = subarr.view(Series)
--> 496         subarr.index = index
    497         subarr.name = name
    498 

/home/dk3810/workspace/python/pda/scripts/src/pandas/pandas/lib.so in pandas.lib.SeriesIndex.__set__ (pandas/lib.c:29775)()

AssertionError: Index length did not match values
```
## A workaround / expected behaviour

``` Python
In [81]: s.resample('T').to_period()
Out[81]: 
2013-04-12 19:15    325.000000
2013-04-12 19:16    326.899994
...
2013-04-12 22:58    305.600006
2013-04-12 22:59    320.444458
Freq: T, Length: 225, dtype: float32
```
## More information

``` Python

In [83]: s
Out[83]: 
2013-04-12 19:15:25    323
2013-04-12 19:15:28    NaN
...
2013-04-12 22:59:55    319
2013-04-12 22:59:56    NaN
2013-04-12 22:59:57    NaN
2013-04-12 22:59:58    NaN
2013-04-12 22:59:59    NaN
Name: aggregate, Length: 13034, dtype: float32

In [76]: s.index
Out[76]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-04-12 19:15:25, ..., 2013-04-12 22:59:59]
Length: 13034, Freq: None, Timezone: None

In [77]: s.head()
Out[77]: 
2013-04-12 19:15:25    323
2013-04-12 19:15:28    NaN
2013-04-12 19:15:29    NaN
2013-04-12 19:15:30    NaN
2013-04-12 19:15:31    327
Name: aggregate, dtype: float32

In [78]: s.resample('T')
Out[78]: 
2013-04-12 19:15:00    325.000000
2013-04-12 19:16:00    326.899994
...
2013-04-12 22:58:00    305.600006
2013-04-12 22:59:00    320.444458
Freq: T, Length: 225, dtype: float32

In [80]: pd.__version__
Out[80]: '0.12.0.dev-f61d7e3'

In [84]: type(s)
Out[84]: pandas.core.series.TimeSeries
```

(Please let me know if you need more info!  I'm using Ubuntu 13.04.  It's entirely possible that this isn't a bug but instead I am doing something stupid.  Oh, and let me take this opportunity to thank the Pandas dev team!  Pandas is awesome!!!  THANK YOU!)

More informative exception when trying to use ``MS`` as period frequency
Closes #5332



Discussion:
This is not supported. As you indicated you can resample then `to_period`, or

`s.to_period().resample('T',kind='period'` will also work

I'll make this an enhancement/bug request, because it should raise a helpful message (or be implemented)

thanks

wow, thanks for the very swift response ;)

I actually don't get an error here. It just hangs.

@cpcloud can you post what you did?

@jreback Yeah, sorry to hit and run like that :)

``` python
dind = period_range('1/1/2001', '1/1/2002').to_timestamp()
s = Series(randn(dind.size), dind)
s.resample('T', kind='period')  # hangs here
```

The other ways of doing this (from above) work fine. Doesn't hang (throws the above error) for the simple case of 

``` python
dind = period_range('1/1/2001', '1/2/2001').to_timestamp()
s = Series(randn(dind.size), dind)
s.resample('T', kind='period')  # hangs here
```

and starts to hang for `dind.size > 2`.

hmm...that might be something else
I replicated @JackKelly was donig by this

```
In [16]: s = Series(range(100),index=date_range('20130101',freq='s',periods=100),dtype='float')

In [17]: s[10:30] = np.nan

In [18]: s.to_period().resample('T',kind='period')
Out[18]: 
2013-01-01 00:00    34.5
2013-01-01 00:01    79.5
Freq: T, dtype: float64

In [19]: s.resample('T',kind='period')
AssertionError: Index length did not match values
```

Yeah that works.

should i open an issue for the above? seems to be a day frequency issue.

yeh....(I put it in the header) so just ref this issue too

@jreback i would like to clear this up since clearing it up would actually close 3 issues: this (#3609), #3612, and #3899. what's the original reason for not supporting this...my current fix loses the last element when resampling from datetimes to period so i'm guessing that might be one issue...but that's because of the ability to choose your resampling either include the start/end point of datetimes which periods don't have

one issue that two element case is not handled

now i'm getting a segfault when i try to use `sum` ... joy

move back to 0.13 then?

I'm not convinced this is reasonable to do. I don't get a keyerror when using "MS" - I think it's just a backwards compatibility thing where MS is always made into milliseconds (which is what "L" means too).

@jtratner I thought you had originally been seeing the key error? When I use Pandas v0.12, I definitely am getting key error.

The ability to use MS to mean milliseconds is inconsistent. There are parts of the Pandas code that explicitly prevent this and others that do not.

@cancan101 seems like the underlying problem is that this fails poorly:

``` python
pd.Period('2013').asfreq(freq='L')
```

Also, Wakari version of pandas gets a `KeyError` on `'MS'`, guess I'm just special.

I actually think there are two issues here. One is that MS, in my opinion,
imcorrectly maps to L. The second problem is that L is not recognized as a
valid period frequency.

On Sunday, October 27, 2013, Jeff Tratner wrote:

> @cancan101 https://github.com/cancan101 seems like the underlying
> problem is that this fails poorly:
> 
> pd.Period('2013').asfreq(freq='L')
> 
> Also, Wakari version of pandas gets a KeyError on 'MS', guess I'm just
> special.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5340#issuecomment-27184392
> .

Prefer to address both. Also, what should MS map to, if anything?

Actually, what I'd really prefer is that passing an unknown frequency raises `ValueError("Unknown frequency: %s")` vs. `KeyError`. It's okay to separate out `'L'` /`'MS'`, etc.  But definitely should warn instead of just removing it.

Initially, I thought that `MS` should map to `MonthBegin` but in thinking about that, it doesn't make sense for a period. This is because a period encompasses both the start and end time stamps so Month (i.e. `M`) should be all that is needed to describe a "month" period. I think that `MS` should not map to anything. As for `ValueError`, that is what my new tests checks for: https://github.com/pydata/pandas/pull/5340/files#diff-f9b276c4aa39a6161726d8b43ce62516R496


Resolving patch:
diff --git a/pandas/tseries/resample.py b/pandas/tseries/resample.py
--- a/pandas/tseries/resample.py
+++ b/pandas/tseries/resample.py
@@ -192,7 +192,9 @@ def _get_time_period_bins(self, axis):
         labels = binner = PeriodIndex(start=axis[0], end=axis[-1],
                                       freq=self.freq)
 
-        end_stamps = (labels + 1).asfreq('D', 's').to_timestamp()
+        end_stamps = (labels + 1).asfreq(self.freq, 's').to_timestamp()
+        if axis.tzinfo:
+            end_stamps = end_stamps.tz_localize(axis.tzinfo)
         bins = axis.searchsorted(end_stamps, side='left')
 
         return binner, bins, labels


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -1836,6 +1836,37 @@ def test_concat_datetime_datetime64_frame(self):
         # it works!
         pd.concat([df1, df2_obj])
 
+    def test_period_resample(self):
+        # GH3609
+        s = Series(range(100),index=date_range('20130101', freq='s', periods=100), dtype='float')
+        s[10:30] = np.nan
+        expected = Series([34.5, 79.5], index=[Period('2013-01-01 00:00', 'T'), Period('2013-01-01 00:01', 'T')])
+        result = s.to_period().resample('T', kind='period')
+        assert_series_equal(result, expected)
+        result2 = s.resample('T', kind='period')
+        assert_series_equal(result2, expected)
+
+    def test_period_resample_with_local_timezone(self):
+        # GH5430
+        _skip_if_no_pytz()
+        import pytz
+
+        local_timezone = pytz.timezone('America/Los_Angeles')
+
+        start = datetime(year=2013, month=11, day=1, hour=0, minute=0, tzinfo=pytz.utc)
+        # 1 day later
+        end = datetime(year=2013, month=11, day=2, hour=0, minute=0, tzinfo=pytz.utc)
+
+        index = pd.date_range(start, end, freq='H')
+
+        series = pd.Series(1, index=index)
+        series = series.tz_convert(local_timezone)
+        result = series.resample('D', kind='period')
+        # Create the expected series
+        expected_index = (pd.period_range(start=start, end=end, freq='D') - 1)  # Index is moved back a day with the timezone conversion from UTC to Pacific
+        expected = pd.Series(1, index=expected_index)
+        assert_series_equal(result, expected)
+
 
 def _simple_ts(start, end, freq='D'):
     rng = date_range(start, end, freq=freq)