Repository: pandas-dev/pandas

Issue:
Vectorised addition of MonthOffset(n=0) returns different values to item-by-item addition
This code returns different values in `0.17.0` and `0.15.2`

``` python
import pandas as pd
from pandas.util.testing import assert_index_equal

pd.show_versions()

offsets = [
    pd.offsets.Day, pd.offsets.MonthBegin,
    pd.offsets.QuarterBegin, pd.offsets.YearBegin,
]

dates = pd.date_range('2011-01-01', '2011-01-05', freq='D')

for offset in offsets:
    # adding each item individually or vectorised should give same answer
    expected_vec = dates + offset(n=0)
    expected = pd.DatetimeIndex([d + offset(n=0) for d in dates])

    msg = "offset: {}, vectorised: {}, individual: {}".format(
        offset, expected_vec, expected
    )
    try:
        if pd.__version__ == '0.17.0':
            assert_index_equal(expected_vec, expected, check_names=False)
        else:
            assert_index_equal(expected_vec, expected)
    except AssertionError as er:
        raise Exception(msg + str(er))
```
### 0.17.0

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.0
setuptools: 18.0.1
Cython: 0.22
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.4.0
dateutil: 2.4.1
pytz: 2015.4
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.4
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.4
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
Traceback (most recent call last):
  File "c:\dev\code\sandbox\pandas_17_vs_15_dateoffsets.py", line 24, in <module>
    raise Exception(msg + str(er))
Exception: offset: <class 'pandas.tseries.offsets.MonthBegin'>, vectorised: DatetimeIndex(['2010-12-01', '2011-01-01', '2011-01-01', '2011-01-01',
               '2011-01-01'],
              dtype='datetime64[ns]', freq=None), individual: DatetimeIndex(['2011-01-01', '2011-02-01', '2011-02-01', '2011-02-01',
               '2011-02-01'],
              dtype='datetime64[ns]', freq=None)Index are different

Index values are different (100.0 %)
[left]:  DatetimeIndex(['2010-12-01', '2011-01-01', '2011-01-01', '2011-01-01',
               '2011-01-01'],
              dtype='datetime64[ns]', freq=None)
[right]: DatetimeIndex(['2011-01-01', '2011-02-01', '2011-02-01', '2011-02-01',
               '2011-02-01'],
              dtype='datetime64[ns]', freq=None)
```
### 0.15.2

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_GB

pandas: 0.15.2
nose: 1.3.7
Cython: 0.22
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.1
pytz: 2015.4
bottleneck: 1.0.0
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
```



Discussion:
This is from #10744, I didn't have the n=0 semantics right (and apparently didn't test!).  It'll be a couple days, but I'll submit a fix.

Many thanks for quick response!

`MonthEnd` also not working:

Test script

``` python
import pandas as pd
from pandas.util.testing import assert_index_equal

pd.show_versions()

offsets = [
    pd.offsets.MonthEnd,
    pd.offsets.QuarterEnd, pd.offsets.YearEnd,
]


dates = pd.date_range('2011-01-01', '2011-01-05', freq='D')

for offset in offsets:
    # adding each item individually or vectorised should give same answer
    expected_vec = dates + offset(n=0)
    expected = pd.DatetimeIndex([d + offset(n=0) for d in dates])

    msg = "offset: {}, vectorised: {}, individual: {}".format(
        offset, expected_vec, expected
    )
    try:
        if pd.__version__ == '0.17.0':
            assert_index_equal(expected_vec, expected, check_names=False)
        else:
            assert_index_equal(expected_vec, expected)
    except AssertionError as er:
        raise Exception(msg + str(er))
```

### 0.17.0

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.0
setuptools: 18.0.1
Cython: 0.22
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.4.0
dateutil: 2.4.1
pytz: 2015.4
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.4
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.4
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
Traceback (most recent call last):
  File "c:\dev\code\sandbox\pandas_17_vs_15_dateoffsets.py", line 34, in <module>
    raise Exception(msg + str(er))
Exception: offset: <class 'pandas.tseries.offsets.MonthEnd'>, vectorised: DatetimeIndex(['2010-12-31', '2010-12-31', '2010-12-31', '2010-12-31',
               '2010-12-31'],
              dtype='datetime64[ns]', freq=None), individual: DatetimeIndex(['2011-01-31', '2011-01-31', '2011-01-31', '2011-01-31',
               '2011-01-31'],
              dtype='datetime64[ns]', freq=None)Index are different

Index values are different (100.0 %)
[left]:  DatetimeIndex(['2010-12-31', '2010-12-31', '2010-12-31', '2010-12-31',
               '2010-12-31'],
              dtype='datetime64[ns]', freq=None)
[right]: DatetimeIndex(['2011-01-31', '2011-01-31', '2011-01-31', '2011-01-31',
               '2011-01-31'],
              dtype='datetime64[ns]', freq=None)
```

### 0.15.2

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_GB

pandas: 0.15.2
nose: 1.3.7
Cython: 0.22
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.1
pytz: 2015.4
bottleneck: 1.0.0
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
```

Probably also wrong for `YearEnd` and `QuarterEnd` too as the counting
logic is shared IIRC.

On Mon, Oct 19, 2015 at 10:31 AM, Petra Chong notifications@github.com
wrote:

> MonthEnd also not working:
> 
> Test script
> 
> import pandas as pdfrom pandas.util.testing import assert_index_equal
> 
> pd.show_versions()
> 
> offsets = [
>     pd.offsets.MonthEnd,
>     pd.offsets.QuarterEnd, pd.offsets.YearEnd,
> ]
> 
> dates = pd.date_range('2011-01-01', '2011-01-05', freq='D')
> for offset in offsets:
>     # adding each item individually or vectorised should give same answer
>     expected_vec = dates + offset(n=0)
>     expected = pd.DatetimeIndex([d + offset(n=0) for d in dates])
> 
> ```
> msg = "offset: {}, vectorised: {}, individual: {}".format(
>     offset, expected_vec, expected
> )
> try:
>     if pd.__version__ == '0.17.0':
>         assert_index_equal(expected_vec, expected, check_names=False)
>     else:
>         assert_index_equal(expected_vec, expected)
> except AssertionError as er:
>     raise Exception(msg + str(er))
> ```
> 
> 0.17.0
> 
> INSTALLED VERSIONS------------------
> commit: None
> python: 2.7.10.final.0
> python-bits: 64
> OS: Windows
> OS-release: 7
> machine: AMD64
> processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
> byteorder: little
> LC_ALL: None
> LANG: None
> 
> pandas: 0.17.0
> nose: 1.3.7
> pip: 7.1.0
> setuptools: 18.0.1
> Cython: 0.22
> numpy: 1.10.1
> scipy: 0.16.0
> statsmodels: 0.6.1
> IPython: 3.2.1
> sphinx: 1.3.1
> patsy: 0.4.0
> dateutil: 2.4.1
> pytz: 2015.4
> blosc: None
> bottleneck: 1.0.0
> tables: 3.2.2
> numexpr: 2.4.4
> matplotlib: 1.4.3
> openpyxl: None
> xlrd: 0.9.4
> xlwt: None
> xlsxwriter: 0.7.3
> lxml: None
> bs4: 4.3.2
> html5lib: 0.999
> httplib2: None
> apiclient: None
> sqlalchemy: 1.0.7
> pymysql: None
> psycopg2: None
> Traceback (most recent call last):
>   File "c:\dev\code\sandbox\pandas_17_vs_15_dateoffsets.py", line 34, in <module>
>     raise Exception(msg + str(er))Exception: offset: <class 'pandas.tseries.offsets.MonthEnd'>, vectorised: DatetimeIndex(['2010-12-31', '2010-12-31', '2010-12-31', '2010-12-31',
>                '2010-12-31'],
>               dtype='datetime64[ns]', freq=None), individual: DatetimeIndex(['2011-01-31', '2011-01-31', '2011-01-31', '2011-01-31',
>                '2011-01-31'],
>               dtype='datetime64[ns]', freq=None)Index are different
> 
> Index values are different (100.0 %)
> [left]:  DatetimeIndex(['2010-12-31', '2010-12-31', '2010-12-31', '2010-12-31',
>                '2010-12-31'],
>               dtype='datetime64[ns]', freq=None)
> [right]: DatetimeIndex(['2011-01-31', '2011-01-31', '2011-01-31', '2011-01-31',
>                '2011-01-31'],
>               dtype='datetime64[ns]', freq=None)
> 
> 0.15.2
> 
> INSTALLED VERSIONS------------------
> commit: None
> python: 2.7.10.final.0
> python-bits: 64
> OS: Windows
> OS-release: 7
> machine: AMD64
> processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
> byteorder: little
> LC_ALL: None
> LANG: en_GB
> 
> pandas: 0.15.2
> nose: 1.3.7
> Cython: 0.22
> numpy: 1.9.2
> scipy: 0.15.1
> statsmodels: None
> IPython: 3.2.1
> sphinx: 1.3.1
> patsy: 0.3.0
> dateutil: 2.4.1
> pytz: 2015.4
> bottleneck: 1.0.0
> tables: 3.2.0
> numexpr: 2.4.3
> matplotlib: 1.4.3
> openpyxl: 1.8.5
> xlrd: 0.9.4
> xlwt: 0.7.5
> xlsxwriter: 0.7.3
> lxml: 3.4.4
> bs4: 4.3.2
> html5lib: 0.999
> httplib2: None
> apiclient: None
> rpy2: None
> sqlalchemy: 1.0.7
> pymysql: None
> psycopg2: None
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/issues/11370#issuecomment-149250402.


Resolving patch:
diff --git a/pandas/tseries/offsets.py b/pandas/tseries/offsets.py
--- a/pandas/tseries/offsets.py
+++ b/pandas/tseries/offsets.py
@@ -444,7 +444,7 @@ def _beg_apply_index(self, i, freq):
         from pandas.tseries.frequencies import get_freq_code
         base, mult = get_freq_code(freq)
         base_period = i.to_period(base)
-        if self.n < 0:
+        if self.n <= 0:
             # when subtracting, dates on start roll to prior
             roll = np.where(base_period.to_timestamp() == i - off,
                             self.n, self.n + 1)
@@ -464,7 +464,7 @@ def _end_apply_index(self, i, freq):
         base, mult = get_freq_code(freq)
         base_period = i.to_period(base)
         if self.n > 0:
-            # when adding, dtates on end roll to next
+            # when adding, dates on end roll to next
             roll = np.where(base_period.to_timestamp(how='end') == i - off,
                             self.n, self.n - 1)
         else:
@@ -1081,8 +1081,7 @@ def apply(self, other):
 
     @apply_index_wraps
     def apply_index(self, i):
-        months = self.n - 1 if self.n >= 0 else self.n
-        shifted = tslib.shift_months(i.asi8, months, 'end')
+        shifted = tslib.shift_months(i.asi8, self.n, 'end')
         return i._shallow_copy(shifted)
 
     def onOffset(self, dt):
@@ -1108,8 +1107,7 @@ def apply(self, other):
 
     @apply_index_wraps
     def apply_index(self, i):
-        months = self.n + 1 if self.n < 0 else self.n
-        shifted = tslib.shift_months(i.asi8, months, 'start')
+        shifted = tslib.shift_months(i.asi8, self.n, 'start')
         return i._shallow_copy(shifted)
 
     def onOffset(self, dt):
@@ -1777,6 +1775,7 @@ def apply(self, other):
     @apply_index_wraps
     def apply_index(self, i):
         freq_month = 12 if self.startingMonth == 1 else self.startingMonth - 1
+        # freq_month = self.startingMonth
         freqstr =  'Q-%s' % (_int_to_month[freq_month],)
         return self._beg_apply_index(i, freqstr)
 


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -2622,7 +2622,8 @@ def test_datetime64_with_DateOffset(self):
                     assert_func(result, exp)
 
             s = klass([Timestamp('2000-01-05 00:15:00'), Timestamp('2000-01-31 00:23:00'),
-                       Timestamp('2000-01-01'), Timestamp('2000-02-29'), Timestamp('2000-12-31')])
+                       Timestamp('2000-01-01'), Timestamp('2000-03-31'),
+                       Timestamp('2000-02-29'), Timestamp('2000-12-31')])
 
             #DateOffset relativedelta fastpath
             relative_kwargs = [('years', 2), ('months', 5), ('days', 3),
@@ -2659,11 +2660,15 @@ def test_datetime64_with_DateOffset(self):
                         else:
                             do = do
                             kwargs = {}
-                        op = getattr(pd.offsets,do)(5, normalize=normalize, **kwargs)
-                        assert_func(klass([x + op for x in s]), s + op)
-                        assert_func(klass([x - op for x in s]), s - op)
-                        assert_func(klass([op + x for x in s]), op + s)
 
+                        for n in [0, 5]:
+                            if (do in ['WeekOfMonth','LastWeekOfMonth',
+                                       'FY5253Quarter','FY5253'] and n == 0):
+                                continue
+                            op = getattr(pd.offsets,do)(n, normalize=normalize, **kwargs)
+                            assert_func(klass([x + op for x in s]), s + op)
+                            assert_func(klass([x - op for x in s]), s - op)
+                            assert_func(klass([op + x for x in s]), op + s)
     # def test_add_timedelta64(self):
     #     rng = date_range('1/1/2000', periods=5)
     #     delta = rng.values[3] - rng.values[1]