Repository: statsmodels/statsmodels

Issue:
TSA bug with frequency inference
Somehow the frequency is not being set properly here and when properly set it gets set to None somewhere.

http://stackoverflow.com/questions/15515019/statsmodel-using-arma



Discussion:
The problem is that these dates do not have a discernible frequency. So we need to be able to not return the wrapped results here or to just assign an index of integers rather.

What should we return for the dates index for prediction in cases of non-consecutive days frequencies, for instance, business days? Should we also return business days dates, or just fall back on an integer index for the series in this case too?

Do you have enough information to create dates, future dates that are in the corresponding business day frequency/sequence?
If you can handle it, we could just pretend it's a different calender where non-business days don't exist, one-step is one business day independent of actual/real time.

The conflicting interpretation (which we don't support yet) is to assume subordination of a continuous time process (for example). The underlying process is in "real" time but we observe it only at irregularly spaced time intervals.
(or think of AR in space with irregular distances instead of time). In this case, one step ahead would be one real time unit, not observed calender time unit.

Yeah I do have enough info. It's no extra work, we get it for free from pandas, I just don't know if it makes sense. I'll go ahead and let business day through. 

The thing to fix is just when there's no frequency, it's irregularly spaced (or there's bank holidays interspersed). Then we'll just return an integer index.

There are a few corner cases here and this is a bit more work than I thought, so I'm going to lay out what needs to be done if I can't finish this shortly, so I can return to it. Some of it is already done in a local branch. When there is no discernible frequency to the dates, we need to
1. Raise an exception when start date is not on the dates index. If dates are 12-30-09, 1-3-10, 1-4-10, we don't know what to do with a start date of 1-1-10. In the future, we will intelligently roll forward onto 1-3-10, but I'm not going to code this until we drop support for pandas 0.7.x, so it's easier and cleaner to do.
2. If end is a date, it must be on the index in-sample. If not, raise an exception. If it is, then predict_dates should just be a slice of the in-sample date index.
3. If you want to predict out of sample, end must be an integer. We won't know what predict from 1-1-2010 1-8-2010 means if the last in-sample time is, for example, 1-2-2010 without any frequency information.
4. Don't forget to also test the forecast method.


Resolving patch:
diff --git a/statsmodels/tsa/arima_model.py b/statsmodels/tsa/arima_model.py
--- a/statsmodels/tsa/arima_model.py
+++ b/statsmodels/tsa/arima_model.py
@@ -477,7 +477,9 @@ def predict(self, params, start=None, end=None, exog=None, dynamic=False):
         end : int, str, or datetime
             Zero-indexed observation number at which to end forecasting, ie.,
             the first forecast is start. Can also be a date string to
-            parse or a datetime type.
+            parse or a datetime type. However, if the dates index does not
+            have a fixed frequency, end must be an integer index if you
+            want out of sample prediction.
         exog : array-like, optional
             If the model is an ARMAX and out-of-sample forecasting is
             requested, exog must be given.
@@ -1251,8 +1253,8 @@ def predict(self, start=None, end=None, exog=None, dynamic=False):
             parse or a datetime type.
         end : int, str, or datetime
             Zero-indexed observation number at which to end forecasting, ie.,
-            the first forecast is start. Can also be a date string to
-            parse or a datetime type.
+            the last forecast is end. Can also be a date string to parse or a
+            datetime type.
         exog : array-like, optional
             If the model is an ARMAX and out-of-sample forecasting is
             requestion, exog must be given.
@@ -1429,7 +1431,9 @@ def predict(self, start=None, end=None, exog=None, typ='linear',
         end : int, str, or datetime
             Zero-indexed observation number at which to end forecasting, ie.,
             the first forecast is start. Can also be a date string to
-            parse or a datetime type.
+            parse or a datetime type. However, if the dates index does not
+            have a fixed frequency, end must be an integer index if you
+            want out of sample prediction.
         exog : array-like, optional
             If the model is an ARMAX and out-of-sample forecasting is
             requestion, exog must be given.
diff --git a/statsmodels/tsa/base/datetools.py b/statsmodels/tsa/base/datetools.py
--- a/statsmodels/tsa/base/datetools.py
+++ b/statsmodels/tsa/base/datetools.py
@@ -28,6 +28,9 @@ def _index_date(date, dates):
     prediction can start one out.
 
     Currently used to validate prediction start dates.
+
+    If there dates are not of a fixed-frequency and date is not on the
+    existing dates, then a ValueError is raised.
     """
     if isinstance(date, basestring):
         date = date_parser(date)
@@ -35,11 +38,22 @@ def _index_date(date, dates):
         return dates.get_loc(date)
     except KeyError, err:
         freq = _infer_freq(dates)
+        if freq is None:
+            #TODO: try to intelligently roll forward onto a date in the
+            # index. Waiting to drop pandas 0.7.x support so this is
+            # cleaner to do.
+            raise ValueError("There is no frequency for these dates and "
+                             "date %s is not in dates index. Try giving a "
+                             "date that is in the dates index or use "
+                             "an integer" % date)
+
         # we can start prediction at the end of endog
         if _idx_from_dates(dates[-1], date, freq) == 1:
             return len(dates)
 
-        raise ValueError("date %s not in date index" % date)
+        raise ValueError("date %s not in date index. Try giving a "
+                         "date that is in the dates index or use an integer"
+                         % date)
 
 def _date_from_idx(d1, idx, freq):
     """
diff --git a/statsmodels/tsa/base/tsa_model.py b/statsmodels/tsa/base/tsa_model.py
--- a/statsmodels/tsa/base/tsa_model.py
+++ b/statsmodels/tsa/base/tsa_model.py
@@ -73,6 +73,13 @@ def _set_exog_names(self, vals):
     #overwrite with writable property for (V)AR models
     exog_names = property(_get_exog_names, _set_exog_names)
 
+    def _get_dates_loc(self, dates, date):
+        if hasattr(dates, 'indexMap'): # 0.7.x
+            date = dates.indexMap[date]
+        else:
+            date = dates.get_loc(date)
+        return date
+
     def _str_to_date(self, date):
         """
         Takes a string and returns a datetime object
@@ -107,11 +114,7 @@ def _get_predict_start(self, start):
             try:
                 dtstart = self._str_to_date(start)
                 self.data.predict_start = dtstart
-                # for pandas 0.7.x vs 0.8.x
-                if hasattr(dates, 'indexMap'): # 0.7.x
-                    start = dates.indexMap[dtstart]
-                else:
-                    start = dates.get_loc(dtstart)
+                start = self._get_dates_loc(dates, dtstart)
             except: # this catches all errors in the above..
                     #FIXME to be less greedy
                 raise ValueError("Start must be in dates. Got %s | %s" %
@@ -132,17 +135,15 @@ def _get_predict_end(self, end):
             end = len(self.data.endog) - 1
 
         dates = self.data.dates
+        freq = self.data.freq
+
         if isinstance(end, str):
             if dates is None:
                 raise ValueError("Got a string for end and dates is None")
             try:
                 dtend = self._str_to_date(end)
                 self.data.predict_end = dtend
-                # for pandas 0.7.x vs 0.8.x
-                if hasattr(dates, 'indexMap'): # 0.7.x
-                    end = dates.indexMap[dtend]
-                else:
-                    end = dates.get_loc(dtend)
+                end = self._get_dates_loc(dates, dtend)
             except KeyError, err: # end is greater than dates[-1]...probably
                 if dtend > self.data.dates[-1]:
                     end = len(self.data.endog) - 1
@@ -150,7 +151,14 @@ def _get_predict_end(self, end):
                     out_of_sample = datetools._idx_from_dates(dates[-1], dtend,
                                             freq)
                 else:
-                    raise err
+                    if freq is None:
+                        raise ValueError("There is no frequency for these "
+                                         "dates and date %s is not in dates "
+                                         "index. Try giving a date that is in "
+                                         "the dates index or use an integer."
+                                         % dtend)
+                    else: #pragma: no cover
+                        raise err # should never get here
             self._make_predict_dates() # attaches self.data.predict_dates
 
         elif isinstance(end, int) and dates is not None:
@@ -160,9 +168,22 @@ def _get_predict_end(self, end):
                 nobs = len(self.data.endog) - 1 # as an index
                 out_of_sample = end - nobs
                 end = nobs
-                freq = self.data.freq
-                self.data.predict_end = datetools._date_from_idx(dates[-1],
-                        out_of_sample, freq)
+                if freq is not None:
+                    self.data.predict_end = datetools._date_from_idx(dates[-1],
+                            out_of_sample, freq)
+                elif out_of_sample <= 0: # have no frequency but are in sample
+                    #TODO: what error to catch here to make sure dates is
+                    #on the index?
+                    try:
+                        self.data.predict_end = self._get_dates_loc(dates,
+                                                end)
+                    except KeyError:
+                        raise
+                else:
+                    self.data.predict_end = end + out_of_sample
+                    self.data.predict_start = self._get_dates_loc(dates,
+                                                self.data.predict_start)
+
             self._make_predict_dates()
 
         elif isinstance(end, int):
@@ -171,6 +192,10 @@ def _get_predict_end(self, end):
                 out_of_sample = end - nobs
                 end = nobs
 
+        elif freq is None: # should have a date with freq = None
+            raise ValueError("When freq is None, you must give an integer "
+                             "index for end.")
+
         return end, out_of_sample
 
     def _make_predict_dates(self):
@@ -178,14 +203,28 @@ def _make_predict_dates(self):
         dtstart = data.predict_start
         dtend = data.predict_end
         freq = data.freq
-        pandas_freq = _freq_to_pandas[freq]
-        try:
-            from pandas import DatetimeIndex
-            dates = DatetimeIndex(start=dtstart, end=dtend,
-                                    freq=pandas_freq)
-        except ImportError, err:
-            from pandas import DateRange
-            dates = DateRange(dtstart, dtend, offset = pandas_freq).values
+
+        if freq is not None:
+            pandas_freq = _freq_to_pandas[freq]
+            try:
+                from pandas import DatetimeIndex
+                dates = DatetimeIndex(start=dtstart, end=dtend,
+                                        freq=pandas_freq)
+            except ImportError, err:
+                from pandas import DateRange
+                dates = DateRange(dtstart, dtend, offset = pandas_freq).values
+        # handle
+        elif freq is None and (isinstance(dtstart, int) and
+                               isinstance(dtend, int)):
+            from pandas import Index
+            dates = Index(range(dtstart, dtend+1))
+        # if freq is None and dtstart and dtend aren't integers, we're
+        # in sample
+        else:
+            dates = self.data.dates
+            start = self._get_dates_loc(dates, dtstart)
+            end = self._get_dates_loc(dates, dtend)
+            dates = dates[start:end+1] # is this index inclusive?
         self.data.predict_dates = dates
 
 class TimeSeriesModelResults(base.LikelihoodModelResults):


Test patch:
diff --git a/statsmodels/tsa/tests/test_arima.py b/statsmodels/tsa/tests/test_arima.py
--- a/statsmodels/tsa/tests/test_arima.py
+++ b/statsmodels/tsa/tests/test_arima.py
@@ -1596,6 +1596,54 @@ def test_arima_predict_q2():
                         [7.306320, 7.313825, 7.321749, 7.329827, 7.337962],
                         5)
 
+def test_arima_predict_pandas_nofreq():
+    # this is issue 712
+    try:
+        from pandas.tseries.api import infer_freq
+    except ImportError:
+        import nose
+        raise nose.SkipTest
+    from pandas import DataFrame, DatetimeIndex
+    dates = ["2010-01-04", "2010-01-05", "2010-01-06", "2010-01-07",
+             "2010-01-08", "2010-01-11", "2010-01-12", "2010-01-11",
+             "2010-01-12", "2010-01-13", "2010-01-17"]
+    close = [626.75, 623.99, 608.26, 594.1, 602.02, 601.11, 590.48, 587.09,
+             589.85, 580.0,587.62]
+    data = DataFrame(close, index=DatetimeIndex(dates), columns=["close"])
+
+    #TODO: fix this names bug for non-string names names
+    arma = ARMA(data, order=(1,0)).fit(disp=-1)
+
+    # first check that in-sample prediction works
+    predict = arma.predict()
+    assert_(predict.index.equals(data.index))
+
+    # check that this raises an exception when date not on index
+    assert_raises(ValueError, arma.predict, start="2010-1-9", end=10)
+    assert_raises(ValueError, arma.predict, start="2010-1-9", end="2010-1-17")
+
+    # raise because end not on index
+    assert_raises(ValueError, arma.predict, start="2010-1-4", end="2010-1-10")
+    # raise because end not on index
+    assert_raises(ValueError, arma.predict, start=3, end="2010-1-10")
+
+    predict = arma.predict(start="2010-1-7", end=10) # should be of length 10
+    assert_(len(predict) == 8)
+    assert_(predict.index.equals(data.index[3:10+1]))
+
+    predict = arma.predict(start="2010-1-7", end=14)
+    assert_(predict.index.equals(pandas.Index(range(3, 15))))
+
+    predict = arma.predict(start=3, end=14)
+    assert_(predict.index.equals(pandas.Index(range(3, 15))))
+
+    # end can be a date if it's in the sample and on the index
+    # predict dates is just a slice of the dates index then
+    predict = arma.predict(start="2010-1-6", end="2010-1-13")
+    assert_(predict.index.equals(data.index[2:10]))
+    predict = arma.predict(start=2, end="2010-1-13")
+    assert_(predict.index.equals(data.index[2:10]))
+
 
 if __name__ == "__main__":
     import nose