Repository: statsmodels/statsmodels

Issue:
DOC/BUG: `py_hamilton_filter` doesn't handle time-dependent transition probabilities the same as `cy_hamilton_filter`.
The documentation for statsmodels.tsa.regime_switching.markov_switching.py_hamilton_filter suggests that it can handle time-dependent transition probabilities, but the following script fails when I run it.

`
import sys
import numpy as np
import statsmodels
import statsmodels.tsa.regime_switching.markov_switching as ms

print "Python version:", sys.version
print "Numpy version:", np.version.full_version
print "statsmodels version:", statsmodels.version.full_version


np.random.seed(10)
k_regimes = 2
nobs = 4
order = 1

_initial_probabilities = np.arange(1, k_regimes + 1, dtype='float64')
initial_probabilities = _initial_probabilities / np.sum(_initial_probabilities)

_regime_transition = np.random.exponential(size=(k_regimes, k_regimes, nobs))
regime_transition = _regime_transition / np.sum(_regime_transition, axis=0)

conditional_likelihoods_shape = (1 + order) * (k_regimes,) + (nobs,)
conditional_likelihoods = np.random.exponential(size=conditional_likelihoods_shape)

res1 = ms.cy_hamilton_filter(initial_probabilities, regime_transition, conditional_likelihoods)
res2 = ms.py_hamilton_filter(initial_probabilities, regime_transition, conditional_likelihoods)
`

Result on my machine:

C:\Data\python\venv\lib\site-packages\statsmodels\compat\pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools
Python version: 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)]
Numpy version: 1.13.3
statsmodels version: 0.8.0
Traceback (most recent call last):
  File "temp/example.py", line 26, in <module>
    res2 = ms.py_hamilton_filter(initial_probabilities, regime_transition, conditional_likelihoods)
  File "C:\Data\python\venv\lib\site-packages\statsmodels\tsa\regime_switching\markov_switching.py", line 198, in py_hamilton_filter
    regime_transition[..., transition_t] *
IndexError: index 3 is out of bounds for axis 2 with size 3

This may just be a documentation bug, as it looks like `regime_transition` needs to be longer than `nobs`, but then it seems like the cython version should also fail above.


Discussion:
Thanks for the report!

In this case there are two things going on:

First, the documentation is incorrect on `py_hamilton_filter` and `cy_hamilton_filter`. The `conditional_likelihoods` array should have last dimension `nobs - order` rather than `nobs`. This is because if `order >= 1`, then you can't compute conditional likelihoods for first `order` periods.

So if you modify your example so that:

```
conditional_likelihoods_shape = (1 + order) * (k_regimes,) + (nobs-order,)
```

then it will work.

The second thing going on here is that, to maximize the speed of the Cython code, we have used the option `boundscheck=False`. This means that Cython will essentially never raise an `IndexError`, but if you pass arrays with the wrong sizes then the results will likely be wrong and maybe a memory error could happen if Cython is trying to access memory it shouldn't. Of course if we use this option, we should have the correct array sizes in the documentation :)
Makes sense.

I could probably throw together a pull request that updates the docstrings and does some validation on the dimensions of the inputs for both functions.  Would that be useful?  Does that require any paperwork?  I don't see anything about paperwork here http://www.statsmodels.org/stable/dev/index.html.  This would be a pretty trivial improvement, so its may not be worth it if the pain of code review and/or paperwork outweighs the benefits.

Thanks for prompt response.
A PR would be great! No paperwork required. Thanks again!

Resolving patch:
diff --git a/statsmodels/tsa/regime_switching/markov_switching.py b/statsmodels/tsa/regime_switching/markov_switching.py
--- a/statsmodels/tsa/regime_switching/markov_switching.py
+++ b/statsmodels/tsa/regime_switching/markov_switching.py
@@ -110,11 +110,18 @@ def py_hamilton_filter(initial_probabilities, regime_transition,
     Parameters
     ----------
     initial_probabilities : array
-        Array of initial probabilities, shaped (k_regimes,).
+        Array of initial probabilities, shaped (k_regimes,) giving the
+        distribution of the regime process at time t = -order where order
+        is a nonnegative integer.
     regime_transition : array
         Matrix of regime transition probabilities, shaped either
         (k_regimes, k_regimes, 1) or if there are time-varying transition
-        probabilities (k_regimes, k_regimes, nobs).
+        probabilities (k_regimes, k_regimes, nobs + order).  Entry [i, j,
+        t] contains the probability of moving from j at time t-1 to i at
+        time t, so each matrix regime_transition[:, :, t] should be left
+        stochastic.  The first order entries and initial_probabilities are
+        used to produce the initial joint distribution of dimension order +
+        1 at time t=0.
     conditional_likelihoods : array
         Array of likelihoods conditional on the last `order+1` regimes,
         shaped (k_regimes,)*(order + 1) + (nobs,).
@@ -136,6 +143,7 @@ def py_hamilton_filter(initial_probabilities, regime_transition,
         the joint probability of the current and previous `order` periods
         being in each combination of regimes conditional on time t
         information. Shaped (k_regimes,) * (order + 1) + (nobs,).
+
     """
 
     # Dimensions
@@ -144,6 +152,14 @@ def py_hamilton_filter(initial_probabilities, regime_transition,
     order = conditional_likelihoods.ndim - 2
     dtype = conditional_likelihoods.dtype
 
+    # Check for compatible shapes.
+    incompatible_shapes = (
+        regime_transition.shape[-1] not in (1, nobs + order)
+        or regime_transition.shape[:2] != (k_regimes, k_regimes)
+        or conditional_likelihoods.shape[0] != k_regimes)
+    if incompatible_shapes:
+        raise ValueError('Arguments do not have compatible shapes')
+
     # Storage
     # Pr[S_t = s_t | Y_t]
     filtered_marginal_probabilities = (
@@ -165,6 +181,11 @@ def py_hamilton_filter(initial_probabilities, regime_transition,
         tmp = np.reshape(regime_transition[..., i], shape + (1,) * i) * tmp
     filtered_joint_probabilities[..., 0] = tmp
 
+    # Check that regime_transition is oriented correctly.
+    if not np.allclose(np.sum(regime_transition, axis=0), 1):
+        raise ValueError('regime_transition does not contain '
+                         'left stochastic matrices.')
+
     # Reshape regime_transition so we can use broadcasting
     shape = (k_regimes, k_regimes)
     shape += (1,) * (order-1)
@@ -182,11 +203,16 @@ def py_hamilton_filter(initial_probabilities, regime_transition,
             transition_t = t
 
         # S_t, S_{t-1}, ..., S_{t-r} | t-1, stored at zero-indexed location t
-        predicted_joint_probabilities[..., t] = (
-            # S_t | S_{t-1}
-            regime_transition[..., transition_t] *
-            # S_{t-1}, S_{t-2}, ..., S_{t-r} | t-1
-            filtered_joint_probabilities[..., t].sum(axis=-1))
+        if order > 0:
+            predicted_joint_probabilities[..., t] = (
+                # S_t | S_{t-1}
+                regime_transition[..., transition_t] *
+                # S_{t-1}, S_{t-2}, ..., S_{t-r} | t-1
+                filtered_joint_probabilities[..., t].sum(axis=-1))
+        else:
+            predicted_joint_probabilities[..., t] = (
+                np.dot(regime_transition[..., transition_t],
+                       filtered_joint_probabilities[..., t]))
 
         # f(y_t, S_t, ..., S_{t-r} | t-1)
         tmp = (conditional_likelihoods[..., t] *
@@ -216,11 +242,18 @@ def cy_hamilton_filter(initial_probabilities, regime_transition,
     Parameters
     ----------
     initial_probabilities : array
-        Array of initial probabilities, shaped (k_regimes,).
+        Array of initial probabilities, shaped (k_regimes,) giving the
+        distribution of the regime process at time t = -order where order
+        is a nonnegative integer.
     regime_transition : array
         Matrix of regime transition probabilities, shaped either
         (k_regimes, k_regimes, 1) or if there are time-varying transition
-        probabilities (k_regimes, k_regimes, nobs).
+        probabilities (k_regimes, k_regimes, nobs + order).  Entry [i, j,
+        t] contains the probability of moving from j at time t-1 to i at
+        time t, so each matrix regime_transition[:, :, t] should be left
+        stochastic.  The first order entries and initial_probabilities are
+        used to produce the initial joint distribution of dimension order +
+        1 at time t=0.
     conditional_likelihoods : array
         Array of likelihoods conditional on the last `order+1` regimes,
         shaped (k_regimes,)*(order + 1) + (nobs,).
@@ -250,6 +283,14 @@ def cy_hamilton_filter(initial_probabilities, regime_transition,
     order = conditional_likelihoods.ndim - 2
     dtype = conditional_likelihoods.dtype
 
+    # Check for compatible shapes.
+    incompatible_shapes = (
+        regime_transition.shape[-1] not in (1, nobs + order)
+        or regime_transition.shape[:2] != (k_regimes, k_regimes)
+        or conditional_likelihoods.shape[0] != k_regimes)
+    if incompatible_shapes:
+        raise ValueError('Arguments do not have compatible shapes')
+
     # Storage
     # Pr[S_t = s_t | Y_t]
     filtered_marginal_probabilities = (


Test patch:
diff --git a/statsmodels/tsa/regime_switching/tests/test_markov_regression.py b/statsmodels/tsa/regime_switching/tests/test_markov_regression.py
--- a/statsmodels/tsa/regime_switching/tests/test_markov_regression.py
+++ b/statsmodels/tsa/regime_switching/tests/test_markov_regression.py
@@ -12,7 +12,7 @@
 import pandas as pd
 from statsmodels.tsa.regime_switching import (markov_switching,
                                               markov_regression)
-from numpy.testing import assert_allclose
+from numpy.testing import assert_allclose, assert_raises
 
 
 current_path = os.path.dirname(os.path.abspath(__file__))
@@ -939,6 +939,102 @@ def test_smoother_output(self, **kwargs):
         assert_allclose(res.smoothed_joint_probabilities,
                         fedfunds_const_short_smoothed_joint_probabilities)
 
+    def test_hamilton_filter_order_zero(self):
+        k_regimes = 3
+        nobs = 4
+        initial_probabilities = np.ones(k_regimes) / k_regimes
+
+        # We don't actually transition between the 3 regimes.
+        regime_transition = np.eye(k_regimes)[:, :, np.newaxis]
+
+        # Regime i correponds to a sequence of iid draws from discrete
+        # random variables that are equally likely to be -1 and i for i=0,
+        # 1, 2.  We observe the sequence -1, -1, 1, -1.  The first two
+        # observations tell us nothing, but the third lets us know that we
+        # are in regime 1 the whole time.
+        conditional_likelihoods = np.ones((k_regimes, nobs)) / 2
+        conditional_likelihoods[:, 2] = [0, 1, 0]
+
+        expected_marginals = np.empty((k_regimes, nobs))
+        expected_marginals[:, :2] = [[1/3], [1/3], [1/3]]
+        expected_marginals[:, 2:] = [[0], [1], [0]]
+
+
+        py_results = markov_switching.py_hamilton_filter(
+            initial_probabilities, regime_transition, conditional_likelihoods)
+        assert_allclose(py_results[0], expected_marginals)
+
+        cy_results = markov_switching.cy_hamilton_filter(
+            initial_probabilities, regime_transition, conditional_likelihoods)
+        assert_allclose(cy_results[0], expected_marginals)
+
+    def test_hamilton_filter_order_zero_with_tvtp(self):
+        k_regimes = 3
+        nobs = 8
+        initial_probabilities = np.ones(k_regimes) / k_regimes
+
+        # We don't actually transition between the 3 regimes except from
+        # t=3 to t=4 where we reset to regimes 1 and 2 being equally
+        # likely.
+        regime_transition = np.zeros((k_regimes, k_regimes, nobs))
+        regime_transition[...] = np.eye(k_regimes)[:, :, np.newaxis]
+        regime_transition[..., 4] = [[0,     0,   0],
+                                     [1/2, 1/2, 1/2],
+                                     [1/2, 1/2, 1/2]]
+
+        # Regime i correponds to a sequence of iid draws from discrete
+        # random variables that are equally likely to be -1, i, and i + 1
+        # for i = 0, 1, 2.  We observe the sequence:
+        #
+        #     t =  0, 1, 2,  3,  4, 5, 6,  7
+        #   X_t = -1, 1, 2, -1, -1, 2, 3, -1
+        #
+        # The first observations tell us nothing, the second tells us that
+        # regime 0 and 1 are equally likely, and the third tells us that we
+        # must be in regime 1 for t = 0, 1, 2, 3.  At t=4 we transition to
+        # state 1 or 2.  In this case, neither a -1 or 2 changes our view
+        # that these states are equally likely, but a 3 tells we must be in
+        # state 2 for the second four timestamps.
+        conditional_likelihoods = np.empty((k_regimes, nobs))
+        conditional_likelihoods[:, 0] = [1/3, 1/3, 1/3]
+        conditional_likelihoods[:, 1] = [1/3, 1/3, 0]
+        conditional_likelihoods[:, 2] = [0, 1/3, 1/3]
+        conditional_likelihoods[:, 3:5] = [[1/3], [1/3], [1/3]]
+        conditional_likelihoods[:, 5] = [0, 1/3, 1/3]
+        conditional_likelihoods[:, 6] = [0, 0, 1/3]
+        conditional_likelihoods[:, 7] = [1/3, 1/3, 1/3]
+
+        expected_marginals = np.empty((k_regimes, nobs))
+        expected_marginals[:, 0] = [1/3, 1/3, 1/3]
+        expected_marginals[:, 1] = [1/2, 1/2, 0]
+        expected_marginals[:, 2:4] = [[0], [1], [0]]
+        expected_marginals[:, 4:6] = [[0], [1/2], [1/2]]
+        expected_marginals[:, 6:8] = [[0], [0], [1]]
+
+        py_results = markov_switching.py_hamilton_filter(
+            initial_probabilities, regime_transition, conditional_likelihoods)
+        assert_allclose(py_results[0], expected_marginals)
+
+        cy_results = markov_switching.cy_hamilton_filter(
+            initial_probabilities, regime_transition, conditional_likelihoods)
+        assert_allclose(cy_results[0], expected_marginals)
+
+    def test_hamilton_filter_shape_checks(self):
+        k_regimes = 3
+        nobs = 8
+        order = 3
+
+        initial_probabilities = np.ones(k_regimes) / k_regimes
+        regime_transition = np.ones((k_regimes, k_regimes, nobs)) / k_regimes
+        conditional_likelihoods = np.ones(order * (k_regimes,) + (nobs,))
+
+        for func in [markov_switching.py_hamilton_filter,
+                     markov_switching.cy_hamilton_filter]:
+            with assert_raises(ValueError):
+                func(initial_probabilities,
+                     regime_transition,
+                     conditional_likelihoods)
+
     def test_py_hamilton_filter(self):
         mod = self.model
         params = self.true['params']