Repository: statsmodels/statsmodels

Issue:
ENH: detrend in statsmodels.tsa.tsatools does not work with pandas. and additional enhancements
Just tried this function and had some problems.

Require the following fixes:
*Output not working well with pandas

Enhancements:
*dampening factor
add to specify the rate of decay in a trend. Default is 1 = no dampening factor. Reason: more accurate modeling

*additional trend types exponential (e^x) and logarithmic. 
Reason: better for growth models and population models

*output option for trend and residuals
Reason: useful for forecasting and pattern detection.



Discussion:
see also #1306 

these functions were initially written as "internal" code in support of other functions and models. Of the functions in `tsatools` only add_trend tries to handle some data structures (rec arrays, but doesn't look updated to handle pandas like add_constant). All the other ones, don't have error checking, data/array conversion, ...

We need to think about how to expose them for "public" use.

Except for the import of add_constant, tsatools is currently a pure numpy (import) module.

@ccsv the main question for your extensions is whether we do this with a full OLS instance or not.

I started something like numpy's polyfit as a convenience function that just uses OLS, but predefines the design matrix, `exog`. I intended to use it just as an example. 
However, we can add them to statsmodels. I haven't seen much interest articulated for this, so it always dropped off my priority list. But it would be useful to have convenient special use functions/models.

The other group of functions are targeted for internal use or similar use cases. I just wrote a few similar ones again. These functions are supposed to have minimal overhead and minimal unnecessary calculations because their main use is inside other models or functions where error checking and array/data structure conversion has already be done.

aside: we don't use much leading underscores for "private" functions. I don't consider them really as private, but some of them are single-purpose functions.

@josef-pkt It is difficult to find something that does trend decomposition in python and people are using the STL function (which hopefully will be available some time) and decomposition in R. Right now R is relying on Fortran code for STL which is not desired so far what I understand is that it has two loops with updating similar to Holt Winters Exponential smoothing so if this is implemented Cython should be used.


Resolving patch:
diff --git a/statsmodels/tools/data.py b/statsmodels/tools/data.py
--- a/statsmodels/tools/data.py
+++ b/statsmodels/tools/data.py
@@ -113,3 +113,10 @@ def _is_using_patsy(endog, exog):
     # we get this when a structured array is passed through a formula
     return (is_design_matrix(endog) and
             (is_design_matrix(exog) or exog is None))
+
+
+def _is_recarray(data):
+    """
+    Returns true if data is a recarray
+    """
+    return isinstance(data, np.core.recarray)
diff --git a/statsmodels/tools/tools.py b/statsmodels/tools/tools.py
--- a/statsmodels/tools/tools.py
+++ b/statsmodels/tools/tools.py
@@ -7,9 +7,8 @@
 import numpy.linalg as L
 from scipy.linalg import svdvals
 from statsmodels.datasets import webuse
-from statsmodels.tools.data import _is_using_pandas
+from statsmodels.tools.data import _is_using_pandas, _is_recarray
 from statsmodels.compat.numpy import np_matrix_rank
-from pandas import DataFrame
 
 
 def _make_dictnames(tmp_arr, offset=0):
@@ -233,108 +232,57 @@ def categorical(data, col=None, dictnames=False, drop=False, ):
             raise IndexError("The index %s is not understood" % col)
 
 
-def _series_add_constant(data, prepend, has_constant):
-    const = np.ones_like(data)
-    if data.var() == 0:
-        if has_constant == 'raise':
-            raise ValueError("data already contains a constant.")
-        elif has_constant == 'skip':
-            return data
-        elif has_constant == 'add':
-            pass
-        else:
-            raise ValueError("Option {0} not understood for "
-                             "has_constant.".format(has_constant))
-    if not prepend:
-        columns = [data.name, 'const']
-    else:
-        columns = ['const', data.name]
-    results = DataFrame({data.name : data, 'const' : const}, columns=columns)
-    return results
-
-
-def _dataframe_add_constant(data, prepend, has_constant):
-    # check for const.
-    if np.any(data.var(0) == 0):
-        if has_constant == 'raise':
-            raise ValueError("data already contains a constant.")
-        elif has_constant == 'skip':
-            return data
-        elif has_constant == 'add':
-            pass
-        else:
-            raise ValueError("Option {0} not understood for "
-                             "has_constant.".format(has_constant))
-    if prepend:
-        data.insert(0, 'const', 1)
-    else:
-        data['const'] = 1
-    return data
-
-
-def _pandas_add_constant(data, prepend, has_constant):
-    from pandas import Series
-    if isinstance(data, Series):
-        return _series_add_constant(data, prepend, has_constant)
-    else:
-        return _dataframe_add_constant(data, prepend, has_constant)
-
-
 # TODO: add an axis argument to this for sysreg
 def add_constant(data, prepend=True, has_constant='skip'):
     """
-    This appends a column of ones to an array if prepend==False.
+    Adds a column of ones to an array
 
     Parameters
     ----------
     data : array-like
         `data` is the column-ordered design matrix
     prepend : bool
-        True and the constant is prepended rather than appended.
+        If true, the constant is in the first column.  Else the constant is
+        appended (last column).
     has_constant : str {'raise', 'add', 'skip'}
-        Behavior if `data` already has a constant. The default will return
+        Behavior if ``data'' already has a constant. The default will return
         data without adding another constant. If 'raise', will raise an
         error if a constant is present. Using 'add' will duplicate the
-        constant, if one is present. Has no effect for structured or
-        recarrays. There is no checking for a constant in this case.
+        constant, if one is present.
 
     Returns
     -------
-    data : array
-        The original array with a constant (column of ones) as the first or
-        last column.
+    data : array, recarray or DataFrame
+        The original values with a constant (column of ones) as the first or
+        last column. Returned value depends on input type.
+
+    Notes
+    -----
+    When the input is recarray or a pandas Series or DataFrame, the added
+    column's name is 'const'.
     """
-    if _is_using_pandas(data, None):
-        # work on a copy
-        return _pandas_add_constant(data.copy(), prepend, has_constant)
-    else:
-        data = np.asarray(data)
-    if not data.dtype.names:
-        var0 = data.var(0) == 0
-        if np.any(var0):
-            if has_constant == 'raise':
-                raise ValueError("data already contains a constant.")
-            elif has_constant == 'skip':
-                return data
-            elif has_constant == 'add':
-                pass
-            else:
-                raise ValueError("Option {0} not understood for "
-                                 "has_constant.".format(has_constant))
-        data = np.column_stack((data, np.ones((data.shape[0], 1))))
-        if prepend:
-            return np.roll(data, 1, 1)
-    else:
-        return_rec = data.__class__ is np.recarray
-        if prepend:
-            ones = np.ones((data.shape[0], 1), dtype=[('const', float)])
-            data = nprf.append_fields(ones, data.dtype.names,
-                                      [data[i] for i in data.dtype.names],
-                                      usemask=False, asrecarray=return_rec)
-        else:
-            data = nprf.append_fields(data, 'const', np.ones(data.shape[0]),
-                                      usemask=False, asrecarray=return_rec)
-    return data
+    if _is_using_pandas(data, None) or _is_recarray(data):
+        from statsmodels.tsa.tsatools import add_trend
+        return add_trend(data, trend='c', prepend=prepend, has_constant=has_constant)
+
+    # Special case for NumPy
+    x = np.asanyarray(data)
+    if x.ndim == 1:
+        x = x[:,None]
+    elif x.ndim > 2:
+        raise ValueError('Only implementd 2-dimensional arrays')
+
+    is_nonzero_const = np.ptp(x, axis=0) == 0
+    is_nonzero_const &= np.all(x != 0.0, axis=0)
+    if is_nonzero_const.any():
+        if has_constant == 'skip':
+            return x
+        elif has_constant == 'raise':
+            raise ValueError("data already contains a constant")
+
+    x = [np.ones(x.shape[0]), x]
+    x = x if prepend else x[::-1]
+    return np.column_stack(x)
 
 
 def isestimable(C, D):
diff --git a/statsmodels/tsa/tsatools.py b/statsmodels/tsa/tsatools.py
--- a/statsmodels/tsa/tsatools.py
+++ b/statsmodels/tsa/tsatools.py
@@ -2,16 +2,16 @@
 
 import numpy as np
 import numpy.lib.recfunctions as nprf
+import pandas as pd
 from pandas import DataFrame
 from pandas.tseries import offsets
 from pandas.tseries.frequencies import to_offset
 
-from statsmodels.tools.tools import add_constant
-from statsmodels.tools.data import _is_using_pandas
 from statsmodels.tools.sm_exceptions import ValueWarning
+from statsmodels.tools.data import _is_using_pandas, _is_recarray
 
 
-def add_trend(X, trend="c", prepend=False, has_constant='skip'):
+def add_trend(x, trend="c", prepend=False, has_constant='skip'):
     """
     Adds a trend and/or constant to an array.
 
@@ -32,6 +32,13 @@ def add_trend(X, trend="c", prepend=False, has_constant='skip'):
         constant. 'skip' will return the data without change. 'skip' is the
         default.
 
+    Returns
+    -------
+    y : array, recarray or DataFrame
+        The original data with the additional trend columns.  If x is a
+        recarray or pandas Series or DataFrame, then the trend column names
+        are 'const', 'trend' and 'trend_squared'.
+
     Notes
     -----
     Returns columns as ["ctt","ct","c"] whenever applicable. There is currently
@@ -39,57 +46,81 @@ def add_trend(X, trend="c", prepend=False, has_constant='skip'):
 
     See also
     --------
-    statsmodels.add_constant
+    statsmodels.tools.add_constant
     """
-    #TODO: could be generalized for trend of aribitrary order
+    # TODO: could be generalized for trend of aribitrary order
     trend = trend.lower()
-    if trend == "c":    # handles structured arrays
-        return add_constant(X, prepend=prepend, has_constant=has_constant)
+    columns = ['const', 'trend', 'trend_squared']
+    if trend == "c":  # handles structured arrays
+        columns = columns[:1]
+        trendorder = 0
     elif trend == "ct" or trend == "t":
+        columns = columns[:2]
+        if trend == "t":
+            columns = columns[1:2]
         trendorder = 1
     elif trend == "ctt":
         trendorder = 2
     else:
         raise ValueError("trend %s not understood" % trend)
 
-    X = np.asanyarray(X)
-    nobs = len(X)
-    trendarr = np.vander(np.arange(1,nobs+1, dtype=float), trendorder+1)
+    is_recarray = _is_recarray(x)
+    is_pandas = _is_using_pandas(x, None) or is_recarray
+    if is_pandas or is_recarray:
+        if is_recarray:
+            descr = x.dtype.descr
+            x = pd.DataFrame.from_records(x)
+        elif isinstance(x, pd.Series):
+            x = pd.DataFrame(x)
+        else:
+            x = x.copy()
+    else:
+        x = np.asanyarray(x)
+
+    nobs = len(x)
+    trendarr = np.vander(np.arange(1, nobs + 1, dtype=np.float64), trendorder + 1)
     # put in order ctt
     trendarr = np.fliplr(trendarr)
     if trend == "t":
-        trendarr = trendarr[:,1]
-    if not X.dtype.names:
-        # check for constant
-        if "c" in trend and np.any(np.ptp(X, axis=0) == 0):
-            if has_constant == 'raise':
-                raise ValueError("X already contains a constant")
-            elif has_constant == 'add':
-                pass
-            elif has_constant == 'skip' and trend == "ct":
-                trendarr = trendarr[:, 1]
-
-        if not prepend:
-            X = np.column_stack((X, trendarr))
+        trendarr = trendarr[:, 1]
+
+    if "c" in trend:
+        if is_pandas or is_recarray:
+            # Mixed type protection
+            def safe_is_const(s):
+                try:
+                    return np.ptp(s) == 0.0 and np.any(s != 0.0)
+                except:
+                    return False
+            col_const = x.apply(safe_is_const, 0)
         else:
-            X = np.column_stack((trendarr, X))
+            col_const = np.logical_and(np.any(np.ptp(np.asanyarray(x), axis=0) == 0, axis=0),
+                                       np.all(x != 0.0, axis=0))
+        if np.any(col_const):
+            if has_constant == 'raise':
+                raise ValueError("x already contains a constant")
+            elif has_constant == 'skip':
+                columns = columns[1:]
+                trendarr = trendarr[:, 1:]
+
+    order = 1 if prepend else -1
+    if is_recarray or is_pandas:
+        trendarr = pd.DataFrame(trendarr, index=x.index, columns=columns)
+        x = [trendarr, x]
+        x = pd.concat(x[::order], 1)
     else:
-        return_rec = data.__clas__ is np.recarray
-        if trendorder == 1:
-            if trend == "ct":
-                dt = [('const',float),('trend',float)]
-            else:
-                dt = [('trend', float)]
-        elif trendorder == 2:
-            dt = [('const',float),('trend',float),('trend_squared', float)]
-        trendarr = trendarr.view(dt)
-        if prepend:
-            X = nprf.append_fields(trendarr, X.dtype.names, [X[i] for i
-                in X.dtype.names], usemask=False, asrecarray=return_rec)
-        else:
-            X = nprf.append_fields(X, trendarr.dtype.names, [trendarr[i] for i
-                in trendarr.dtype.names], usemask=False, asrecarray=return_rec)
-    return X
+        x = [trendarr, x]
+        x = np.column_stack(x[::order])
+
+    if is_recarray:
+        x = x.to_records(index=False, convert_datetime64=False)
+        new_descr = x.dtype.descr
+        extra_col = len(new_descr) - len(descr)
+        descr = new_descr[:extra_col] + descr if prepend else descr + new_descr[-extra_col:]
+        x = x.astype(np.dtype(descr))
+
+    return x
+
 
 def add_lag(x, col=None, lags=1, drop=False, insert=True):
     """
@@ -219,8 +250,10 @@ def add_lag(x, col=None, lags=1, drop=False, insert=True):
         return np.column_stack((x[lags:,first_cols],ndlags,
                     x[lags:,last_cols]))
 
+
 def detrend(x, order=1, axis=0):
-    '''detrend an array with a trend of given order along axis 0 or 1
+    """
+    Detrend an array with a trend of given order along axis 0 or 1
 
     Parameters
     ----------
@@ -239,25 +272,25 @@ def detrend(x, order=1, axis=0):
     detrended data series : ndarray
         The detrended series is the residual of the linear regression of the
         data on the trend of given order.
+    """
+    if x.ndim == 2 and int(axis) == 1:
+        x = x.T
+    elif x.ndim > 2:
+        raise NotImplementedError('x.ndim > 2 is not implemented until it is needed')
 
-
-    '''
-    x = np.asarray(x)
     nobs = x.shape[0]
     if order == 0:
-        return x - np.expand_dims(x.mean(axis), axis)
+        # Special case demean
+        resid = x - x.mean(axis=0)
     else:
-        if x.ndim == 2 and lrange(2)[axis]==1:
-            x = x.T
-        elif x.ndim > 2:
-            raise NotImplementedError('x.ndim>2 is not implemented until it is needed')
-        #could use a polynomial, but this should work also with 2d x, but maybe not yet
-        trends = np.vander(np.arange(nobs).astype(float), N=order+1)
-        beta = np.linalg.lstsq(trends, x)[0]
+        trends = np.vander(np.arange(float(nobs)), N=order + 1)
+        beta = np.linalg.pinv(trends).dot(x)
         resid = x - np.dot(trends, beta)
-        if x.ndim == 2 and lrange(2)[axis]==1:
-            resid = resid.T
-        return resid
+
+    if x.ndim == 2 and int(axis) == 1:
+        resid = resid.T
+
+    return resid
 
 
 def lagmat(x, maxlag, trim='forward', original='ex', use_pandas=False):


Test patch:
diff --git a/statsmodels/tools/tests/test_tools.py b/statsmodels/tools/tests/test_tools.py
--- a/statsmodels/tools/tests/test_tools.py
+++ b/statsmodels/tools/tests/test_tools.py
@@ -7,6 +7,8 @@
 from numpy.testing import (assert_equal, assert_array_equal,
                            assert_almost_equal, assert_string_equal, TestCase)
 from nose.tools import (assert_true, assert_false, assert_raises)
+import pandas as pd
+from pandas.util.testing import assert_frame_equal, assert_series_equal
 
 from statsmodels.datasets import longley
 from statsmodels.tools import tools
@@ -31,7 +33,7 @@ def test_add_constant_1d(self):
     def test_add_constant_has_constant1d(self):
         x = np.ones(5)
         x = tools.add_constant(x, has_constant='skip')
-        assert_equal(x, np.ones(5))
+        assert_equal(x, np.ones((5,1)))
 
         assert_raises(ValueError, tools.add_constant, x, has_constant='raise')
 
@@ -48,6 +50,54 @@ def test_add_constant_has_constant2d(self):
         assert_equal(tools.add_constant(x, has_constant='add'),
                      np.column_stack((np.ones(4), x)))
 
+    def test_add_constant_recarray(self):
+        dt = np.dtype([('', int), ('', '<S4'), ('', np.float32), ('', np.float64)])
+        x = np.array([(1, 'abcd', 1.0, 2.0),
+                      (7, 'abcd', 2.0, 4.0),
+                      (21, 'abcd', 2.0, 8.0)], dt)
+        x = x.view(np.recarray)
+        y = tools.add_constant(x)
+        assert_equal(y['const'],np.array([1.0,1.0,1.0]))
+        for f in x.dtype.fields:
+            assert_true(y[f].dtype == x[f].dtype)
+
+    def test_add_constant_series(self):
+        s = pd.Series([1.0,2.0,3.0])
+        output = tools.add_constant(s)
+        expected = pd.Series([1.0,1.0,1.0],name='const')
+        assert_series_equal(expected, output['const'])
+
+    def test_add_constant_dataframe(self):
+        df = pd.DataFrame([[1.0, 'a', 4], [2.0, 'bc', 9], [3.0, 'def', 16]])
+        output = tools.add_constant(df)
+        expected = pd.Series([1.0, 1.0, 1.0], name='const')
+        assert_series_equal(expected, output['const'])
+        dfc = df.copy()
+        dfc.insert(0, 'const', np.ones(3))
+        assert_frame_equal(dfc, output)
+
+    def test_add_constant_zeros(self):
+        a = np.zeros(100)
+        output = tools.add_constant(a)
+        assert_equal(output[:,0],np.ones(100))
+
+        s = pd.Series([0.0,0.0,0.0])
+        output = tools.add_constant(s)
+        expected = pd.Series([1.0, 1.0, 1.0], name='const')
+        assert_series_equal(expected, output['const'])
+
+        df = pd.DataFrame([[0.0, 'a', 4], [0.0, 'bc', 9], [0.0, 'def', 16]])
+        output = tools.add_constant(df)
+        dfc = df.copy()
+        dfc.insert(0, 'const', np.ones(3))
+        assert_frame_equal(dfc, output)
+
+        df = pd.DataFrame([[1.0, 'a', 0], [0.0, 'bc', 0], [0.0, 'def', 0]])
+        output = tools.add_constant(df)
+        dfc = df.copy()
+        dfc.insert(0, 'const', np.ones(3))
+        assert_frame_equal(dfc, output)
+
     def test_recipr(self):
         X = np.array([[2,1],[-1,0]])
         Y = tools.recipr(X)
diff --git a/statsmodels/tsa/tests/test_tsa_tools.py b/statsmodels/tsa/tests/test_tsa_tools.py
--- a/statsmodels/tsa/tests/test_tsa_tools.py
+++ b/statsmodels/tsa/tests/test_tsa_tools.py
@@ -2,14 +2,12 @@
 
 '''
 import unittest
-from nose.tools import assert_raises
-
 from statsmodels.compat.python import zip
 
-import pandas as pd
-from pandas.util.testing import assert_frame_equal
 import numpy as np
-from numpy.testing import assert_array_almost_equal, assert_equal
+from numpy.testing import assert_array_almost_equal, assert_equal, assert_raises
+import pandas as pd
+from pandas.util.testing import assert_frame_equal, assert_series_equal
 
 import statsmodels.api as sm
 import statsmodels.tsa.stattools as tsa
@@ -376,20 +374,179 @@ def test_freq_to_period():
         assert_equal(tools.freq_to_period(to_offset(i)), j)
 
 
-def test_detrend():
-    data = np.arange(5)
-    assert_array_almost_equal(sm.tsa.detrend(data, order=1),
-                              np.zeros_like(data))
-    assert_array_almost_equal(sm.tsa.detrend(data, order=0), [-2, -1, 0, 1, 2])
-    data = np.arange(10).reshape(5, 2)
-    assert_array_almost_equal(sm.tsa.detrend(data, order=1, axis=0),
-                              np.zeros_like(data))
-    assert_array_almost_equal(sm.tsa.detrend(data, order=0, axis=0),
-                              [[-4, -4], [-2, -2], [0, 0], [2, 2], [4, 4]])
-    assert_array_almost_equal(sm.tsa.detrend(data, order=0, axis=1),
-                              [[-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5],
-                               [-0.5, 0.5], [-0.5, 0.5]])
-
+class TestDetrend(unittest.TestCase):
+    @classmethod
+    def setup_class(cls):
+        cls.data_1d = np.arange(5.0)
+        cls.data_2d = np.arange(10.0).reshape(5, 2)
+
+    def test_detrend_1d(self):
+        data = self.data_1d
+        assert_array_almost_equal(sm.tsa.detrend(data, order=1), np.zeros_like(data))
+        assert_array_almost_equal(sm.tsa.detrend(data, order=0), [-2, -1, 0, 1, 2])
+
+    def test_detrend_2d(self):
+        data = self.data_2d
+        assert_array_almost_equal(sm.tsa.detrend(data, order=1, axis=0), np.zeros_like(data))
+        assert_array_almost_equal(sm.tsa.detrend(data, order=0, axis=0), [[-4, -4], [-2, -2], [0, 0], [2, 2], [4, 4]])
+        assert_array_almost_equal(sm.tsa.detrend(data, order=0, axis=1),
+                                  [[-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5]])
+
+    def test_detrend_series(self):
+        data = pd.Series(self.data_1d, name='one')
+        detrended = sm.tsa.detrend(data, order=1)
+        assert_array_almost_equal(detrended.values, np.zeros_like(data))
+        assert_series_equal(detrended, pd.Series(detrended.values, name='one'))
+        detrended = sm.tsa.detrend(data, order=0)
+        assert_array_almost_equal(detrended.values, pd.Series([-2, -1, 0, 1, 2]))
+        assert_series_equal(detrended, pd.Series(detrended.values, name='one'))
+
+    def test_detrend_dataframe(self):
+        columns = ['one', 'two']
+        index = [c for c in 'abcde']
+        data = pd.DataFrame(self.data_2d, columns=columns, index=index)
+
+        detrended = sm.tsa.detrend(data, order=1, axis=0)
+        assert_array_almost_equal(detrended.values, np.zeros_like(data))
+        assert_frame_equal(detrended, pd.DataFrame(detrended.values, columns=columns, index=index))
+
+        detrended = sm.tsa.detrend(data, order=0, axis=0)
+        assert_array_almost_equal(detrended.values, [[-4, -4], [-2, -2], [0, 0], [2, 2], [4, 4]])
+        assert_frame_equal(detrended, pd.DataFrame(detrended.values, columns=columns, index=index))
+
+        detrended = sm.tsa.detrend(data, order=0, axis=1)
+        assert_array_almost_equal(detrended.values, [[-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5], [-0.5, 0.5]])
+        assert_frame_equal(detrended, pd.DataFrame(detrended.values, columns=columns, index=index))
+
+    def test_detrend_dim_too_large(self):
+        assert_raises(NotImplementedError, sm.tsa.detrend, np.ones((3, 3, 3)))
+
+class TestAddTrend(unittest.TestCase):
+    @classmethod
+    def setup_class(cls):
+        cls.n = 200
+        cls.arr_1d = np.arange(float(cls.n))
+        cls.arr_2d = np.tile(np.arange(float(cls.n))[:, None], 2)
+        cls.c = np.ones(cls.n)
+        cls.t = np.arange(1.0, cls.n + 1)
+
+    def test_series(self):
+        s = pd.Series(self.arr_1d)
+        appended = tools.add_trend(s)
+        expected = pd.DataFrame(s)
+        expected['const'] = self.c
+        assert_frame_equal(expected, appended)
+
+        prepended = tools.add_trend(s, prepend=True)
+        expected = pd.DataFrame(s)
+        expected.insert(0, 'const', self.c)
+        assert_frame_equal(expected, prepended)
+
+        s = pd.Series(self.arr_1d)
+        appended = tools.add_trend(s, trend='ct')
+        expected = pd.DataFrame(s)
+        expected['const'] = self.c
+        expected['trend'] = self.t
+        assert_frame_equal(expected, appended)
+
+    def test_dataframe(self):
+        df = pd.DataFrame(self.arr_2d)
+        appended = tools.add_trend(df)
+        expected = df.copy()
+        expected['const'] = self.c
+        assert_frame_equal(expected, appended)
+
+        prepended = tools.add_trend(df, prepend=True)
+        expected = df.copy()
+        expected.insert(0, 'const', self.c)
+        assert_frame_equal(expected, prepended)
+
+        df = pd.DataFrame(self.arr_2d)
+        appended = tools.add_trend(df, trend='t')
+        expected = df.copy()
+        expected['trend'] = self.t
+        assert_frame_equal(expected, appended)
+
+        df = pd.DataFrame(self.arr_2d)
+        appended = tools.add_trend(df, trend='ctt')
+        expected = df.copy()
+        expected['const'] = self.c
+        expected['trend'] = self.t
+        expected['trend_squared'] = self.t ** 2
+        assert_frame_equal(expected, appended)
+
+    def test_recarray(self):
+        recarray = pd.DataFrame(self.arr_2d).to_records(index=False, convert_datetime64=False)
+        appended = tools.add_trend(recarray)
+        expected = pd.DataFrame(self.arr_2d)
+        expected['const'] = self.c
+        expected = expected.to_records(index=False, convert_datetime64=False)
+        assert_equal(expected, appended)
+
+        prepended = tools.add_trend(recarray, prepend=True)
+        expected = pd.DataFrame(self.arr_2d)
+        expected.insert(0, 'const', self.c)
+        expected = expected.to_records(index=False, convert_datetime64=False)
+        assert_equal(expected, prepended)
+
+        appended = tools.add_trend(recarray, trend='ctt')
+        expected = pd.DataFrame(self.arr_2d)
+        expected['const'] = self.c
+        expected['trend'] = self.t
+        expected['trend_squared'] = self.t ** 2
+        expected = expected.to_records(index=False, convert_datetime64=False)
+        assert_equal(expected, appended)
+
+    def test_duplicate_const(self):
+        assert_raises(ValueError, tools.add_trend, x=self.c, trend='c', has_constant='raise')
+        assert_raises(ValueError, tools.add_trend, x=self.c, trend='ct', has_constant='raise')
+        df = pd.DataFrame(self.c)
+        assert_raises(ValueError, tools.add_trend, x=df, trend='c', has_constant='raise')
+        assert_raises(ValueError, tools.add_trend, x=df, trend='ct', has_constant='raise')
+
+        skipped = tools.add_trend(self.c, trend='c')
+        assert_equal(skipped, self.c[:,None])
+
+        skipped_const = tools.add_trend(self.c, trend='ct', has_constant='skip')
+        expected = np.vstack((self.c, self.t)).T
+        assert_equal(skipped_const, expected)
+
+        added = tools.add_trend(self.c, trend='c', has_constant='add')
+        expected = np.vstack((self.c, self.c)).T
+        assert_equal(added, expected)
+
+        added = tools.add_trend(self.c, trend='ct', has_constant='add')
+        expected = np.vstack((self.c, self.c, self.t)).T
+        assert_equal(added, expected)
+
+    def test_mixed_recarray(self):
+        dt = np.dtype([('c0', np.float64), ('c1', np.int8), ('c2', 'S4')])
+        ra = np.array([(1.0, 1, 'aaaa'), (1.1, 2, 'bbbb')], dtype=dt).view(np.recarray)
+        added = tools.add_trend(ra, trend='ct')
+        dt = np.dtype([('c0', np.float64), ('c1', np.int8), ('c2', 'S4'), ('const', np.float64), ('trend', np.float64)])
+        expected = np.array([(1.0, 1, 'aaaa', 1.0, 1.0), (1.1, 2, 'bbbb', 1.0, 2.0)], dtype=dt).view(np.recarray)
+        assert_equal(added, expected)
+
+    def test_dataframe_duplicate(self):
+        df = pd.DataFrame(self.arr_2d, columns=['const', 'trend'])
+        tools.add_trend(df, trend='ct')
+        tools.add_trend(df, trend='ct', prepend=True)
+
+    def test_array(self):
+        base = np.vstack((self.arr_1d, self.c, self.t, self.t ** 2)).T
+        assert_equal(tools.add_trend(self.arr_1d), base[:, :2])
+        assert_equal(tools.add_trend(self.arr_1d, trend='t'), base[:, [0, 2]])
+        assert_equal(tools.add_trend(self.arr_1d, trend='ct'), base[:, :3])
+        assert_equal(tools.add_trend(self.arr_1d, trend='ctt'), base)
+
+        base = np.hstack((self.c[:, None], self.t[:, None], self.t[:, None] ** 2, self.arr_2d))
+        assert_equal(tools.add_trend(self.arr_2d, prepend=True), base[:, [0, 3, 4]])
+        assert_equal(tools.add_trend(self.arr_2d, trend='t', prepend=True), base[:, [1, 3, 4]])
+        assert_equal(tools.add_trend(self.arr_2d, trend='ct', prepend=True), base[:, [0, 1, 3, 4]])
+        assert_equal(tools.add_trend(self.arr_2d, trend='ctt', prepend=True), base)
+
+    def test_unknown_trend(self):
+        assert_raises(ValueError, tools.add_trend, x=self.arr_1d, trend='unknown')
 
 if __name__ == '__main__':
     import nose