Repository: statsmodels/statsmodels

Issue:
LinAlgError using IV-2SLS 
When using IV2SLS from statsmodels.sandbox.regression.gmm I get a LinAlgError ("0-dimensional array given. Array must be at least two-dimensional") when giving 1D inputs.

A dummy way of triggering the error might be:

``` python
import numpy as np
from statsmodels.sandbox.regression.gmm import IV2SLS

IV2SLS(np.arange(10), np.arange(10), np.arange(10)).fit()
```

As it makes perfect sense to use 1D instrument, endo- and exo-geneous variables I think this is a bug.



Discussion:
It's likely that there is not enough shape checking (and automatic conversion to 2-d)

However, that's really an unusual case that I never thought about.
Note, we are not adding automatically a constant in the array interface.

So this would be one endogenous "reponse" variable, one endogenous regressor and one instrument, where neither of the two regression has a constant.

What is your example or application with this case?

The only problem should be in the first stage auxiliary regression, where we need to check the instrument array and convert 1-D to 2-D column if necessary.

Maybe I'm confused with the interface. In the first stage I'm ok to add an intercept to the regression, but I have only one (continuous) instrument variable. 

So not sure how to express that as whenever I put a 1D instrument this triggers the error:

``` python
IV2SLS(np.arange(10), sm.add_constant(np.arange(10)), np.arange(10)).fit()
```

Having a first stage without intercept may be a valid case too (but not for my application).

PS: I was able to compute a coefficient estimate using two OLS models, but now I want a correct confidence interval for the coefficient - hence testing the sandbox IV2SLS :)

I haven't checked the code yet.

But the following two versions would add a constant to the first stage regression, with or without a constant in the main regression

`IV2SLS(np.arange(10), sm.add_constant(np.arange(10)), sm.add_constant(np.arange(10))).fit()`
`IV2SLS(np.arange(10), np.arange(10), sm.add_constant(np.arange(10))).fit()`

Assuming this is just a shape, problem the following should also work, i.e. explicitly use a 2-D single-column instrument array: 

`IV2SLS(np.arange(10), sm.add_constant(np.arange(10)), np.arange(10)[:, None]).fit()`

As reminder: If you have exogenous regressors in the main equation, then they need to be added to the instruments. 

Aside: Unit test coverage for IV2SLS should be pretty good. The main problem are still some interface choices. So, if you have any comments on the API and usability, then I'd appreciate to hear them.

There is no special data handling beyond the default endog exog handling. Obviously I never went back to this because I only used numpy arrays in the examples and tests.

`missing` handling including instruments can now (>0.6 IIRC) also be delegated to super.
asarray and shape check for instruments need to be in the class itself.

```
__init__(self, endog, exog, instrument=None):
        self.instrument = instrument
        super(IV2SLS, self).__init__(endog, exog)
```

related: 
`from_formula` is inherited but needs to be overwritten to handle formulas for instruments. 
We now have also cases for that, AFAIK, most likely MixedLM.

---

one main API and internal decision is whether to split up the regressors into an endogenous and exogenous part, as Stata's ivreg and ivreg2 do. However, it makes inherited data handling more difficult, but that should be feasible now with statsmodels >=0.7 (?)
The split is irrelevant for the way it's currently calculated, but we don't have much in terms of IV diagnostics yet, especially with respect to possible weak instruments problems.

Ok, so in my case the system equations are (omitting noise):
- $x = \alpha_1 + \beta_1 z$
- $y = \alpha_2 + \beta_2 x$

So I ended with the following call: `IV2SLS(y, sm.add_constant(x), sm.add_constant(z)).fit()`

I think the API might be sensible, but the docstring could be improved (especially for people like me taming their first 2SLS in python). Including an example with the system equations and the corresponding IV2SLS calls would be a must :)

Apart from that point, I suppose that the return value is the result of the second regression - which is sensible again - given that the user is provided with 2 additional infos:
- correct confidence interval over the coefficients : I'm told it should differ from the naive OLS of the second stage since the regressor is an estimate (for now I'm not sure since in my case the two method give the same CI)
- the value of the F-statistic for the first stage : a rule of thumb is that it should be > 10 for a strong instrument
  Maybe both infos are already there but not documented/accessible through `summary()`, haven't checked it atm.

If it's not too involved and given proper guidance I might be able to provide a PR targeting an improved docstring (+shape checks ?). Let me know if it helps.

@oddskool   Thanks, yes the feedback helps, and PR would be welcome.

about docs: Yes, it's still lacking and improvements would be helpful. I started to work on that module again, but only on the GMM part. Adding or starting a notebook with an example would also be useful. (I started several on GMM)

about confidence interval: They **should** be correct, but I need to check test coverage if they are directly tested (confint are inherited). Are your bse and tvalues also the same between OLS and IV2SLS? I guess they should only be close to each other if you have an almost perfectly correlated instrument.

about info for first stage:  This is still completely open. Currently no information about the first stage is stored. One design question is when to calculate it and what to store. In general we try to calculate extra results only on demand, for example only when `summary` is called. However, in this case storing the entire first stage might use up more memory in large data cases.
I guess calculating R squared assuming the presence of a constant would be relatively straight forward and not computationally expensive. We should be able to calculate the F-statistic, f_values directly from it. (I guess it will be useful just a statistic and will not be a valid F-test under robust covariance assumptions, heteroscedasticity, correlation, ...)
I guess summary could be improved with more specific IV 2SLS information.
From my personal interest I targeted more IVGMM for enhancements than IV2SLS. 

The simplest solution for the exog check is to
- use asarray
- if instruments.ndim == 1: check length is the same as endog then reshape otherwise raise
- add instrument keyword to super `__init__` call


Resolving patch:
diff --git a/statsmodels/sandbox/regression/gmm.py b/statsmodels/sandbox/regression/gmm.py
--- a/statsmodels/sandbox/regression/gmm.py
+++ b/statsmodels/sandbox/regression/gmm.py
@@ -50,8 +50,11 @@
 
 from __future__ import print_function
 from statsmodels.compat.python import lrange
+from statsmodels.compat.numpy import np_matrix_rank
+
 import numpy as np
 from scipy import optimize, stats
+
 from statsmodels.tools.numdiff import approx_fprime, approx_hess
 from statsmodels.base.model import (Model,
                                     LikelihoodModel, LikelihoodModelResults)
@@ -59,7 +62,7 @@
                                                  RegressionResultsWrapper)
 import statsmodels.stats.sandwich_covariance as smcov
 from statsmodels.tools.decorators import (resettable_cache, cache_readonly)
-from statsmodels.compat.numpy import np_matrix_rank
+from statsmodels.tools.tools import _ensure_2d
 
 DEBUG = 0
 
@@ -70,38 +73,36 @@ def maxabs(x):
 
 
 class IV2SLS(LikelihoodModel):
-    '''
-    Class for instrumental variables estimation using Two-Stage Least-Squares
+    """
+    Instrumental variables estimation using Two-Stage Least-Squares (2SLS)
 
 
     Parameters
     ----------
-    endog: array 1d
-       endogenous variable
+    endog: array
+       Endogenous variable, 1-dimensional or 2-dimensional array nobs by 1
     exog : array
-       explanatory variables
+       Explanatory variables, 1-dimensional or 2-dimensional array nobs by k
     instruments : array
-       instruments for explanatory variables, needs to contain those exog
-       variables that are not instrumented out
+       Instruments for explanatory variables. Must contain both exog
+       variables that are not being instrumented and instruments
 
     Notes
     -----
     All variables in exog are instrumented in the calculations. If variables
-    in exog are not supposed to be instrumented out, then these variables
-    need also to be included in the instrument array.
+    in exog are not supposed to be instrumented, then these variables
+    must also to be included in the instrument array.
 
     Degrees of freedom in the calculation of the standard errors uses
     `df_resid = (nobs - k_vars)`.
     (This corresponds to the `small` option in Stata's ivreg2.)
-
-
-    '''
+    """
 
     def __init__(self, endog, exog, instrument=None):
-        self.instrument = instrument
+        self.instrument, self.instrument_names = _ensure_2d(instrument, True)
         super(IV2SLS, self).__init__(endog, exog)
         # where is this supposed to be handled
-        #Note: Greene p.77/78 dof correction is not necessary (because only
+        # Note: Greene p.77/78 dof correction is not necessary (because only
         #       asy results), but most packages do it anyway
         self.df_resid = self.exog.shape[0] - self.exog.shape[1]
         #self.df_model = float(self.rank - self.k_constant)
diff --git a/statsmodels/tools/tools.py b/statsmodels/tools/tools.py
--- a/statsmodels/tools/tools.py
+++ b/statsmodels/tools/tools.py
@@ -6,6 +6,8 @@
 import numpy.lib.recfunctions as nprf
 import numpy.linalg as L
 from scipy.linalg import svdvals
+import pandas as pd
+
 from statsmodels.datasets import webuse
 from statsmodels.tools.data import _is_using_pandas, _is_recarray
 from statsmodels.compat.numpy import np_matrix_rank
@@ -511,3 +513,46 @@ def __init__(self, **kw):
                       message='webuse will be removed from the tools '
                               'namespace in the 0.7.0 release. Please use the'
                               ' new import.')
+
+
+def _ensure_2d(x, ndarray=False):
+    """
+
+    Parameters
+    ----------
+    x : array, Series, DataFrame or None
+        Input to verify dimensions, and to transform as necesary
+    ndarray : bool
+        Flag indicating whether to always return a NumPy array. Setting False
+        will return an pandas DataFrame when the input is a Series or a
+        DataFrame.
+
+    Returns
+    -------
+    out : array, DataFrame or None
+        array or DataFrame with 2 dimensiona.  One dimensional arrays are
+        returned as nobs by 1. None is returned if x is None.
+    names : list of str or None
+        list containing variables names when the input is a pandas datatype.
+        Returns None if the input is an ndarray.
+
+    Notes
+    -----
+    Accepts None for simplicity
+    """
+    if x is None:
+        return x
+    is_pandas = _is_using_pandas(x, None)
+    if x.ndim == 2:
+        if is_pandas:
+            return x, x.columns
+        else:
+            return x, None
+    elif x.ndim > 2:
+        raise ValueError('x mst be 1 or 2-dimensional.')
+
+    name = x.name if is_pandas else None
+    if ndarray:
+        return np.asarray(x)[:, None], name
+    else:
+        return pd.DataFrame(x), name


Test patch:
diff --git a/statsmodels/sandbox/regression/tests/test_gmm.py b/statsmodels/sandbox/regression/tests/test_gmm.py
--- a/statsmodels/sandbox/regression/tests/test_gmm.py
+++ b/statsmodels/sandbox/regression/tests/test_gmm.py
@@ -7,8 +7,10 @@
 """
 from __future__ import print_function
 from statsmodels.compat.python import lrange, lmap
+
 import numpy as np
 from numpy.testing import assert_allclose, assert_almost_equal, assert_equal
+import pandas as pd
 
 from statsmodels import iolib
 from statsmodels.tools.tools import add_constant
@@ -671,7 +673,6 @@ def test_hausman(self):
         assert_allclose(hausm[0], res2.hausman['DWH'], rtol=1e-11, atol=0)
         assert_allclose(hausm[1], res2.hausman['DWHp'], rtol=1e-10, atol=1e-25)
 
-
     def test_smoke(self):
         res1 = self.res1
         res1.summary()
@@ -692,3 +693,51 @@ def setup_class(self):
 
         from .results_ivreg2_griliches import results_small as results
         self.res2 = results
+
+
+    # See GH #2720
+    def test_input_dimensions(self):
+        rs = np.random.RandomState(1234)
+        x = rs.randn(200, 2)
+        z = rs.randn(200)
+        x[:, 0] = np.sqrt(0.5) * x[:, 0] + np.sqrt(0.5) * z
+        z = np.column_stack((x[:, [1]], z[:, None]))
+        e = np.sqrt(0.5) * rs.randn(200) + np.sqrt(0.5) * x[:, 0]
+
+        y_1d = y = x[:, 0] + x[:, 1] + e
+        y_2d = y[:, None]
+        y_series = pd.Series(y)
+        y_df = pd.DataFrame(y_series)
+        x_1d = x[:, 0]
+        x_2d = x
+        x_df = pd.DataFrame(x)
+        x_df_single = x_df.iloc[:, [0]]
+        x_series = x_df.iloc[:, 0]
+        z_2d = z
+        z_series = pd.Series(z[:, 1])
+        z_1d = z_series.values
+        z_df = pd.DataFrame(z)
+
+        ys = (y_df, y_series, y_2d, y_1d)
+        xs = (x_2d, x_1d, x_df_single, x_df, x_series)
+        zs = (z_1d, z_2d, z_series, z_df)
+        res2 = gmm.IV2SLS(y_1d, x_2d, z_2d).fit()
+        res1 = gmm.IV2SLS(y_1d, x_1d, z_1d).fit()
+        res1_2sintr = gmm.IV2SLS(y_1d, x_1d, z_2d).fit()
+
+
+        for _y in ys:
+            for _x in xs:
+                for _z in zs:
+                    x_1d = np.size(_x) == _x.shape[0]
+                    z_1d = np.size(_z) == _z.shape[0]
+                    if z_1d and not x_1d:
+                        continue
+                    res = gmm.IV2SLS(_y, _x, _z).fit()
+                    if z_1d:
+                        assert_allclose(res.params, res1.params)
+                    elif x_1d and not z_1d:
+                        assert_allclose(res.params, res1_2sintr.params)
+                    else:
+                        assert_allclose(res.params, res2.params)
+
diff --git a/statsmodels/tools/tests/test_tools.py b/statsmodels/tools/tests/test_tools.py
--- a/statsmodels/tools/tests/test_tools.py
+++ b/statsmodels/tools/tests/test_tools.py
@@ -546,3 +546,37 @@ def test_66(self):
         test_res = tools.nan_dot(self.mx_6, self.mx_6)
         expected_res = np.array([[  7.,  10.], [ 15.,  22.]])
         assert_array_equal(test_res, expected_res)
+
+class TestEnsure2d(TestCase):
+    @classmethod
+    def setUpClass(cls):
+        x = np.arange(400.0).reshape((100,4))
+        cls.df = pd.DataFrame(x, columns = ['a','b','c','d'])
+        cls.series = cls.df.iloc[:,0]
+        cls.ndarray = x
+
+    def test_enfore_numpy(self):
+        results = tools._ensure_2d(self.df, True)
+        assert_array_equal(results[0], self.ndarray)
+        assert_array_equal(results[1], self.df.columns)
+        results = tools._ensure_2d(self.series, True)
+        assert_array_equal(results[0], self.ndarray[:,[0]])
+        assert_array_equal(results[1], self.df.columns[0])
+
+    def test_pandas(self):
+        results = tools._ensure_2d(self.df, False)
+        assert_frame_equal(results[0], self.df)
+        assert_array_equal(results[1], self.df.columns)
+
+        results = tools._ensure_2d(self.series, False)
+        assert_frame_equal(results[0], self.df.iloc[:,[0]])
+        assert_equal(results[1], self.df.columns[0])
+
+    def test_numpy(self):
+        results = tools._ensure_2d(self.ndarray)
+        assert_array_equal(results[0], self.ndarray)
+        assert_equal(results[1], None)
+
+        results = tools._ensure_2d(self.ndarray[:,0])
+        assert_array_equal(results[0], self.ndarray[:,[0]])
+        assert_equal(results[1], None)