Repository: statsmodels/statsmodels

Issue:
Issue with lowess() smoother in statsmodels
Following Josef's suggestion (https://groups.google.com/forum/#!topic/pystatsmodels/A5KMexQA1D8), I am using `lowess()` from statsmodels.nonparametric.smoothers_lowess to do Lowess smoothing. When I try it on this data set of X and Y values "test_data.txt", available here: https://gist.github.com/yarden/5929702
then I get the error:

```
  File "/home/yarden/.local/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-linux-x86_64.egg/statsmodels/nonparametric/smoothers_lowess.py", line 162, in lowess
    x = np.array(x[sort_index])
IndexError: index 13632 is out of bounds for size 10146
```

My code just calls `lowess` on the x, y values in the file:

```
df = pandas.read_table("./test_data.txt", sep="\t")
from statsmodels.nonparametric.smoothers_lowess import lowess
y_vals = lowess(df["Y"], df["X"], return_sorted=False)
```

where `NaN` values in the `test_data.txt` file just represent missing values. What is going wrong here? 

Also, if I do:

```
df = df.dropna(subset=["X", "Y"], how="any")
```

Then it seems to work, but I thought NaN values are by default dropped (based on the `missing` argument to `lowess()`), so I am not sure what caused the problem in this case.

Thanks very much for your help.



Discussion:
BUG, my mistake for not having a test case with nans.

argsort on line 161 should use x not exog

`sort_index = np.argsort(x)`
not
`sort_index = np.argsort(exog)`

exog is the original full length, x, y have fewer rows if there are missing values

thanks very much for your prompt reply and fix!

Fix will land in master within a day, with test case.

Thanks for reporting it.

Quick followup: if I pass `lowess` the argument `missing="none", return_sorted=False`, it runs with no error. Does that mean that missing values (`nan`s) are ignored and `nan` and x values that are missing are simply returned as `nan`? That's the behavior I hope to achieve but I wanted to make sure I am not misunderstanding.

`missing="none"` means we don't check for nans. (to save computation when we already know we don't have NaNs or infs.)
If there are nans, then they are treated as nans in the floating point operations. All code just runs with float (double). I never checked how NaNs propagate in this case.
My guess from similar code is that all smoothed values that have a nan in their neighborhood will also turn into nans.

maybe you want `missing="drop", return_sorted=False` which drops nans and sorts the array for the calculations, but then put's it back in the same order and shape as the original data, with nans in the position where either x or y had a nan in the data.

That's the intended behavior, the unit tests might not include a case with nans given the previous error. (not all option combinations are unit tested.)

Ah I see, on closer look, it does put `nan`s on all values that are nearby `nan`s which is definitely not what I intended. I am looking for `missing="drop", return_sorted=False` based on your description: dropping `nan`s for the lowess operation, and then putting `nan`s back to preserve you get the same length arrays. 

I just tried with your data `return_sorted=True` and `return_sorted=False` return exactly the same _valid_ points.

If you want to speed up the calculations with a large dataset, then you could use the `delta` option which skips points that are not a minimum distance apart. (I never really tried it out. Carl implemented it to have the same fast option as in R.)

Did you change anything in the code? I don't see updates to `master` branch and when I try it with a recently cloned repository and `missing="drop"` (with either set of `return_sorted`) I get the error I mentioned above about dimensions being incorrect.

no, I just made the change `exog -> x` locally, in the code of statsmodels that my python is using.

I was mainly looking at plots, and trying to figure out how to write additional unit tests for these options.

You could change it in your installed statsmodels, then you can run it right away.


Resolving patch:
diff --git a/statsmodels/nonparametric/smoothers_lowess.py b/statsmodels/nonparametric/smoothers_lowess.py
--- a/statsmodels/nonparametric/smoothers_lowess.py
+++ b/statsmodels/nonparametric/smoothers_lowess.py
@@ -158,7 +158,7 @@ def lowess(endog, exog, frac=2.0/3.0, it=3, delta=0.0, is_sorted=False,
 
     if not is_sorted:
         # Sort both inputs according to the ascending order of x values
-        sort_index = np.argsort(exog)
+        sort_index = np.argsort(x)
         x = np.array(x[sort_index])
         y = np.array(y[sort_index])
 


Test patch:
diff --git a/statsmodels/nonparametric/tests/test_lowess.py b/statsmodels/nonparametric/tests/test_lowess.py
--- a/statsmodels/nonparametric/tests/test_lowess.py
+++ b/statsmodels/nonparametric/tests/test_lowess.py
@@ -11,7 +11,8 @@
 
 import os
 import numpy as np
-from numpy.testing import assert_almost_equal, assert_, assert_raises
+from numpy.testing import (assert_almost_equal, assert_, assert_raises,
+                           assert_equal)
 #import statsmodels.api as sm
 from statsmodels.nonparametric.smoothers_lowess import lowess
 
@@ -125,11 +126,30 @@ def test_options(self):
         # check with nans,  this changes the arrays
         y[[5, 6]] = np.nan
         x[3] = np.nan
-        actual_lowess1[[3, 5, 6], 1] = np.nan
+        mask_valid = np.isfinite(x) & np.isfinite(y)
+        #actual_lowess1[[3, 5, 6], 1] = np.nan
         actual_lowess = lowess(y, x, is_sorted=True)
-        assert_almost_equal(actual_lowess1, actual_lowess1, decimal=13)
+        actual_lowess1 = lowess(y[mask_valid], x[mask_valid], is_sorted=True)
+        assert_almost_equal(actual_lowess, actual_lowess1, decimal=13)
         assert_raises(ValueError, lowess, y, x, missing='raise')
 
+        perm_idx = np.arange(len(x))
+        np.random.shuffle(perm_idx)
+        yperm = y[perm_idx]
+        xperm = x[perm_idx]
+        actual_lowess2 = lowess(yperm, xperm, is_sorted=False)
+        assert_almost_equal(actual_lowess, actual_lowess2, decimal=13)
+
+        actual_lowess3 = lowess(yperm, xperm, is_sorted=False,
+                                return_sorted=False)
+        mask_valid = np.isfinite(xperm) & np.isfinite(yperm)
+        assert_equal(np.isnan(actual_lowess3), ~mask_valid)
+        # get valid sorted smoothed y from actual_lowess3
+        sort_idx = np.argsort(xperm)
+        yhat = actual_lowess3[sort_idx]
+        yhat = yhat[np.isfinite(yhat)]
+        assert_almost_equal(yhat, actual_lowess2[:,1], decimal=13)
+