Repository: scikit-learn/scikit-learn
Pull request: [MRG+1] Return nan in RadiusNeighborsRegressor for empty neighbor set
#### Reference Issue
Fixes #9654 



#### What does this implement/fix? Explain your changes.
RadiusNeighborsRegressor is behaving differently when there are no neighbors for a sample between when weights are or aren't used. This PR fixes this inconsistency. This PR also fixes raised error when no available data points for `RadiusNeighborRegression` using non-uniform weights.

Problem statement:
RadiusNeighborRegression error
#### Description
RadiusNeighborRegression has inconsistent output depending whether using weights that are uniform or by distance. 

- When using uniform weights then if no observations are available within the specified radius of an observation point then no it returns `np.nan`, 

- When using distance weights it raises `ZeroDivisionError: Weights sum to zero, can't be normalized` is raised from `np.average` as there are no weights to use for the observation. This is demonstrated with the following example below - copied from `RadiusNeighborsRegressor` where distance is specified and for X used for prediction is different,

#### Steps/Code to Reproduce
```python
from sklearn.neighbors import RadiusNeighborsRegressor   

X = [[0], [1], [2], [3]]
y = [[0], [0], [1], [1]]

neigh = RadiusNeighborsRegressor(radius=1.0, weights='distance')
neigh.fit(X, y) 

y_hat = neigh.predict([[-2],[0]])
```

#### Expected Results
```python
array([[ nan],
       [  0.]])
```

#### Actual Results
```
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-6-c1cb420157ca> in <module>()
      7 neigh.fit(X, y)
      8 
----> 9 y_hat = neigh.predict([[-1.5],[1]])

~\AppData\Local\Continuum\Miniconda3\lib\site-packages\sklearn\neighbors\regression.py in predict(self, X)
    294             y_pred = np.array([(np.average(_y[ind, :], axis=0,
    295                                            weights=weights[i]))
--> 296                                for (i, ind) in enumerate(neigh_ind)])
    297 
    298         if self._y.ndim == 1:

~\AppData\Local\Continuum\Miniconda3\lib\site-packages\sklearn\neighbors\regression.py in <listcomp>(.0)
    294             y_pred = np.array([(np.average(_y[ind, :], axis=0,
    295                                            weights=weights[i]))
--> 296                                for (i, ind) in enumerate(neigh_ind)])
    297 
    298         if self._y.ndim == 1:

~\AppData\Local\Continuum\Miniconda3\lib\site-packages\numpy\lib\function_base.py in average(a, axis, weights, returned)
   1138         if np.any(scl == 0.0):
   1139             raise ZeroDivisionError(
-> 1140                 "Weights sum to zero, can't be normalized")
   1141 
   1142         avg = np.multiply(a, wgt, dtype=result_dtype).sum(axis)/scl

ZeroDivisionError: Weights sum to zero, can't be normalized
```


#### Versions
Windows-10-10.0.15063-SP0
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)]
NumPy 1.13.1
SciPy 0.19.1
Scikit-Learn 0.18.2

I encountered the same problem on Mac OS and Linux machines.

Hints from issue discussion:

Commit message: Fix to issue 9654

Fixes problem with raise error when no available data points for RadiusNeighborRegression using non-uniform weights.

Patch to review:
diff --git a/sklearn/neighbors/regression.py b/sklearn/neighbors/regression.py
index 1180850b8d21a..e1ca93df2f13d 100644
--- a/sklearn/neighbors/regression.py
+++ b/sklearn/neighbors/regression.py
@@ -291,9 +291,28 @@ def predict(self, X):
             y_pred = np.array([np.mean(_y[ind, :], axis=0)
                                for ind in neigh_ind])
         else:
-            y_pred = np.array([(np.average(_y[ind, :], axis=0,
-                                           weights=weights[i]))
-                               for (i, ind) in enumerate(neigh_ind)])
+            has_neighors = np.array(list(map(len, neigh_ind)))>0
+
+            if np.min(has_neighors):
+                y_pred = np.array([(np.average(_y[ind, :], axis=0,
+                                               weights=weights[i]))
+                                   for (i, ind) in enumerate(neigh_ind)])
+
+            else:
+
+                y_pred = np.full((X.shape[0],_y.shape[1]), np.nan)
+
+                neigh_ind_sub = [(i,n) for (i,(n, has_neighbor))
+                                 in enumerate(zip(neigh_ind,has_neighors))
+                                 if has_neighbor]
+
+
+                i, ind =  list(neigh_ind_sub)[0]
+
+                y_pred[np.where(has_neighors),:] = \
+                        np.array([(np.average(_y[ind, :], axis=0,
+                                              weights=weights[i]))
+                                  for (i, ind) in neigh_ind_sub])
 
         if self._y.ndim == 1:
             y_pred = y_pred.ravel()
diff --git a/sklearn/neighbors/tests/test_neighbors.py b/sklearn/neighbors/tests/test_neighbors.py
index 052c83c71d2e7..1d9278a54baee 100644
--- a/sklearn/neighbors/tests/test_neighbors.py
+++ b/sklearn/neighbors/tests/test_neighbors.py
@@ -658,6 +658,9 @@ def test_radius_neighbors_regressor(n_samples=40,
             y_pred = neigh.predict(X[:n_test_pts] + epsilon)
             assert_true(np.all(abs(y_pred - y_target) < radius / 2))
 
+            # test fix for issue #9654
+            # test that nan is returned when no nearby observations
+            assert_true(np.isnan(neigh.predict([[-1]])[0,0]))
 
 def test_RadiusNeighborsRegressor_multioutput_with_uniform_weight():
     # Test radius neighbors in multi-output regression (uniform weight)


Code review comments:
diff_hunk: @@ -291,9 +291,28 @@ def predict(self, X):
             y_pred = np.array([np.mean(_y[ind, :], axis=0)
                                for ind in neigh_ind])
         else:
-            y_pred = np.array([(np.average(_y[ind, :], axis=0,
-                                           weights=weights[i]))
path: sklearn/neighbors/regression.py
text: @user1:
can't we just add `if len(ind) else np.nan` here?

@author:
Right, great idea - I've changed this now.

@author:
Hmm, on second thought I guess that it should match the second axis of `_y`? I have made a commit where it is inserting `np.full(_y.shape[1], np.nan)` under the else clause.

@user1:
Yes, that's right. Is it tested?

@author:
Yes, I've tested it - works for one or more _y variables. Should I document these tests?