Repository: scipy/scipy

Issue:
Using cdist with wminkowski disagrees with wminkowski
It appears that running `cdist` using the `wminkowski` metric reports inaccurate results that disagree with running `wminkowski` on each vector combination individually. The issue goes away if the absolute value of `w` is taken before giving it to `cdist`. However this is done for the user in `wminkowski` matching the equation in the docs. Hence the disagreement. Below is an example of this behavior with details about how to install a `conda` environment that will reproduce the issue.


Example:

<details>

```python
In [1]: import numpy as np

In [2]: import scipy.spatial.distance as spdist

In [3]: np.random.seed(0)

In [4]: a_u = 2 * np.random.random((7, 6)) - 1

In [5]: a_v = 2 * np.random.random((3, 6)) - 1

In [6]: p = 1.6

In [7]: w = 2 * np.random.random(a_u.shape[-1:]) - 1

In [8]: spdist.cdist(a_u, a_v, metric="wminkowski", p=p, w=w)
Out[8]: 
array([[ nan,  nan,  nan],
       [ nan,  nan,  nan],
       [ nan,  nan,  nan],
       [ nan,  nan,  nan],
       [ nan,  nan,  nan],
       [ nan,  nan,  nan],
       [ nan,  nan,  nan]])

In [9]: d = np.empty((len(a_u), len(a_v)), dtype=float)
   ...: for i in range(len(a_u)):
   ...:     for j in range(len(a_v)):
   ...:         d[i, j] = spdist.wminkowski(a_u[i], a_v[j], p=p, w=w)
   ...:         

In [10]: d
Out[10]: 
array([[ 1.24606122,  1.16395049,  1.25037412],
       [ 1.84842639,  1.06442873,  1.44461084],
       [ 1.96393331,  1.98499393,  1.75982539],
       [ 1.51823304,  1.68759411,  1.87569175],
       [ 1.63240776,  1.32480509,  1.47219684],
       [ 1.50025279,  1.54607896,  1.40125501],
       [ 1.0061864 ,  1.26567631,  1.31885044]])

In [11]: spdist.cdist(a_u, a_v, metric="wminkowski", p=p, w=abs(w))
Out[11]: 
array([[ 1.24606122,  1.16395049,  1.25037412],
       [ 1.84842639,  1.06442873,  1.44461084],
       [ 1.96393331,  1.98499393,  1.75982539],
       [ 1.51823304,  1.68759411,  1.87569175],
       [ 1.63240776,  1.32480509,  1.47219684],
       [ 1.50025279,  1.54607896,  1.40125501],
       [ 1.0061864 ,  1.26567631,  1.31885044]])
```

</details>

<br>
Environment:
<br>
<br>

<details>

```yaml
name: test
channels:
- conda-forge
- defaults
dependencies:
- appnope=0.1.0=py36_0
- blas=1.1=openblas
- ca-certificates=2017.7.27.1=0
- certifi=2017.7.27.1=py36_0
- decorator=4.1.2=py36_0
- ipython=6.2.0=py36_0
- ipython_genutils=0.2.0=py36_0
- jedi=0.10.2=py36_0
- libgfortran=3.0.0=0
- ncurses=5.9=10
- numpy=1.13.1=py36_blas_openblas_200
- openblas=0.2.19=2
- openssl=1.0.2l=0
- pexpect=4.2.1=py36_0
- pickleshare=0.7.4=py36_0
- pip=9.0.1=py36_0
- prompt_toolkit=1.0.15=py36_0
- ptyprocess=0.5.2=py36_0
- pygments=2.2.0=py36_0
- python=3.6.2=0
- readline=6.2=0
- scipy=0.19.1=py36_blas_openblas_202
- setuptools=36.3.0=py36_0
- simplegeneric=0.8.1=py36_0
- six=1.11.0=py36_1
- sqlite=3.13.0=1
- tk=8.5.19=2
- traitlets=4.3.2=py36_0
- wcwidth=0.1.7=py36_0
- wheel=0.30.0=py_1
- xz=5.2.3=0
- zlib=1.2.11=0
```

<details>


Discussion:
FWICT looks like the weight isn't inside the `abs` function call.

https://github.com/scipy/scipy/blob/676da8f65d38cd62c89799aec7444df227424e04/scipy/spatial/src/distance_impl.h#L358
With recent master (676da8f) that gives me ``ValueError: array must not contain infs or NaNs`` for ``In [8]: spdist.cdist...``. We've made a recent fix to it because of backwards compat issues - what version are you using?


Interesting. Using 0.19.1. If this is meant to be an error raising case, it seems that the error should be updated to explicitly mention negative weights are not allowed.
Negative weights raising an informative error sounds right to me. It is a change from previous versions, but I doubt that allowing negative weights was ever a conscious decision.

@apbard thoughts?
> I doubt that allowing negative weights was ever a conscious decision.

Totally agree, probably just an overlooked different behaviour from C code and linalg.norm.

In scipy 1.0, the `dist = norm(w * (u - v), ord=p)` has been translated into [norm(np.sqrt(w) * (u - v), ord=p)](https://github.com/scipy/scipy/blob/master/scipy/spatial/distance.py#L491),  thus we should get the NaNs also with the python weighted version of `minkowski`. 

I suspect that all new metrics version would be somehow affected by negative weights. To solve this I think we the most straightforward solution is to force all weights to be positive.
Actually, there are some weird corner cases where negative weights might be meaningful...We could normalise the weights so that they sum up to 1 (with the only constraint that the sum is >0) and solve "sqrt" issues metric by metric...but it is probably not worth the effort.
> thus we should get the NaNs also with the python weighted version of minkowski.

NaN's or the ValueError we get right now in master? I'd prefer the latter I think. Would be good to have a check for that for all metrics.

Resolving patch:
diff --git a/scipy/spatial/distance.py b/scipy/spatial/distance.py
--- a/scipy/spatial/distance.py
+++ b/scipy/spatial/distance.py
@@ -327,6 +327,13 @@ def _validate_vector(u, dtype=None):
     return u
 
 
+def _validate_weights(w, dtype=None):
+    w = _validate_vector(w, dtype=dtype)
+    if np.any(w < 0):
+        raise ValueError("Input weights should be all non-negative")
+    return w
+
+
 def _validate_wminkowski_kwargs(X, m, n, **kwargs):
     w = kwargs.pop('w', None)
     if w is None:
@@ -483,7 +490,7 @@ def minkowski(u, v, p=2, w=None):
         raise ValueError("p must be at least 1")
     u_v = u - v
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         if p == 1:
             root_w = w
         if p == 2:
@@ -549,7 +556,7 @@ def wminkowski(u, v, p, w):
     1.0
 
     """
-    w = _validate_vector(w)
+    w = _validate_weights(w)
     return minkowski(u, v, p=p, w=w**p)
 
 
@@ -641,7 +648,7 @@ def sqeuclidean(u, v, w=None):
     u_v = u - v
     u_v_w = u_v  # only want weights applied once
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         u_v_w = w * u_v
     return np.dot(u_v, u_v_w)
 
@@ -680,7 +687,7 @@ def correlation(u, v, w=None, centered=True):
     u = _validate_vector(u)
     v = _validate_vector(v)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
     if centered:
         umu = np.average(u, weights=w)
         vmu = np.average(v, weights=w)
@@ -788,7 +795,7 @@ def hamming(u, v, w=None):
         raise ValueError('The 1d arrays must have equal lengths.')
     u_ne_v = u != v
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
     return np.average(u_ne_v, weights=w)
 
 
@@ -841,7 +848,7 @@ def jaccard(u, v, w=None):
     nonzero = np.bitwise_or(u != 0, v != 0)
     unequal_nonzero = np.bitwise_and((u != v), nonzero)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         nonzero = w * nonzero
         unequal_nonzero = w * unequal_nonzero
     dist = np.double(unequal_nonzero.sum()) / np.double(nonzero.sum())
@@ -897,7 +904,7 @@ def kulsinski(u, v, w=None):
     if w is None:
         n = float(len(u))
     else:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         n = w.sum()
     (nff, nft, ntf, ntt) = _nbool_correspond_all(u, v, w=w)
 
@@ -986,7 +993,7 @@ def cityblock(u, v, w=None):
     v = _validate_vector(v)
     l1_diff = abs(u - v)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         l1_diff = w * l1_diff
     return l1_diff.sum()
 
@@ -1038,7 +1045,7 @@ def mahalanobis(u, v, VI):
     return np.sqrt(m)
 
 
-def chebyshev(u, v):
+def chebyshev(u, v, w=None):
     """
     Compute the Chebyshev distance.
 
@@ -1055,6 +1062,9 @@ def chebyshev(u, v):
         Input vector.
     v : (N,) array_like
         Input vector.
+    w : (N,) array_like, optional
+        The weights for each value in `u` and `v`. Default is None,
+        which gives each value a weight of 1.0
 
     Returns
     -------
@@ -1072,6 +1082,12 @@ def chebyshev(u, v):
     """
     u = _validate_vector(u)
     v = _validate_vector(v)
+    if w is not None:
+        w = _validate_weights(w)
+        has_weight = w > 0
+        if has_weight.sum() < w.size:
+            u = u[has_weight]
+            v = v[has_weight]
     return max(abs(u - v))
 
 
@@ -1117,7 +1133,7 @@ def braycurtis(u, v, w=None):
     l1_diff = abs(u - v)
     l1_sum = abs(u + v)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         l1_diff = w * l1_diff
         l1_sum = w * l1_sum
     return l1_diff.sum() / l1_sum.sum()
@@ -1166,7 +1182,7 @@ def canberra(u, v, w=None):
     u = _validate_vector(u)
     v = _validate_vector(v, dtype=np.float64)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
     olderr = np.seterr(invalid='ignore')
     try:
         abs_uv = abs(u - v)
@@ -1222,7 +1238,7 @@ def yule(u, v, w=None):
     u = _validate_vector(u)
     v = _validate_vector(v)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
     (nff, nft, ntf, ntt) = _nbool_correspond_all(u, v, w=w)
     return float(2.0 * ntf * nft / np.array(ntt * nff + ntf * nft))
 
@@ -1281,7 +1297,7 @@ def dice(u, v, w=None):
     u = _validate_vector(u)
     v = _validate_vector(v)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
     if u.dtype == v.dtype == bool and w is None:
         ntt = (u & v).sum()
     else:
@@ -1341,7 +1357,7 @@ def rogerstanimoto(u, v, w=None):
     u = _validate_vector(u)
     v = _validate_vector(v)
     if w is not None:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
     (nff, nft, ntf, ntt) = _nbool_correspond_all(u, v, w=w)
     return float(2.0 * (ntf + nft)) / float(ntt + nff + (2.0 * (ntf + nft)))
 
@@ -1397,7 +1413,7 @@ def russellrao(u, v, w=None):
         ntt = (u * v).sum()
         n = float(len(u))
     else:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         ntt = (u * v * w).sum()
         n = w.sum()
     return float(n - ntt) / n
@@ -1455,7 +1471,7 @@ def sokalmichener(u, v, w=None):
         ntt = (u * v).sum()
         nff = ((1.0 - u) * (1.0 - v)).sum()
     else:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         ntt = (u * v * w).sum()
         nff = ((1.0 - u) * (1.0 - v) * w).sum()
     (nft, ntf) = _nbool_correspond_ft_tf(u, v)
@@ -1512,7 +1528,7 @@ def sokalsneath(u, v, w=None):
     elif w is None:
         ntt = (u * v).sum()
     else:
-        w = _validate_vector(w)
+        w = _validate_weights(w)
         ntt = (u * v * w).sum()
     (nft, ntf) = _nbool_correspond_ft_tf(u, v, w=w)
     denom = np.array(ntt + 2.0 * (ntf + nft))


Test patch:
diff --git a/scipy/spatial/tests/test_distance.py b/scipy/spatial/tests/test_distance.py
--- a/scipy/spatial/tests/test_distance.py
+++ b/scipy/spatial/tests/test_distance.py
@@ -361,8 +361,11 @@ def wrapped(*args, **kwargs):
 wmatching = whamming = _weight_checked(hamming, dud_test=False)
 wyule = _weight_checked(yule)
 wdice = _weight_checked(dice)
+wcityblock = _weight_checked(cityblock)
+wchebyshev = _weight_checked(chebyshev)
 wcosine = _weight_checked(cosine)
 wcorrelation = _weight_checked(correlation)
+wkulsinski = _weight_checked(kulsinski)
 wminkowski = _weight_checked(minkowski, const_test=False)
 wjaccard = _weight_checked(jaccard)
 weuclidean = _weight_checked(euclidean, const_test=False)
@@ -1907,6 +1910,19 @@ def test_Xdist_deprecated_args():
                 assert_(len(log) == 2)
 
 
+def test_Xdist_non_negative_weights():
+    X = eo['random-float32-data'][::5, ::2]
+    w = np.ones(X.shape[1])
+    w[::5] = -w[::5]
+    for metric in _METRICS_NAMES:
+        if metric in ['seuclidean', 'mahalanobis']:
+            continue
+
+        for m in [metric, eval(metric), "test_" + metric]:
+            assert_raises(ValueError, pdist, X, m, w=w)
+            assert_raises(ValueError, cdist, X, X, m, w=w)
+
+
 def test__validate_vector():
     x = [1, 2, 3]
     y = _validate_vector(x)