Repository: scipy/scipy

Issue:
logsumexp with zero scaling factor
``` python
>>> scipy.misc.logsumexp([-1000])
-1000.0
>>> scipy.misc.logsumexp([-1000, 0], b=[1, 0])
-inf
```



Discussion:
Could anyone point to the problem faced here. Isn't the output the expected one ?
It matches with the answer of 

```
>>>a=[-1000,0]
>>>b=[1,0]
>>>np.log(np.sum(b*np.exp(a)))
-inf
```

This is due to taking the exponent of `a - max(a)`. So the second element matters (because it's the largest value) even if `b == 0` for that element. 

I don't think that this is a bug though. Implementing a special case for `b == 0` looks wrong, because `b` is expected to be floating point and comparing floating point zero to `0` is problematic. Also, the current code is consistent in that the answer is OK when taking the limit of b going to zero from above.

So closing.

@argriffing please reopen if I'm missing your point.

Thanks for the comments. I'm not trying to re-open this issue but I just thought I'd explain the rationale in a bit more detail. Maybe too much detail :) Also I'd thought the original motivation had been from a stackoverflow question but I can't seem to find it.

> Could anyone point to the problem faced here. Isn't the output the expected one ?
> It matches with the answer of
> 
> ```
> >>a=[-1000,0]
> >>b=[1,0]
> >>np.log(np.sum(b*np.exp(a)))
> -inf
> ```

One of the purposes of `logsumexp` is to compute the solution more stably than using the naive evaluation.

> This is due to taking the exponent of `a - max(a)`. So the second element matters (because it's the largest value) even if `b == 0` for that element.

Yes this agrees with my understanding of the current implementation in scipy.

> I don't think that this is a bug though.

Right it's not necessarily a bug, just an opportunity to further improve the stability of the evaluation.

> Implementing a special case for `b == 0` looks wrong, because b is expected to be floating point and comparing floating point zero to 0 is problematic.

It is true that exact comparison of floating point numbers is often incorrect. On the other hand, in some cases it makes sense.  For example [`xlogy`](https://github.com/scipy/scipy/blob/maintenance/0.15.x/scipy/special/_xlogy.pxd) uses exact comparison of a floating point coefficient to zero.

> Also, the current code is consistent in that the answer is OK when taking the limit of b going to zero from above.

I'm not completely sure I understand this point, so I'll address two different interpretations.

1) Maybe you mean that `logsumexp([-1000, 0], b=[1, eps])` is OK (in the sense of mathematical correctness) when taking the limit `eps -> 0` from above.  This looks plausible from a plot, but it is not true. For example in the following code the top subplot looks like it is linearly decreasing towards negative infinity as the log of `eps` approaches negative infinity.

```
import numpy as np
import matplotlib.pyplot as plt
from scipy.misc import logsumexp
def mkplot(ax, k):
    n = 41
    x1 = np.logspace(-1, -k, num=n)
    x0 = np.ones_like(x1)
    b = np.vstack((x0, x1)).T
    a = np.array([[-k, 0.0]]).repeat(n, axis=0)
    y = logsumexp(a, axis=1, b=b)
    line, = ax.plot(x1, y)
    ax.set_xscale('log')
fig, (ax1, ax2) = plt.subplots(2)
mkplot(ax1, 1000)
mkplot(ax2, 20)
plt.savefig('logsumexp.png')
```

![logsumexp](https://cloud.githubusercontent.com/assets/101416/7350813/77011f34-eccf-11e4-93e2-5d745ad8a41f.png)

But in the bottom subplot (which uses a less numerically taxing value of `a`) you can see that this pattern doesn't hold for the whole domain, and that the limit of the `logsumexp` is finite as `eps -> 0` from above.

2) Maybe you mean that `logsumexp([-1000, 0], b=[1, eps])` is OK when taking the limit of `eps -> 0` from above, in the sense that it is numerically consistent with `logsumexp([-1000, 0], b=[1, 0])` because both give the value `-inf`, and that this consistency is good.

```
>>> logsumexp([-1000.0, 0.0], b=[1.0, np.exp(-740)])
-739.99742066489091
>>> logsumexp([-1000.0, 0.0], b=[1.0, np.exp(-760)])
-inf
>>> logsumexp([-1000.0, 0.0], b=[1.0, 0.0])
-inf
```

The current implementation of the evaluation jumps from about -750 to `-inf` when `eps` becomes smaller than floating point numbers can handle, whereas an alternative implementation would jump from about -750 to the more accurate value of `-1000` when `eps` reaches this threshold. So the alternative implementation would not be less consistent, at least for this example.

Hmm, that is a quite elaborate description of what I meant (I was thinking mostly about 2) and how my interpretation was a bit too fast:)

I'm still not sure if it's worth making a change here, but it's at least worth thinking about it.


Resolving patch:
diff --git a/scipy/misc/common.py b/scipy/misc/common.py
--- a/scipy/misc/common.py
+++ b/scipy/misc/common.py
@@ -9,7 +9,7 @@
 import numpy as np
 from numpy import (exp, log, asarray, arange, newaxis, hstack, product, array,
                    zeros, eye, poly1d, r_, sum, fromstring, isfinite,
-                   squeeze, amax, reshape)
+                   squeeze, amax, reshape, sign, broadcast_arrays)
 
 from scipy._lib._version import NumpyVersion
 
@@ -20,7 +20,7 @@
 _NUMPY_170 = (NumpyVersion(numpy.__version__) >= NumpyVersion('1.7.0'))
 
 
-def logsumexp(a, axis=None, b=None, keepdims=False):
+def logsumexp(a, axis=None, b=None, keepdims=False, return_sign=False):
     """Compute the log of the sum of exponentials of input elements.
 
     Parameters
@@ -41,17 +41,27 @@ def logsumexp(a, axis=None, b=None, keepdims=False):
         .. versionadded:: 0.15.0
     b : array-like, optional
         Scaling factor for exp(`a`) must be of the same shape as `a` or
-        broadcastable to `a`.
+        broadcastable to `a`. These values may be negative in order to
+        implement subtraction.
 
         .. versionadded:: 0.12.0
+    return_sign : bool, optional
+        If this is set to True, the result will be a pair containing sign
+        information; if False, results that are negative will be returned
+        as NaN. Default is False (no sign information).
 
+        .. versionadded:: 0.16.0
     Returns
     -------
     res : ndarray
         The result, ``np.log(np.sum(np.exp(a)))`` calculated in a numerically
         more stable way. If `b` is given then ``np.log(np.sum(b*np.exp(a)))``
         is returned.
-
+    sgn : ndarray
+        If return_sign is True, this will be an array of floating-point
+        numbers matching res and +1, 0, or -1 depending on the sign
+        of the result. If False, only one result is returned.
+        
     See Also
     --------
     numpy.logaddexp, numpy.logaddexp2
@@ -79,9 +89,20 @@ def logsumexp(a, axis=None, b=None, keepdims=False):
     9.9170178533034665
     >>> np.log(np.sum(b*np.exp(a)))
     9.9170178533034647
+
+    Returning a sign flag
+
+    >>> logsumexp([1,2],b=[1,-1],return_sign=True)
+    (1.5413248546129181, -1.0)
+
     """
     a = asarray(a)
-
+    if b is not None:
+        a, b = broadcast_arrays(a,b)
+        if np.any(b == 0):
+            a = a + 0.  # promote to at least float
+            a[b == 0] = -np.inf
+    
     # keepdims is available in numpy.sum and numpy.amax since NumPy 1.7.0
     #
     # Because SciPy supports versions earlier than 1.7.0, we have to handle
@@ -109,20 +130,25 @@ def logsumexp(a, axis=None, b=None, keepdims=False):
             a_max = 0
 
         if b is not None:
-            b = asarray(b)
             tmp = b * exp(a - reshape(a_max, sh_keepdims))
         else:
             tmp = exp(a - reshape(a_max, sh_keepdims))
 
         # suppress warnings about log of zero
         with np.errstate(divide='ignore'):
-            out = log(sum(tmp, axis=axis))
+            s = sum(tmp, axis=axis)
+            if return_sign:
+                sgn = sign(s)
+                s *= sgn  # /= makes more sense but we need zero -> zero
+            out = log(s)
 
         out += a_max
 
         if keepdims:
             # Put back the reduced axes with size one
             out = reshape(out, sh_keepdims)
+            if return_sign:
+                sgn = reshape(sgn, sh_keepdims)
     else:
         # This is a more elegant implementation, requiring NumPy >= 1.7.0
         a_max = amax(a, axis=axis, keepdims=True)
@@ -140,14 +166,20 @@ def logsumexp(a, axis=None, b=None, keepdims=False):
 
         # suppress warnings about log of zero
         with np.errstate(divide='ignore'):
-            out = log(sum(tmp, axis=axis, keepdims=keepdims))
+            s = sum(tmp, axis=axis, keepdims=keepdims)
+            if return_sign:
+                sgn = sign(s)
+                s *= sgn  # /= makes more sense but we need zero -> zero
+            out = log(s)
 
         if not keepdims:
             a_max = squeeze(a_max, axis=axis)
-
         out += a_max
 
-    return out
+    if return_sign:
+        return out, sgn
+    else:
+        return out
 
 
 def central_diff_weights(Np, ndiv=1):


Test patch:
diff --git a/scipy/misc/tests/test_common.py b/scipy/misc/tests/test_common.py
--- a/scipy/misc/tests/test_common.py
+++ b/scipy/misc/tests/test_common.py
@@ -2,7 +2,7 @@
 
 import numpy as np
 from numpy.testing import (assert_array_equal, assert_almost_equal,
-                           assert_array_almost_equal, assert_equal)
+                           assert_array_almost_equal, assert_equal, assert_)
 
 from scipy.misc import pade, logsumexp, face, ascent
 
@@ -112,6 +112,63 @@ def test_logsumexp_b():
     assert_array_almost_equal(np.exp(logsumexp(logX, b=B, axis=1)),
                                 (B * X).sum(axis=1))
 
+def test_logsumexp_sign():
+    a = [1,1,1]
+    b = [1,-1,-1]
+
+    r, s = logsumexp(a, b=b, return_sign=True)
+    assert_almost_equal(r,1)
+    assert_equal(s,-1)
+
+def test_logsumexp_sign_zero():
+    a = [1,1]
+    b = [1,-1]
+
+    r, s = logsumexp(a, b=b, return_sign=True)
+    assert_(not np.isfinite(r))
+    assert_(not np.isnan(r))
+    assert_(r < 0)
+    assert_equal(s,0)
+
+def test_logsumexp_sign_shape():
+    a = np.ones((1,2,3,4))
+    b = np.ones_like(a)
+
+    r, s = logsumexp(a, axis=2, b=b, return_sign=True)
+
+    assert_equal(r.shape, s.shape)
+    assert_equal(r.shape, (1,2,4))
+
+    if NumpyVersion(np.__version__) >= NumpyVersion('1.7.0'):
+        r, s = logsumexp(a, axis=(1,3), b=b, return_sign=True)
+
+        assert_equal(r.shape, s.shape)
+        assert_equal(r.shape, (1,3))
+
+def test_logsumexp_shape():
+    a = np.ones((1,2,3,4))
+    b = np.ones_like(a)
+
+    r = logsumexp(a, axis=2, b=b)
+
+    assert_equal(r.shape, (1,2,4))
+    if NumpyVersion(np.__version__) >= NumpyVersion('1.7.0'):
+        r = logsumexp(a, axis=(1,3), b=b)
+
+        assert_equal(r.shape, (1,3))
+
+def test_logsumexp_b_zero():
+    a = [1,10000]
+    b = [1,0]
+
+    assert_almost_equal(logsumexp(a, b=b), 1)
+
+def test_logsumexp_b_shape():
+    a = np.zeros((4,1,2,1))
+    b = np.ones((3,1,5))
+
+    logsumexp(a, b=b)
+
 
 def test_face():
     assert_equal(face().shape, (768, 1024, 3))