Repository: scipy/scipy

Issue:
scipy.stats.mode truncates strings
It seems that it will truncate the results to the shortest string it encountered. This can of course actually lead to misleading results.

```
>>> import numpy as np
>>> from scipy import stats
>>> a = np.array([['rain', 'showers'], ['showers', 'showers']])
>>> a
array([['rain', 'showers'],
       ['showers', 'showers']], 
      dtype='|S7')
>>> stats.mode(a)
(array([['rain', 'show']], 
      dtype='|S4'), array([[ 1.,  2.]]))
```

Misleading results:

```
>>> a = np.array([['b', 'a'], ['aa', 'aa']])
>>> stats.mode(a, axis=None)
(array(['a'], 
      dtype='|S1'), array([ 2.]))
```

Also, I don't know how empty strings are supposed to be handled, but it doesn't seem to like them.

```
>>> a = np.array([['', 'a', 'aa', 'aa']])
>>> stats.mode(a, axis=0)
(array([[ '\x00=\xca\x01\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x11\x03',
        'a', 'aa', 'aa']], 
      dtype='|S32'), array([[ 1.,  1.,  1.,  1.]]))
```



Discussion:
@pfctdayelise what numpy and scipy versions do you use? With numpy 1.5.1 and scipy master I get:

```
In [14]: a = np.array([['rain', 'showers'], ['showers', 'showers']])

In [15]: a
Out[15]: 
array([['rain', 'showers'],
       ['showers', 'showers']], 
      dtype='|S7')

In [16]: stats.mode(a)
Out[16]: 
(array([['rain', 'showers']], 
      dtype='|S8'), array([[ 1.,  2.]]))
```

I'm not surprised that this doesn't work well. `mode` and all of `scipy.stats` aren't really designed for working with arrays of strings, and I think there's not a single test for behavior with string input.

We should probably just document that `stats` isn't meant for strings. @josef-pkt @WarrenWeckesser do you agree with that?

By my reading, changing line https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L619
from
`oldmostfreq = np.zeros(testshape)`
to
`oldmostfreq = np.zeros(testshape, dtype=a.dtype)`

fixes the problem...it would be nicer to do that and mode is one statistical function that makes plenty of sense to use with strings!
I am using numpy 1.7.1 with scipy 0.12.0.

If the fix is that simple, then we can of course make it.

Agreed that `mode` makes sense with strings, my worry is that we end up with a similar situation as masked arrays where half the functions work and half don't, and it's very hard to guess which ones will work. So I'd still like to document that strings aren't supported, and then for the couple of functions where strings do make sense to explicitly put that in the docstring.

I think there was a change in numpy at some point that started to truncate strings. I get different results between numpy 1.5 and numpy 1.6 (and different scipy versions).

The fix looks right to me.
There are only a few functions in scipy.stats that make sense for strings. And I don't see why we shouldn't support string, expecially if the function is not optimized for a numeric type.

BTW: numpy strings get truncated also in 1.5 if length is >8.

numpy 1.5.1

```
>>> o = np.where(np.arange(2) > 0.5, 'rain', np.zeros(2))
>>> o
array(['0.0', 'rain'], 
      dtype='|S8')
>>> np.where(np.arange(2) > 0.5, 'showers curtain', o)
array(['0.0', 'showers '], 
      dtype='|S8')
```

numpy 1.6.1

```
>>> o = np.where(np.arange(2) > 0.5, 'rain', np.zeros(2))
>>> o
array(['0.0', 'rain'], 
      dtype='|S4')
>>> o = np.where(np.arange(2) > 0.5, 'showers', o)
>>> o
array(['0.0', 'show'], 
      dtype='|S4')
```

@josef-pkt I think that might be more about having the right dtype to start with, rather than string handling in general?

So would you like me to try and prepare a PR or will someone here handle it?

@pfctdayelise a PR would be excellent!

+1 for a fix to properly handle strings.  (If we're not going to fix it, then we should add code to raise a `ValueError` when given any unacceptable dtype; but then, if we're going to touch the code, and handling strings is easy, we might as well handle them.)

Note that the suggested change introduces an API change, because the dtype of the first returned value will now depend on the dtype of the argument.  Consider this example:

```
In [51]: x
Out[51]: array([1, 5, 4, 3, 3, 3, 1], dtype=uint8)

In [52]: vals, counts = mode(x)

In [53]: vals
Out[53]: array([ 3.])
```

When the fix is made, the dtype of `vals` will be `np.uint8`.

I consider the change to be a bug fix (i.e. it has been a bug all along that the dtype of the first return value doesn't match the dtype of the input), so I wouldn't insist on a deprecation cycle.

Also, don't forget (as I often do!) to at least take a look at the corresponding `mstats` function `scipy.stats.mstats.mode`.  The fix there is probably not so simple.

@WarrenWeckesser hmm, indeed...I haven't used masked arrays before now. 

Is it normal that the [mstats docs](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.mode.html) just show the regular stats code example? That seems strange to me.

mstats.mode is using find_repeats, which doesn't work with strings. I could rewrite it to use the same method as stats.mode, but...I also note this bug, [stats.mode seems suprisingly slow (at least on 1-2 dimensions)](https://github.com/scipy/scipy/issues/3035) which proposed to rewrite stats.mode to use find_repeats.

In my work we usually use a representation of string arrays like a lookup table, ie array of ints that correspond to indices in a simple Python list of strings. Is there a simple or natural way to do this in numpy? Doing this would enable use of find_repeats.

Sigh, this is getting complicated.

@pfctdayelise   The minimal pull request for this would just add the dtype to fix the function and add unit tests (for longer strings and, maybe, object arrays).

Improving performance might need to branch on dtype, using the more efficient version for purely numeric data. (np.histogram calculates the frequency just by sorting and getting the diff AFAIK, which should also work for all dtypes, but might have problems with a vectorized solution.)


Resolving patch:
diff --git a/scipy/stats/stats.py b/scipy/stats/stats.py
--- a/scipy/stats/stats.py
+++ b/scipy/stats/stats.py
@@ -616,7 +616,7 @@ def mode(a, axis=0):
     scores = np.unique(np.ravel(a))       # get ALL unique values
     testshape = list(a.shape)
     testshape[axis] = 1
-    oldmostfreq = np.zeros(testshape)
+    oldmostfreq = np.zeros(testshape, dtype=a.dtype)
     oldcounts = np.zeros(testshape)
     for score in scores:
         template = (a == score)


Test patch:
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -8,6 +8,7 @@
 """
 from __future__ import division, print_function, absolute_import
 
+import sys
 import warnings
 from collections import namedtuple
 
@@ -1160,6 +1161,71 @@ def test_basic(self):
         assert_almost_equal(vals[0][0],6)
         assert_almost_equal(vals[1][0],3)
 
+    def test_axes(self):
+        data1 = [10,10,30,40]
+        data2 = [10,10,10,10]
+        data3 = [20,10,20,20]
+        data4 = [30,30,30,30]
+        data5 = [40,30,30,30]
+        arr = np.array([data1, data2, data3, data4, data5])
+
+        vals = stats.mode(arr, axis=None)
+        assert_almost_equal(vals[0],np.array([30]))
+        assert_almost_equal(vals[1],np.array([8]))
+        
+        vals = stats.mode(arr, axis=0)
+        assert_almost_equal(vals[0],np.array([[10,10,30,30]]))
+        assert_almost_equal(vals[1],np.array([[2,3,3,2]]))
+
+        vals = stats.mode(arr, axis=1)
+        assert_almost_equal(vals[0],np.array([[10],[10],[20],[30],[30]]))
+        assert_almost_equal(vals[1],np.array([[2],[4],[3],[4],[3]]))
+
+    def test_strings(self):
+        data1 = ['rain', 'showers', 'showers']
+        vals = stats.mode(data1)
+        expected = ['showers']
+        assert_equal(vals[0][0], 'showers')
+        assert_equal(vals[1][0], 2)
+
+    @dec.knownfailureif(sys.version_info > (3,), 'numpy github issue 641')
+    def test_mixed_objects(self):
+        objects = [10, True, np.nan, 'hello', 10]
+        arr = np.empty((5,), dtype=object)
+        arr[:] = objects
+        vals = stats.mode(arr)
+        assert_equal(vals[0][0], 10)
+        assert_equal(vals[1][0], 2)
+
+    def test_objects(self):
+        """Python objects must be sortable (le + eq) and have ne defined
+        for np.unique to work. hash is for set.
+        """
+        class Point(object):
+            def __init__(self, x):
+                self.x = x
+
+            def __eq__(self, other):
+                return self.x == other.x
+
+            def __ne__(self, other):
+                return self.x != other.x
+
+            def __lt__(self, other):
+                return self.x < other.x
+
+            def __hash__(self):
+                return hash(self.x)
+
+        points = [Point(x) for x in [1,2,3,4,3,2,2,2]]
+        arr = np.empty((8,), dtype=object)
+        arr[:] = points
+        assert len(set(points)) == 4
+        assert_equal(np.unique(arr).shape, (4,))
+        vals = stats.mode(arr)
+        assert_equal(vals[0][0], Point(2))
+        assert_equal(vals[1][0], 4)
+
 
 class TestVariability(TestCase):